Time
S
Nick
Message
08:41 juancorr joined #dataverse
11:28 pdurbin joined #dataverse
11:37 pdurbin
Good morning, everyone. Is anyone blocked?
14:08 donsizemore joined #dataverse
14:10 donsizemore
@pdurbin i'd probably be more productive with a margarita.
14:24 pdurbin
I like your style, Dude.
14:24 * pdurbin
fetches donsizemore a margarita
14:37 andrewSC joined #dataverse
14:45 pdurbin
donsizemore: can you please reminde me about Counter Processor and ~/.dataverse/ec2.env? If I add the right line, will it be installed? Or are you still working on this?
14:52 donsizemore
@pdurbin it's a syntax issue and i'm guilty of not working on it over the weekend, but i'm actively poking at it this morning
14:53 pdurbin
donsizemore: don't work weekends! Thanks! Meanwhile, I should be able to spin up a new EC2 instance and run some command line tomfoolery to get it installed, right? That's my understanding anyway.
14:54 donsizemore
@pdurbin you could rerun the playbook with the "counter" tag
14:54 pdurbin
Ok. Do you have the complete command to run handy?
14:57 donsizemore
the script should spit out most of it, you'd just want to add --tags="counter". gimme a sec to pull the rest from the ec2 script
14:57 pdurbin
ok, thanks
14:58 pdurbin
I'm trying to get this working end to end. Click around in Dataverse, parse the log into JSON . Shove the JSON through a new API into the database.
14:58 donsizemore
the script runs ansible-playbook -v -i dataverse/inventory dataverse/dataverse.pb --connection=local --extra-vars "$extra_vars"
14:59 donsizemore
extra_vars would probably be moot on the 2nd run, then you'd add --tags "counter"
15:00 pdurbin
Ok, so I can leave out the "extra vars" stuff. Also, I saw you're installed counter processor in /usr/local now. Thanks!
15:23 pameyer joined #dataverse
15:26 pdurbin
donsizemore: I just tried the tags thing but it didn't seem to work: https://paste.fedoraproject.org/paste/azNB6APNBhdMK4aPDxcGvw/raw
15:30 pameyer
looks like the role isn't matching the host
15:34 pdurbin
yeah, I was wondering about that output
15:34 pdurbin
/home/centos/dataverse/inventory did not meet host_list requirements, check plugin documentation if this is unexpected
15:34 pdurbin
/home/centos/dataverse/inventory did not meet script requirements, check plugin documentation if this is unexpected
15:48 donsizemore
the host stuff is just a warning. it didn't match the conditional
15:53 pdurbin
Ah ha! I kept hacking until I got it to work.
16:00 pdurbin
Huh. Getting weird errors trying to create a dataverse though: Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://localhost:8983/solr/collection1: Exception writing document id dataverse_3 to the index; possible analysis error.
16:51 pdurbin
But I was dinking around with Solr stuff on that EC2 instance. Not sure what's going on.
17:04 pdurbin
Ok, I reinstalled Solr on that EC2 instance and it's working now. Sorry for the noise.
18:05 donsizemore joined #dataverse
18:06 donsizemore
@pdurbin i'll retrace the solr task against the 4.10 release installation instructions and make sure the needful is being done
18:23 pdurbin
donsizemore: ok, I'm not blocked though. Any news on ec2.env?
18:26 donsizemore joined #dataverse
18:52 Jim__ joined #dataverse
18:53 Jim__
Hey pdurbin - was just reading through the irc log and say discussion on the 9th about silent upload failures...
18:53 pameyer
Jim__ are you seeing those too?
18:54 Jim__
I just recently noticed that my fix to catch errors like proxy timeouts broke when provenance uploads came in.
18:54 * pdurbin
looks at http://irclog.iq.harvard.edu/dataverse/2019-01-09#i_84292
18:54 Jim__
So I'm working on a PR now to bring the error reporting back...
18:55 Jim__
In the meantime - checking any proxy timeouts would be another suggestion
18:55 pdurbin
Jim__: cool, do you want to use https://github.com/IQSS/dataverse/issues/5434 as the issue for the pull request? Or make a new one?
18:56 Jim__
Yeah - I can do that.
18:56 pdurbin
awesome, thanks!
19:15 Jim__
#5466 - not sure if that's all the changes, but I think so (had to fix this in working with Hypothesis uploads, so I may have missed some other code change that's needed.)
19:20 pdurbin
Jim__: thanks! I moved the issue and pull request to code review.
19:22 Jim__
Now if I/we could just figure out why kcondon can't build/run #5049! THe latest is that he's getting a class not found error for an aws error that is in a jar that is in WEB-INF/lib as it should be...
19:23 pdurbin
Very strange. I haven't been looking.
19:24 Jim__
(still don't know why he saw the log conflict error, but I did manage to exclude the log jars coming from duracloud that seems to fix that).
19:27 pdurbin
Yeah, I saw a commit about that.
19:37 yoh joined #dataverse
19:53 Jim__
progress - more than one aws jar with the same class in them - hopefully another exclusion fixes #5049.
19:55 pdurbin
Jim__: yep, I just saw your comment. Fingers crossed.
19:56 Jim__
I know class loaders get upset when files aren't on the right path - I wonder if kcondon somehow has a different load order that is allowing him to catch the conflicts...
19:57 pdurbin
He just wrote "that fixed it".
20:01 pdurbin
Jim__: now something about the database script though.
20:02 pdurbin
And if you're switching to that branch I have a tiny Sphinx tip for you.
20:04 Jim__
go for it...
20:04 pdurbin
We use :doc: to link to other pages.
20:05 pdurbin
Is `make html` working for you?
20:15 pdurbin
Jim__: I have the doc change tested and ready if you want me to push it to your branch.
20:17 Jim__
sure - please push. I was just looking at how to do it so an example would be great
20:20 Jim__
I haven't used make html
20:22 pdurbin
Jim__: sure please look at https://github.com/IQSS/dataverse/commit/8027fe7c3 and it would be nice if you could get `make html` working because the workflows page looks a little funky in your branch.
20:24 pdurbin
I can push a fix for that too if you want, while I'm in here. You just need to add a newline above ".. code:: json"
20:26 Jim__
please do - thanks!
20:27 pdurbin
sure, pushed: https://github.com/IQSS/dataverse/commit/5122fd98d
20:28 pdurbin
Jim__: I'm a little weirded out that upgrade_v4.9.4_to_v4.10.sql shows changes
20:30 Jim__
the command for this PR was in there and was just removed - not sure if there are whitespace changes left?
20:30 pdurbin
Not sure. If it were me I would fix it with something like this: git checkout be044e2 scripts/database/upgrades/upgrade_v4.9.4_to_v4.10.sql
20:31 pdurbin
be044e2 being the latest in develop
20:33 Jim__
looks like unix vs windows line breaks...
20:34 pdurbin
Ah.
20:34 pdurbin
Wait, Dataverse runs on Windows?!?
20:35 Jim__
no - my eclipse editing environment
20:36 Jim__
changed when I added and then removed whats now in the upgrade to v4.11 script
20:36 Jim__
in any case - fixed on github
20:40 pdurbin
Jim__: looks great now. Thanks! Sorry to nitpick. Where do you deploy to? Also Windows?
20:41 Jim__
I push to github and we have jenkins and chef building/deploying to aws...
20:41 pdurbin
No fair. You have all the toys.
20:43 Jim__
QDR had/has good help on that - it's pretty slick. I like the DV setup to build/report through github too. Why when I started program on Waterloo Fortran 4 on punch cards...
20:51 pdurbin
now you sound like pameyer :)
20:52 Jim__
:-)
20:55 pdurbin
Jim__: HTML version of your docs: http://guides.dataverse.org/en/4706-DPN_Submission_of_archival_copies/admin/integrations.html#duracloud-chronopolis
20:55 pameyer
I'm sure you could hook-up fortran to jenkins if you wanted to :) and I know fortran works with docker
21:03 Jim__
I'll just program an arduino to read all the punch cards and we're all set then :-)
21:04 pameyer
and we can 3d-print a card punch too :)
21:06 Jim__
and everyone's worried about being able to run old software... I think we've got it covered.
21:07 Jim__
pdurbin - thanks for pointing to the docs...
21:12 pdurbin
Jim__: sure, Kevin and I were just discussing them. Have a moment? I have about 20 minutes until I need to leave to pick up the kids.
21:15 Jim__
yeah
21:15 pdurbin
So a few things.
21:15 pdurbin
- curl commands have smart quotes so the don't "just work"
21:16 pdurbin
- the "integrations" page is mostly intended to raise awareness and link to other pages such as the config page in the install guide
21:16 pdurbin
Maybe I only have two things. :)
21:18 Jim__
OK - the quotes don't look smart in the source - what do I need to do for that?
21:20 Jim__
And, w.r.t. where to put things - I'm open as to where stuff goes, just need guidance or for you guys to move it as you see fit. The tricky part here is that the archiver mechanism is generic/extensible now and could be used by others.
21:21 Jim__
I could see the Duracloud part going to config, but the archiver mechanism is more developer-level?
21:21 pdurbin
Sounds like a good split to me. Duracloud stuff in the config. Dev stuff in the dev guide.
21:23 Jim__
OK - I can have a go at that.
21:23 pdurbin
I'm just going to fix all these curl examples now if you don't mind. Are you able to get Sphinx installed? There are some out of date instructions at http://guides.dataverse.org/en/4.10.1/developers/documentation.html
21:25 Jim__
OK. I don't have Sphinx, but did find the make.bat file.
21:28 pdurbin
Ok, I got rid of the smart quotes like this: https://github.com/IQSS/dataverse/commit/7b7f4b5ff . On my way out let me ask Derek if he can jump in here with any Windows advice. That's where he primarily uses Sphinx. Also, Kevin wants to know if he should merge or wait for doc changes.
21:31 pdurbin
If you could ping Kevin on the doc stuff I'd appreciate it.
21:32 Jim__
It would be so nice to get this merged, I'd be happy to follow up with a doc PR... :-)
21:32 pdurbin
Sure, that's one way to think of it.
21:32 Jim__
As for Sphinx, it doesn't look hard, just haven't gotten to it...
21:33 pdurbin
sure
21:42 DerekM joined #dataverse
21:47 DerekM
Hey Jim. I installed Sphinx years ago with the instruction of a developer, so I don't remember the details but I don't think it was too much of a hassle. Just gotta get it installed so you can use the command line to navigate to the folder with the make.bat file and then input "make html" to create html previews of all docs
21:51 Jim__
OK - thanks...
