IQSS logo

IRC log for #dataverse, 2020-02-26

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

All times shown according to UTC.

Time S Nick Message
07:54 jri joined #dataverse
09:40 stefankasberger joined #dataverse
11:00 pdurbin poikilotherm: hi. Did you see https://github.com/IQSS/dataverse-installations/issues/55#issuecomment-591219086 ? :)
11:01 poikilotherm Good morning pdurbin
11:01 poikilotherm Yeah, but had no time so far to get going with it
11:01 pdurbin no worries, please take your time. I tested it. It should "just work". :)
11:03 pdurbin stefankasberger: how's the upgrade going? Any time to think about https://github.com/IQSS/dataverse/issues/6661 ? :)
11:13 donsizemore joined #dataverse
11:31 donsizemore @pdurbin yeah I like the JaCoCo report better than the Jenkins plugin report. there's an open github issue about the Jenkins graphs not being clickable, BTW.
11:33 pdurbin gotcha
11:34 pdurbin donsizemore: I've been thinking, I'd really like to reply to my "thoughts on automated testing" post at https://groups.google.com/d/msg/dataverse-dev/ISot5k4VjZQ/t-hzPk8tAwAJ to say that we've basically achieved all of the short term goals I laid out.
11:35 pdurbin At this point we could move on to the last item: "Once it's clear from code coverage reports which features are not being exercised by API tests, prioritize critical features for testing and add APIs to make them testable if necessary.
11:35 pdurbin Now that the next to last item is basically done, I mean: "Get https://jenkins.dataverse.org to measure code coverage of integration tests using the method  documented as "Measuring Coverage of Integration Tests" on the testing page of the dev guide."
11:36 pdurbin donsizemore: you got all that done in less than a year. It's amazing!
11:46 donsizemore @pdurbin if i were smarter i could write the tests
11:47 pdurbin we can get you writing tests if you want :)
11:47 pdurbin at the very least you can help with the "prioritize critical features for testing" part :)
11:48 donsizemore speaking for Odum's archivists, in general, they want the UI
11:49 donsizemore any time we have trouble and I try to help, I suggest the API but they want the UI
11:49 poikilotherm pdurbin: I think I just found a bug in OAuth2 regarding the username prefill :-(
11:51 pdurbin donsizemore: yeah, no dvcli for them
11:51 pdurbin poikilotherm: fix it, fix it
11:54 poikilotherm pdurbin: I saw your comment on #6679 and replied https://github.com/IQSS/dataverse/issues/6679#issuecomment-591366879
11:55 poikilotherm Good catch man :-)
11:56 poikilotherm Do you think this should be a separated issue?
11:58 poikilotherm "Smallest chunk style"
12:02 pdurbin poikilotherm: well, I think you and Peter might disagree over if "affiliation" should be editable by the user or controlled by the institution. :)
12:02 poikilotherm LOL. Easy to avoid. Just make it configurable
12:03 pdurbin That would be great. I think donsizemore's boss likes that the Shib provider overwrites all the values. That was his take years ago anyway.
12:04 poikilotherm So is this a "yeah, please split" answer?
12:04 pdurbin Well, what are the chunks?
12:06 poikilotherm 4 of em. Bugfix preferred_username, Refactor configuration to use proper JSON, Make configurable choices to disallow changes to attributes per provider, Make use of email_verified
12:07 poikilotherm Choices to disallow changes could be even split up into two issues recursively
12:08 poikilotherm One for first login, the other one for user detail view
12:08 pdurbin Would fixing that bug bring you joy? Should we simply start with that?
12:10 pdurbin Also, could you please leave a comment on Peter's issue about your thoughts on making the auth providers configurable?
12:10 poikilotherm Huh?
12:10 poikilotherm Ah you mean 6515?
12:11 poikilotherm Meh, @pkiraly is not with us today... Easier to discuss here :-D
12:12 poikilotherm I'm already writing a bug report for the username thingy
12:12 poikilotherm I'll create another issue for the config thing anyway (this should be refactored...)
12:13 poikilotherm So maybe I could as well create a fourth :-D
12:14 pdurbin Yeah, I did mean that issue. I just left a comment on your comment.
12:15 pdurbin And after all that you'll add your installation to the map? :)
12:22 poikilotherm https://graydon.law/wp-content/uploads/2018/04/be-prepared-not-just-for-scouts.jpg
12:35 pdurbin :)
12:59 donsizemore @pdurbin love these. c.f. https://img.shields.io/jenkins/coverage/jacoco?jobUrl=https%3A%2F%2Fjenkins.dataverse.org%2Fjob%2FIQSS-dataverse-develop&label=Integration%20Test%20Coverage vs. https://github.com/OdumInstitute/dataverse/tree/6682_coverage_button
12:59 donsizemore sometimes it loads, sometimes it doesn't.
13:03 pdurbin donsizemore: nice! But for me it says "Integration Test Coverage inaccessible"
13:07 donsizemore what i mean
13:08 donsizemore try it now
13:11 donsizemore yeah, the request isn't coming through shields.io or even github... it's from the client.
13:12 donsizemore so if we want our dynamic code coverage button we have to leave jenkins exposed to every base64-encoded-payload-tha​t-attempts-to-open-a-shell from every bot on the Net
13:12 donsizemore that also means that when they manage to knock jenkins over, the README won't load, or at least won't load properly
13:26 pdurbin Now the button says 41%!!! Perfect!! Ship it!!
13:38 donsizemore joined #dataverse
13:38 donsizemore @pdurbin it probably won't stay perfect, though. not with jenkins running standalone on a plain ole' VM with no DoS protection, etc. how do you feel about a static button that gets periodically updated when the coverage significantly changes?
13:39 donsizemore @pdurbin I mean, I _can_ leave it open, there will just be noise from time to time
13:41 Benjamin_Peuch joined #dataverse
13:42 Benjamin_Peuch Gratings.
14:10 pdurbin Benjamin_Peuch: hi. I have a CC0 vs CC-BY video for you and Freya and poikilotherm and the rest to watch. :)
14:11 Benjamin_Peuch Oh, sounds very interesting!
14:11 poikilotherm Hit me
14:12 Benjamin_Peuch Hey pdurbin, do you know if the exact changes made to the Citation and Geospatial Metadata TSV files with v4.19 were documented somewhere?
14:12 Benjamin_Peuch I'm going through the milestones but it's hard to pinpoint.
14:25 dataverse-user joined #dataverse
14:28 pdurbin Benjamin_Peuch: I can teach you what I do to answer such questions. It involves some clicking around in GitHub.
14:30 Benjamin_Peuch I'd learn with pleasure. It tried to compare versions but I still have difficulty getting around with GitHub (as you no doubt saw with my latest PR).
14:30 pdurbin Cool. You want to start with the default branch (develop). And let's do citation.tsv first. Here it is: https://github.com/IQSS/dataverse/blob/develop/scripts/api/data/metadatablocks/citation.tsv
14:31 pdurbin already we can see "Update Citation.tsv Part Two" but resist the temptation to click that :)
14:31 pdurbin Please click "History" instead.
14:32 pdurbin Now start clicking commits from top to bottom until you see more than just "develop" at the top.
14:33 pdurbin The 4th one down has more than just develop: https://github.com/IQSS/dataverse/commit/c9618d55693ae36b0cf0667df3f14a7632ae5bde#diff-bfa06ddee7dc844bed159a5d62b7504c
14:33 pdurbin It has three tagged releases:
14:33 pdurbin v4.19 v4.18.1 v4.18
14:34 pdurbin So, this means that citation.tsv hasn't changed since 4.18 but will change in the next release which will probably be 4.20.
14:34 pdurbin Does that make sense?
14:34 pdurbin It's easier to see it than to describe it. :)
14:44 Benjamin_Peuch Now it all makes sense!
14:44 Benjamin_Peuch Thank you so much, pdurbin.
14:44 Benjamin_Peuch I feel dumb for not looking around just a bit more. I should have spotted that "History" button."
14:44 Benjamin_Peuch Hard to make it more obvious than that.
14:44 Benjamin_Peuch And since I know the date of the 4
14:45 Benjamin_Peuch 4.19 release, I know when to stop in the list of commits.
14:56 stefankasberger @pdurbin: Upgrade is tricky on one side, and easy on the other. Tricky it is, because we have some problem with shibboleth, where we can not find out, what the issue is. The shibd process runs all the time 100% on one core, and grabs away the free memory (RAM), ~600MB each hour. On the other side, the Dataverse Upgrades themself where really easy. The experimental database upgrade worked out fine.
14:59 stefankasberger And the best part of it: I have worked out several GH repos for DevOps: 1. dataverse_deployment: A collection of nested shell scripts to to deployment stuff on different instances, controlled by one shell file. With all the html, settings files, api calls ets inside. One repo to rule them all, basically. 2. a Jenkins test repo, with some basic tests for Dataverse in Selenium (with pytest). This is a starting point, from wh
14:59 stefankasberger ich I will try to generalize the scripts, so it can be adapted easier by others, and 3) a aussda-test-data repo with our first collection of testdata, coming from the pyDataverse templates (dataverses.csv, datasets.csv, datafiles.csv).
15:00 stefankasberger All of them are still private, cause there are some internal informations in (API Tokens, passwords, customizations), but for all of them we want to go open as soon as possible.
15:00 stefankasberger but needs some re-factoring for that.
15:01 stefankasberger And, if everything works out fine with the birth, I will be in Boston in June. :)
15:43 poikilotherm stefankasberger: I don't know in which federation you joined, but if its edugain, that's pretty normal on startup... It's a long list and Shibboleth sucks.
15:44 poikilotherm pdurbin: you might be interested in my newest issue... https://github.com/IQSS/dataverse/issues/6694 Especially that JSON schema ;-)
16:13 donsizemore joined #dataverse
16:41 pdurbin Benjamin_Peuch: sure. I look for tags, not dates but whatever works for you. :)
16:42 stefankasberger joined #dataverse
16:43 pdurbin stefankasberger: the dataverse_deployment repo sounds great! For shib stuff I recommend the shib users mailing list. They have been very helpful.
16:48 Benjamin_Peuch Thanks again for the help, pdurbin. :) I've got to run. TTYL.
16:55 poikilotherm pdurbin: didn't you want to send us a video CC-0 vs CC-BY?
16:56 pdurbin poikilotherm: I do, I do. I figured I'd comment on the CC0 issue.
16:57 poikilotherm :-D
17:00 pdurbin poikilotherm: and yes, the refactoring auth provider issue is interesting. I just left some questions in a comment. Thanks!
17:02 poikilotherm I just replied as my last action for today
17:02 poikilotherm I'm off to construction site now :-)
17:02 pdurbin have fun!
18:14 donsizemore joined #dataverse
19:24 donsizemore @pdurbin on for 3?
19:28 Jim16 joined #dataverse
19:47 Jim16 @donsizemore - I'm finally trying to run ec2-create-instance and am getting a security group dataverse-sg doesn't exist error. Any ideas? Is there a parameter I need to supply that I'm not?
19:49 donsizemore @jim16 ah, that's specific to IQSS' AWS space
19:50 donsizemore @jim16 you'll need to edit line 122 of the create script (and we should make that a configurable flag)
19:50 Jim16 what should I put there?
19:51 donsizemore you'll just want an AWS security group for the instance (you can call yours dataverse-sg if you want)
19:52 Jim16 Also - I saw the REPO_URL_DEFAULT entry so first tried to not supply a -r REPO_URL, but the default doesn't get used - do you want issues for minor things like this?
19:52 donsizemore issues are fine, pull requests are great =)
19:53 Jim16 you may not want my script pulls, but yeah
19:53 pdurbin_m joined #dataverse
19:53 donsizemore @jim16 there are three layers of precedence going on between ansible group_vars and script defaults
19:54 pdurbin_m donsizemore: yes. On at 3. Thanks!
19:54 Jim16 so with dataverse-sg, since it gives an error rather than creating a group with that name, maybe the check needs a tweak?
19:55 donsizemore @jim16 sounds like it's not creating it. i've only run the script in AWS' space
19:56 Jim16 OK - I'll investigate - thanks!
19:56 donsizemore @jim16 (and in my case, that security group pre-existed)
20:01 pdurbin_m but we're still in sprint planning
20:02 Jim16 @donsizemore - looks like the check at line 123 just kills the script instead of returning a non-zero error code to the if statement. Just commenting things out let's me go forward for now.
21:23 pdurbin Jim16: need any help with that script? Some blood on my hands with it.
21:24 pdurbin You only have to create the security group once so we commented it out to make it faster. Lazy to do this. Sorry.
21:24 pdurbin If memory serves Matthew said it takes a while. Since we all share the same AWS account I never created the security group myself. He created it and the script "just worked" for me.
21:34 pdurbin donsizemore: I just emailed my old crony about fail2ban. I guess we could also ask in #crimsonfu.
21:44 pdurbin Ah, he wrote back already with some handy commands from their wiki. They *do* use fail2ban still and recommend it. The other tool I forgot was denyhosts but fail2ban does more. Here are the handy commands:
21:45 pdurbin - To view a list of IP's that are in jail, run:
21:45 pdurbin -  # iptables -L -n.
21:45 pdurbin - To check the status of Fail2ban, run:
21:45 pdurbin -  # fail2ban-client status.
21:45 pdurbin - To unblock an IP address, run:
21:45 pdurbin - # fail2ban-client set ssh-iptables unbanip [XX.XX.XX.XX].
21:46 pdurbin donsizemore: I hope that helps. Again, I'm also cool with other solutions we've talked about but it seems like the immediate problem is bad actors with IP addresses that we can hopefully block.
21:47 pdurbin If it helps, I can create an issue in https://github.com/IQSS/dataverse-jenkins/issues about this and see if I can get it put into a future sprint so we (probably me) can dedicate some time to it.
21:49 pdurbin CI is critical these days. Super valuable.
22:31 donsizemore @pdurbin OK, I installed fail2ban and gave it some basic jails. nginx is only serving as a proxy, so we'll see how much it catches.
22:31 donsizemore @pdurbin you shall have your dynamically-generated button!
23:38 pdurbin hooray!

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.