Time
S
Nick
Message
07:54 jri joined #dataverse
09:40 stefankasberger joined #dataverse
11:00 pdurbin
poikilotherm: hi. Did you see https://github.com/IQSS/dataverse-installations/issues/55#issuecomment-591219086 ? :)
11:01 poikilotherm
Good morning pdurbin
11:01 poikilotherm
Yeah, but had no time so far to get going with it
11:01 pdurbin
no worries, please take your time. I tested it. It should "just work". :)
11:03 pdurbin
stefankasberger: how's the upgrade going? Any time to think about https://github.com/IQSS/dataverse/issues/6661 ? :)
11:13 donsizemore joined #dataverse
11:31 donsizemore
@pdurbin yeah I like the JaCoCo report better than the Jenkins plugin report. there's an open github issue about the Jenkins graphs not being clickable, BTW.
11:33 pdurbin
gotcha
11:34 pdurbin
donsizemore: I've been thinking, I'd really like to reply to my "thoughts on automated testing" post at https://groups.google.com/d/msg/dataverse-dev/ISot5k4VjZQ/t-hzPk8tAwAJ to say that we've basically achieved all of the short term goals I laid out.
11:35 pdurbin
At this point we could move on to the last item: "Once it's clear from code coverage reports which features are not being exercised by API tests, prioritize critical features for testing and add APIs to make them testable if necessary.
11:35 pdurbin
Now that the next to last item is basically done, I mean: "Get https://jenkins.dataverse.org to measure code coverage of integration tests using the method documented as "Measuring Coverage of Integration Tests" on the testing page of the dev guide."
11:36 pdurbin
donsizemore: you got all that done in less than a year. It's amazing!
11:46 donsizemore
@pdurbin if i were smarter i could write the tests
11:47 pdurbin
we can get you writing tests if you want :)
11:47 pdurbin
at the very least you can help with the "prioritize critical features for testing" part :)
11:48 donsizemore
speaking for Odum's archivists, in general, they want the UI
11:49 donsizemore
any time we have trouble and I try to help, I suggest the API but they want the UI
11:49 poikilotherm
pdurbin: I think I just found a bug in OAuth2 regarding the username prefill :-(
11:51 pdurbin
donsizemore: yeah, no dvcli for them
11:51 pdurbin
poikilotherm: fix it, fix it
11:54 poikilotherm
pdurbin: I saw your comment on #6679 and replied https://github.com/IQSS/dataverse/issues/6679#issuecomment-591366879
11:55 poikilotherm
Good catch man :-)
11:56 poikilotherm
Do you think this should be a separated issue?
11:58 poikilotherm
"Smallest chunk style"
12:02 pdurbin
poikilotherm: well, I think you and Peter might disagree over if "affiliation" should be editable by the user or controlled by the institution. :)
12:02 poikilotherm
LOL. Easy to avoid. Just make it configurable
12:03 pdurbin
That would be great. I think donsizemore's boss likes that the Shib provider overwrites all the values. That was his take years ago anyway.
12:04 poikilotherm
So is this a "yeah, please split" answer?
12:04 pdurbin
Well, what are the chunks?
12:06 poikilotherm
4 of em. Bugfix preferred_username, Refactor configuration to use proper JSON , Make configurable choices to disallow changes to attributes per provider, Make use of email_verified
12:07 poikilotherm
Choices to disallow changes could be even split up into two issues recursively
12:08 poikilotherm
One for first login, the other one for user detail view
12:08 pdurbin
Would fixing that bug bring you joy? Should we simply start with that?
12:10 pdurbin
Also, could you please leave a comment on Peter's issue about your thoughts on making the auth providers configurable?
12:10 poikilotherm
Huh?
12:10 poikilotherm
Ah you mean 6515?
12:11 poikilotherm
Meh, @pkiraly is not with us today... Easier to discuss here :-D
12:12 poikilotherm
I'm already writing a bug report for the username thingy
12:12 poikilotherm
I'll create another issue for the config thing anyway (this should be refactored...)
12:13 poikilotherm
So maybe I could as well create a fourth :-D
12:14 pdurbin
Yeah, I did mean that issue. I just left a comment on your comment.
12:15 pdurbin
And after all that you'll add your installation to the map? :)
12:22 poikilotherm
https://graydon.law/wp-content/uploads/2018/04/be-prepared-not-just-for-scouts.jpg
12:35 pdurbin
:)
12:59 donsizemore
@pdurbin love these. c.f. https://img.shields.io/jenkins/coverage/jacoco?jobUrl=https%3A%2F%2Fjenkins.dataverse.org%2Fjob%2FIQSS-dataverse-develop&label=Integration%20Test%20Coverage vs. https://github.com/OdumInstitute/dataverse/tree/6682_coverage_button
12:59 donsizemore
sometimes it loads, sometimes it doesn't.
13:03 pdurbin
donsizemore: nice! But for me it says "Integration Test Coverage inaccessible"
13:07 donsizemore
what i mean
13:08 donsizemore
try it now
13:11 donsizemore
yeah, the request isn't coming through shields.io or even github... it's from the client.
13:12 donsizemore
so if we want our dynamic code coverage button we have to leave jenkins exposed to every base64-encoded-payload-that-attempts-to-open-a-shell from every bot on the Net
13:12 donsizemore
that also means that when they manage to knock jenkins over, the README won't load, or at least won't load properly
13:26 pdurbin
Now the button says 41%!!! Perfect!! Ship it!!
13:38 donsizemore joined #dataverse
13:38 donsizemore
@pdurbin it probably won't stay perfect, though. not with jenkins running standalone on a plain ole' VM with no DoS protection, etc. how do you feel about a static button that gets periodically updated when the coverage significantly changes?
13:39 donsizemore
@pdurbin I mean, I _can_ leave it open, there will just be noise from time to time
13:41 Benjamin_Peuch joined #dataverse
13:42 Benjamin_Peuch
Gratings.
14:10 pdurbin
Benjamin_Peuch: hi. I have a CC0 vs CC-BY video for you and Freya and poikilotherm and the rest to watch. :)
14:11 Benjamin_Peuch
Oh, sounds very interesting!
14:11 poikilotherm
Hit me
14:12 Benjamin_Peuch
Hey pdurbin, do you know if the exact changes made to the Citation and Geospatial Metadata TSV files with v4.19 were documented somewhere?
14:12 Benjamin_Peuch
I'm going through the milestones but it's hard to pinpoint.
14:25 dataverse-user joined #dataverse
14:28 pdurbin
Benjamin_Peuch: I can teach you what I do to answer such questions. It involves some clicking around in GitHub.
14:30 Benjamin_Peuch
I'd learn with pleasure. It tried to compare versions but I still have difficulty getting around with GitHub (as you no doubt saw with my latest PR).
14:30 pdurbin
Cool. You want to start with the default branch (develop). And let's do citation.tsv first. Here it is: https://github.com/IQSS/dataverse/blob/develop/scripts/api/data/metadatablocks/citation.tsv
14:31 pdurbin
already we can see "Update Citation.tsv Part Two" but resist the temptation to click that :)
14:31 pdurbin
Please click "History" instead.
14:32 pdurbin
Now start clicking commits from top to bottom until you see more than just "develop" at the top.
14:33 pdurbin
The 4th one down has more than just develop: https://github.com/IQSS/dataverse/commit/c9618d55693ae36b0cf0667df3f14a7632ae5bde#diff-bfa06ddee7dc844bed159a5d62b7504c
14:33 pdurbin
It has three tagged releases:
14:33 pdurbin
v4.19 v4.18.1 v4.18
14:34 pdurbin
So, this means that citation.tsv hasn't changed since 4.18 but will change in the next release which will probably be 4.20.
14:34 pdurbin
Does that make sense?
14:34 pdurbin
It's easier to see it than to describe it. :)
14:44 Benjamin_Peuch
Now it all makes sense!
14:44 Benjamin_Peuch
Thank you so much, pdurbin.
14:44 Benjamin_Peuch
I feel dumb for not looking around just a bit more. I should have spotted that "History" button."
14:44 Benjamin_Peuch
Hard to make it more obvious than that.
14:44 Benjamin_Peuch
And since I know the date of the 4
14:45 Benjamin_Peuch
4.19 release, I know when to stop in the list of commits.
14:56 stefankasberger
@pdurbin: Upgrade is tricky on one side, and easy on the other. Tricky it is, because we have some problem with shibboleth, where we can not find out, what the issue is. The shibd process runs all the time 100% on one core, and grabs away the free memory (RAM ), ~600MB each hour. On the other side, the Dataverse Upgrades themself where really easy. The experimental database upgrade worked out fine.
14:59 stefankasberger
And the best part of it: I have worked out several GH repos for DevOps: 1. dataverse_deployment: A collection of nested shell scripts to to deployment stuff on different instances, controlled by one shell file. With all the html, settings files, api calls ets inside. One repo to rule them all, basically. 2. a Jenkins test repo, with some basic tests for Dataverse in Selenium (with pytest). This is a starting point, from wh
14:59 stefankasberger
ich I will try to generalize the scripts, so it can be adapted easier by others, and 3) a aussda-test-data repo with our first collection of testdata, coming from the pyDataverse templates (dataverses.csv, datasets.csv, datafiles.csv).
15:00 stefankasberger
All of them are still private, cause there are some internal informations in (API Tokens, passwords, customizations), but for all of them we want to go open as soon as possible.
15:00 stefankasberger
but needs some re-factoring for that.
15:01 stefankasberger
And, if everything works out fine with the birth, I will be in Boston in June. :)
15:43 poikilotherm
stefankasberger: I don't know in which federation you joined, but if its edugain, that's pretty normal on startup... It's a long list and Shibboleth sucks.
15:44 poikilotherm
pdurbin: you might be interested in my newest issue... https://github.com/IQSS/dataverse/issues/6694 Especially that JSON schema ;-)
16:13 donsizemore joined #dataverse
16:41 pdurbin
Benjamin_Peuch: sure. I look for tags, not dates but whatever works for you. :)
16:42 stefankasberger joined #dataverse
16:43 pdurbin
stefankasberger: the dataverse_deployment repo sounds great! For shib stuff I recommend the shib users mailing list. They have been very helpful.
16:48 Benjamin_Peuch
Thanks again for the help, pdurbin. :) I've got to run. TTYL.
16:55 poikilotherm
pdurbin: didn't you want to send us a video CC-0 vs CC-BY?
16:56 pdurbin
poikilotherm: I do, I do. I figured I'd comment on the CC0 issue.
16:57 poikilotherm
:-D
17:00 pdurbin
poikilotherm: and yes, the refactoring auth provider issue is interesting. I just left some questions in a comment. Thanks!
17:02 poikilotherm
I just replied as my last action for today
17:02 poikilotherm
I'm off to construction site now :-)
17:02 pdurbin
have fun!
18:14 donsizemore joined #dataverse
19:24 donsizemore
@pdurbin on for 3?
19:28 Jim16 joined #dataverse
19:47 Jim16
@donsizemore - I'm finally trying to run ec2-create-instance and am getting a security group dataverse-sg doesn't exist error. Any ideas? Is there a parameter I need to supply that I'm not?
19:49 donsizemore
@jim16 ah, that's specific to IQSS' AWS space
19:50 donsizemore
@jim16 you'll need to edit line 122 of the create script (and we should make that a configurable flag)
19:50 Jim16
what should I put there?
19:51 donsizemore
you'll just want an AWS security group for the instance (you can call yours dataverse-sg if you want)
19:52 Jim16
Also - I saw the REPO_URL_DEFAULT entry so first tried to not supply a -r REPO_URL, but the default doesn't get used - do you want issues for minor things like this?
19:52 donsizemore
issues are fine, pull requests are great =)
19:53 Jim16
you may not want my script pulls, but yeah
19:53 pdurbin_m joined #dataverse
19:53 donsizemore
@jim16 there are three layers of precedence going on between ansible group_vars and script defaults
19:54 pdurbin_m
donsizemore: yes. On at 3. Thanks!
19:54 Jim16
so with dataverse-sg, since it gives an error rather than creating a group with that name, maybe the check needs a tweak?
19:55 donsizemore
@jim16 sounds like it's not creating it. i've only run the script in AWS' space
19:56 Jim16
OK - I'll investigate - thanks!
19:56 donsizemore
@jim16 (and in my case, that security group pre-existed)
20:01 pdurbin_m
but we're still in sprint planning
20:02 Jim16
@donsizemore - looks like the check at line 123 just kills the script instead of returning a non-zero error code to the if statement. Just commenting things out let's me go forward for now.
21:23 pdurbin
Jim16: need any help with that script? Some blood on my hands with it.
21:24 pdurbin
You only have to create the security group once so we commented it out to make it faster. Lazy to do this. Sorry.
21:24 pdurbin
If memory serves Matthew said it takes a while. Since we all share the same AWS account I never created the security group myself. He created it and the script "just worked" for me.
21:34 pdurbin
donsizemore: I just emailed my old crony about fail2ban. I guess we could also ask in #crimsonfu.
21:44 pdurbin
Ah, he wrote back already with some handy commands from their wiki. They *do* use fail2ban still and recommend it. The other tool I forgot was denyhosts but fail2ban does more. Here are the handy commands:
21:45 pdurbin
- To view a list of IP 's that are in jail, run:
21:45 pdurbin
- # iptables -L -n.
21:45 pdurbin
- To check the status of Fail2ban, run:
21:45 pdurbin
- # fail2ban-client status.
21:45 pdurbin
- To unblock an IP address, run:
21:45 pdurbin
- # fail2ban-client set ssh-iptables unbanip [XX.XX.XX.XX].
21:46 pdurbin
donsizemore: I hope that helps. Again, I'm also cool with other solutions we've talked about but it seems like the immediate problem is bad actors with IP addresses that we can hopefully block.
21:47 pdurbin
If it helps, I can create an issue in https://github.com/IQSS/dataverse-jenkins/issues about this and see if I can get it put into a future sprint so we (probably me) can dedicate some time to it.
21:49 pdurbin
CI is critical these days. Super valuable.
22:31 donsizemore
@pdurbin OK, I installed fail2ban and gave it some basic jails. nginx is only serving as a proxy, so we'll see how much it catches.
22:31 donsizemore
@pdurbin you shall have your dynamically-generated button!
23:38 pdurbin
hooray!
