IQSS logo

IRC log for #dataverse, 2020-05-22

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

All times shown according to UTC.

Time S Nick Message
06:03 juancorr joined #dataverse
07:10 jri joined #dataverse
09:07 jri_ joined #dataverse
09:38 ppeter joined #dataverse
09:41 ppeter joined #dataverse
09:43 jri joined #dataverse
09:56 ppeter Hello, I installed a dataverse test instance and tried to configure shibboleth with EduID, which is a SAML discovery service. The shibboleth configuration seems to work, and I can login manually at /Shibboleth.sso/Login and verify I have a shibboleth session by loading /Shibboleth.sso/Session .
09:59 ppeter However, the dataverse GUI seems to offer to log in using "Your Institution" with an empty list.
10:01 ppeter Does anyone know how dataverse is supposed to get the list of the institutions? Do I need to adjust my shibboleth authentication? How can dataverse be configured to use a discovery service?
10:23 gyufi joined #dataverse
10:28 ppeter it seems the institution selection was solved by configuring the DiscoFeed by adding a correct MetadataProvider.
10:29 ppeter It still does not recognize the logged-in status, maybe due to bad attribute mapping or because not getting all the required attributes...
11:15 ppeter I managed to find this in the logs: "The SAML assertion for "Shib-Identity-Provider" was null. Please contact support.|#]"
11:57 pkiraly joined #dataverse
13:06 ppeter I have found similar error here: http://irclog.iq.harvard.edu/dataverse/2019-12-02
13:07 ppeter but it refers to missing 'attributePrefix="AJP_"' declaration for cuase, which I do have in my shibboleth config
13:10 ppeter The /Shibboleth.sso/Session URL shows the "Identity Provider:" , so shibboleth and apache are aware of it, just somehow glassfish does not get it, or not in that variable...
13:10 pkiraly Hi, DiscoFeed is the default option, but you can configure Dataverse to use the actual loging response to extract the affiliation from it
13:13 nils`` joined #dataverse
13:14 ppeter I am content with discoFeed now, but would like to know why Shib-Identity-Provider is not received by glassfish
13:14 pkiraly ppeter, see details here http://guides.dataverse.org/en/latest/installation/config.html?highlight=shibaffiliationattribute#shibaffiliationattribute
13:14 pkiraly ppeter, I do not have idea for that
13:17 ppeter how can I enable shibboleth debug mode? The log says "Shibboleth dev mode has not been configured. Returning a sane default: PRODUCTION|#]"
13:17 ppeter I meant dev mode... :)
13:19 ppeter It may provide more logging.
13:32 sivoais joined #dataverse
13:32 ppeter94 joined #dataverse
13:39 donsizemore joined #dataverse
13:47 pdurbin ppeter94: hi, still around?
13:49 ppeter94 yes
13:50 pdurbin Great. Gimme a couple minutes to finish a reply on the mailing list and I'll see if I can help. The first thing I'll want to know is if you are getting JSON out of the DiscoFeed URL.
13:51 ppeter94 ok, I will look into it
13:52 ppeter94 Most probably yes, because I can choose an IDP and login
13:53 ppeter94 Just glassfish does not seem to get the authentication variables
13:57 ppeter94 debian 10, shibboleth 3.0.4, Apache/2.4.38, glassfish payara-5.201
13:59 ppeter94 the url /Shibboleth.sso/DiscoFeed shows a nice json that seems correct
14:00 pdurbin Oh, you're farther along than I thought. I must have missed something in the IRC log. Did you add AJP_ to the shib config? I think that's what it's called. I'll go look.
14:01 ppeter94 I can point you to my shib config template, I forked the official ansible repo and adopted it to debian
14:02 ppeter94 if it starts to work, you can also get a pull request :)
14:02 pdurbin donsizemore: ^^ Debian support!
14:02 donsizemore @pdurbin Dataverse should run just fine in Debian
14:02 ppeter94 https://github.com/pallinger/dataverse-ansible/blob/master/templates/shibboleth2.xml.j2
14:04 pdurbin ppeter94: I do see `attributePrefix="AJP_"` in your pull request. Good. Anything in server.log?
14:04 ppeter94 I will paste the generated result to pastebin, if that is helpful.
14:05 ppeter94 yes:  The SAML assertion for "Shib-Identity-Provider" was null. Please contact support.|#]
14:06 pdurbin Hmmm. How does /Session look?
14:06 ppeter94 i can paste the whole log of a login attempt, if you are interested
14:07 pdurbin this: https://dataverse.example.edu/Shibboleth.sso/Session
14:07 pdurbin (with your hostname, of course)
14:07 ppeter94 it is good /Shibboleth.sso/Session
14:07 ppeter94 should I paste it here?
14:07 pdurbin but does it show something like "Identity Provider: https://samltest.id/saml/idp" ?
14:08 pdurbin Maybe just paste that one line here.
14:10 ppeter94 Identity Provider: https://l-aai.sztaki.hu/idp
14:11 pdurbin Great, but why wouldn't it be passed to Dataverse? Hmm. Maybe I'll look at the old log you linked.
14:11 pdurbin Right. Night Owl.
14:11 ppeter94 I am trying to get it work with EduID, which is the Hungarian research IDP federation
14:11 pdurbin I think that ended up being a config problem.
14:12 ppeter94 Quite possible
14:13 ppeter94 For apache, I am using the https://github.com/pallinger/dataverse-ansible/blob/master/templates/http.proxy.conf.j2
14:13 ppeter94 This apache is then behind a haproxy, which does the https termination and nothing else.
14:13 donsizemore @pkiraly just seeing your troubleshooting in jenkins. PG deadlocks are the bane of integration tests
14:14 donsizemore @ppeter94 everything in dataverse-ansible was written for and tested in CentOS. if you're in Debian-land some of those service templates may give you trouble
14:14 pdurbin ppeter94: that config looks fine to me. nightowl was behind some sort of AWS proxy.
14:15 ppeter94 We have another java application that works with apache+shibboleth+EduID, but that uses tomcat and mod_jk instead of glassfish+mod_proxy_ajp
14:16 pdurbin lemme dive into our ticketing system and see if there are any clues... she did get it working
14:17 ppeter94 @donsizemore I did end up changing lot of the playbooks. Also had to change a lot of things to have a semblance of idempotence and not always reinstall solr and glassfish.
14:18 ppeter94 @donsizemore I hope the CentOS side will also benefit, if you can merge my changes.
14:18 donsizemore pull requests are welcome
14:18 pdurbin ppeter94: she ended up opening (and then closing) https://github.com/IQSS/dataverse/issues/6449 which has some details
14:18 pdurbin "Just an update on this...the issue was due to my target group in AWS (that the ELB forwarded to) being configured to route on port 80 instead of port 443. I created a new target group that routes https on port 443 and registered the target also on port 443 and shibboleth works as expected."
14:19 ppeter94 thank you, I will look into it
14:19 donsizemore @ppeter94 i've never aimed for idempotence. while it's "the ansible way," the dataverse installer is itself not idempotent and dataverse upgrades are not consistent in their patterns
14:20 ppeter94 I think that is not the case here: this is no https-behind-https
14:20 pdurbin ppeter94: sure. Maybe as a troubleshooting step you could remove haproxy.
14:21 ppeter94 however, if you want to maintain your configuration in ansible, then you practically need idempotence. At least it is very murky without it.
14:24 ppeter94 @pdurbin Ok, I will try disabling haproxy and switch apache to https mode.
14:26 donsizemore @ppeter94 semaphores around solr and postgres are fine, but if you want to re-run the dataverse installer you basically have to blow away glassfish and start over.
14:27 donsizemore @ppeter94 i've primarily used dataverse-ansible in VMs for testing or for initial installation. after that, each upgrade or configuration change is unique
14:27 ppeter94 That is OK, I will look how I implemented it
14:28 donsizemore @ppeter94 the above said, it's only my perspective, and pull requests are welcomed =)
14:28 ppeter94 The unique configuration changes make it a headache if there are multiple sysadmins
14:28 pkiraly donsizemore, Are there any function which creates a "clean state" before running an integration test?
14:28 donsizemore @pkiraly first they ran in docker, but sluggish stoage manifested as postgres deadlocks
14:29 donsizemore @pkiraly so the decision was made to run them in AWS, which hopefully had more responsive storage. that was fine for a while. now the integration tests manifest postgres deadlocks again
14:30 donsizemore @pkiraly if you'd like i can spin up your branch manually and grab you a server.log
14:31 pkiraly @donsizemore, As I understand docker is set up only once, before the first test. So one test could insert some value into the database which might cause a problem for another test
14:32 pkiraly @donsizemore, In unit test we have specific functions to clean the state, thus we can assure that each tests are independent, and their values are not dependent of unknown state of the environment
14:33 donsizemore @pkiraly the current integration test flow starts from a clean slate each time.
14:34 donsizemore @pkiraly there have been multiple issues opened regarding integration tests failures, usually revolving around postgres deadlocks
14:35 donsizemore @pkiraly hence my suggestion to ignore those test failures from your PR unless you thought they were directly related to your code
14:36 pkiraly @donsizemore, Yes, I am ignoring this time. I just would like to learn more about this integration tests. There might be false assumption in a later phase.
14:37 pdurbin ppeter94: thanks, it would do my heart good to hear that it's working with fewer moving parts. :)
14:39 pkiraly @donsizemore, I have another question: I see that adding integration tests doesn't increase code coverage numbers. Is there a report, or a plan to get a report on the coverage of these tests?
14:39 donsizemore @pkiraly you're PR 6921, right? your unit tests succeeded: [WARNING] Tests run: 1117, Failures: 0, Errors: 0, Skipped: 6
14:39 donsizemore @pkiraly https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/view/change-requests/job/PR-6921/8/
14:40 pkiraly @donsizemore, yes, that's my PR
14:41 pkiraly @donsizemore, I have created both unit tests and integration tests
14:41 donsizemore @pkiraly you can see the console output here: https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/view/change-requests/job/PR-6921/8/console
14:42 donsizemore and testing and logging output from your most recent build here https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/view/change-requests/job/PR-6921/8/execution/node/3/ws/target/
14:42 pkiraly @donsizemore, Thanks!
14:43 donsizemore @pkiraly for the record, i've been whining about PG deadlocks for a good while
14:43 pdurbin a good long while
14:44 pdurbin and so have I :)
14:53 pkiraly @donsizemore, good luck!
14:54 donsizemore @pkiraly if my pot of all-flash storage ever falls from the sky...
14:58 donsizemore @ppeter94 there is something of a standard configuration in the dvinstall.zip of each release, but if you browse through the release upgrade instructions they're all different and sometimes installation-dependent
14:58 donsizemore @ppeter94 i'm only a volunteer and it's never been a priority for me, but i would welcome your PR to see what you've done
16:08 nils`` joined #dataverse
16:57 ppeter94 @pdurbin I tried it with apache, but alas the result is the same
16:58 ppeter94 I had to do a major rework on the ssl.conf for apache to work, it is a bit of a mess now
16:59 ppeter94 so I will need some time to clean it up before I push it
17:00 pdurbin ppeter94: still not working? Drat.
17:01 pdurbin ppeter94: a completely different direction to try would be OIDC. It supports Shibboleth/SAML.
17:07 ppeter94 OIDC?
17:08 pdurbin ppeter94: https://github.com/IQSS/dataverse/issues/6701
17:09 ppeter94 ok, I will look into it, but if shibboleth fails to transfer the variables to glassfish then it is moot anyways
17:09 ppeter94 I have to go, I will call in tomorrow, and also talk with the IDP guys before
17:10 pdurbin Well, with OIDC, the variables come from the, uh, auth service, that you run.
17:10 pdurbin It's a different protocol.
17:11 pdurbin I've never touched it myself, apart from testing the setup from poikilotherm (who contributed the feature).
17:11 pdurbin ppeter94: another thought is that you could test with samltest or whatever they call it these days.
17:12 donsizemore @pdurbin @poikilotherm points out: "beware: the Dataverse side is not perfectly ready for this yet."
17:12 pdurbin this thing https://samltest.id (which we mention in our docs)
17:12 pdurbin ppeter94: but you're so close to having it working if /Session is looking good.
17:13 donsizemore @pdurbin i'm sad to report that docker-aio doesn't "just work" with podman-cli
17:13 pdurbin ppeter94: oh and please don't call in tomorrow. It's the weekend. And it's a holiday. See you Tuesday. :)
17:14 pdurbin donsizemore: are we gonna end up with a podman-aio?
17:15 donsizemore @pdurbin doubtful. podmain is preferable because it uses no daemon and doesn't need root permissions
17:15 pdurbin I just saw a tweet from Dan Walsh that it's supported on Debian now.
17:16 pdurbin accepted into Debian unstable: https://twitter.com/rhatdan/status/1262471247398612992
17:16 donsizemore [dls@irss-ovirt02 docker-aio]$ podman info --debug cannot clone: Invalid argument user namespaces are not enabled in /proc/sys/user/max_user_namespaces Error: could not get runtime: cannot re-exec process
17:17 donsizemore which requires sudo, which torpedoes the claim of unprivileged operation
17:18 pdurbin When I did all that Dataverse on Openshift stuff we had to get the images to run not as root.
17:32 donsizemore if there's enough interest i could make a podman-aio with payara5 for testing purposes but oliver's already done a ton in dataverse-k8s
17:37 pdurbin Well, the main users of docker-aio at the moment are pameyer and pkiraly, as far as I know. What's driving your interest in podman? I heard something like Red Hat is going to ship it instead of docker.
17:51 donsizemore i was mostly interested in its daemon-less root-less operation
17:53 pdurbin gotcha
17:56 poikilotherm Just checking in very quickly...
17:56 poikilotherm Saw your podman talk @donsizemore and @pdurbin
17:56 poikilotherm IIRC I added a few things about using it to the cloud guide
17:56 donsizemore @poikilotherm more importantly, how's the house?
17:57 pdurbin and the blueprints made in inkscape
17:58 donsizemore there's a cloud guide?
17:58 poikilotherm There you go https://dataverse-k8s.readthedocs.io/en/v4.19/images/build.html
17:58 donsizemore ah, the k8s guide
17:59 poikilotherm The house is still in progress...
18:00 poikilotherm Made my way with building drywall from the roof to the basement
18:00 poikilotherm donsizemore I'm writing that guide for a while now...
18:00 donsizemore oh, i've seen it, i just considered it your k8s docs
18:05 poikilotherm :-D
18:11 poikilotherm Driving home now guys... Read you all on Monday.
18:12 pdurbin Tuesday. Because holiday. :)
18:46 jri joined #dataverse
19:06 ppeter94 @pdurbin /Session was good even through haproxy
19:07 ppeter94 @pdurbin OK, I will try some things on my own on monday
19:08 ppeter94 Have a nice weekend!
19:08 pdurbin You too!
19:08 pdurbin I know it was good. You're so close.
20:10 donsizemore joined #dataverse
20:45 pdurbin Ok, I'm heading out. Have a good long weekend. See you Tuesday.
20:45 pdurbin left #dataverse

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.