Time
S
Nick
Message
06:03 juancorr joined #dataverse
07:10 jri joined #dataverse
09:07 jri_ joined #dataverse
09:38 ppeter joined #dataverse
09:41 ppeter joined #dataverse
09:43 jri joined #dataverse
09:56 ppeter
Hello, I installed a dataverse test instance and tried to configure shibboleth with EduID, which is a SAML discovery service. The shibboleth configuration seems to work, and I can login manually at /Shibboleth.sso/Login and verify I have a shibboleth session by loading /Shibboleth.sso/Session .
09:59 ppeter
However, the dataverse GUI seems to offer to log in using "Your Institution" with an empty list.
10:01 ppeter
Does anyone know how dataverse is supposed to get the list of the institutions? Do I need to adjust my shibboleth authentication? How can dataverse be configured to use a discovery service?
10:23 gyufi joined #dataverse
10:28 ppeter
it seems the institution selection was solved by configuring the DiscoFeed by adding a correct MetadataProvider.
10:29 ppeter
It still does not recognize the logged-in status, maybe due to bad attribute mapping or because not getting all the required attributes...
11:15 ppeter
I managed to find this in the logs: "The SAML assertion for "Shib-Identity-Provider" was null. Please contact support.|#]"
11:57 pkiraly joined #dataverse
13:06 ppeter
I have found similar error here: http://irclog.iq.harvard.edu/dataverse/2019-12-02
13:07 ppeter
but it refers to missing 'attributePrefix="AJP_"' declaration for cuase, which I do have in my shibboleth config
13:10 ppeter
The /Shibboleth.sso/Session URL shows the "Identity Provider:" , so shibboleth and apache are aware of it, just somehow glassfish does not get it, or not in that variable...
13:10 pkiraly
Hi, DiscoFeed is the default option, but you can configure Dataverse to use the actual loging response to extract the affiliation from it
13:13 nils`` joined #dataverse
13:14 ppeter
I am content with discoFeed now, but would like to know why Shib-Identity-Provider is not received by glassfish
13:14 pkiraly
ppeter, see details here http://guides.dataverse.org/en/latest/installation/config.html?highlight=shibaffiliationattribute#shibaffiliationattribute
13:14 pkiraly
ppeter, I do not have idea for that
13:17 ppeter
how can I enable shibboleth debug mode? The log says "Shibboleth dev mode has not been configured. Returning a sane default: PRODUCTION|#]"
13:17 ppeter
I meant dev mode... :)
13:19 ppeter
It may provide more logging.
13:32 sivoais joined #dataverse
13:32 ppeter94 joined #dataverse
13:39 donsizemore joined #dataverse
13:47 pdurbin
ppeter94: hi, still around?
13:49 ppeter94
yes
13:50 pdurbin
Great. Gimme a couple minutes to finish a reply on the mailing list and I'll see if I can help. The first thing I'll want to know is if you are getting JSON out of the DiscoFeed URL .
13:51 ppeter94
ok, I will look into it
13:52 ppeter94
Most probably yes, because I can choose an IDP and login
13:53 ppeter94
Just glassfish does not seem to get the authentication variables
13:57 ppeter94
debian 10, shibboleth 3.0.4, Apache/2.4.38, glassfish payara-5.201
13:59 ppeter94
the url /Shibboleth.sso/DiscoFeed shows a nice json that seems correct
14:00 pdurbin
Oh, you're farther along than I thought. I must have missed something in the IRC log. Did you add AJP_ to the shib config? I think that's what it's called. I'll go look.
14:01 ppeter94
I can point you to my shib config template, I forked the official ansible repo and adopted it to debian
14:02 ppeter94
if it starts to work, you can also get a pull request :)
14:02 pdurbin
donsizemore: ^^ Debian support!
14:02 donsizemore
@pdurbin Dataverse should run just fine in Debian
14:02 ppeter94
https://github.com/pallinger/dataverse-ansible/blob/master/templates/shibboleth2.xml.j2
14:04 pdurbin
ppeter94: I do see `attributePrefix="AJP_"` in your pull request. Good. Anything in server.log?
14:04 ppeter94
I will paste the generated result to pastebin, if that is helpful.
14:05 ppeter94
yes: The SAML assertion for "Shib-Identity-Provider" was null. Please contact support.|#]
14:06 pdurbin
Hmmm. How does /Session look?
14:06 ppeter94
i can paste the whole log of a login attempt, if you are interested
14:07 pdurbin
this: https://dataverse.example.edu/Shibboleth.sso/Session
14:07 pdurbin
(with your hostname, of course)
14:07 ppeter94
it is good /Shibboleth.sso/Session
14:07 ppeter94
should I paste it here?
14:07 pdurbin
but does it show something like "Identity Provider: https://samltest.id/saml/idp " ?
14:08 pdurbin
Maybe just paste that one line here.
14:10 ppeter94
Identity Provider: https://l-aai.sztaki.hu/idp
14:11 pdurbin
Great, but why wouldn't it be passed to Dataverse? Hmm. Maybe I'll look at the old log you linked.
14:11 pdurbin
Right. Night Owl.
14:11 ppeter94
I am trying to get it work with EduID, which is the Hungarian research IDP federation
14:11 pdurbin
I think that ended up being a config problem.
14:12 ppeter94
Quite possible
14:13 ppeter94
For apache, I am using the https://github.com/pallinger/dataverse-ansible/blob/master/templates/http.proxy.conf.j2
14:13 ppeter94
This apache is then behind a haproxy, which does the https termination and nothing else.
14:13 donsizemore
@pkiraly just seeing your troubleshooting in jenkins. PG deadlocks are the bane of integration tests
14:14 donsizemore
@ppeter94 everything in dataverse-ansible was written for and tested in CentOS. if you're in Debian-land some of those service templates may give you trouble
14:14 pdurbin
ppeter94: that config looks fine to me. nightowl was behind some sort of AWS proxy.
14:15 ppeter94
We have another java application that works with apache+shibboleth+EduID, but that uses tomcat and mod_jk instead of glassfish+mod_proxy_ajp
14:16 pdurbin
lemme dive into our ticketing system and see if there are any clues... she did get it working
14:17 ppeter94
@donsizemore I did end up changing lot of the playbooks. Also had to change a lot of things to have a semblance of idempotence and not always reinstall solr and glassfish.
14:18 ppeter94
@donsizemore I hope the CentOS side will also benefit, if you can merge my changes.
14:18 donsizemore
pull requests are welcome
14:18 pdurbin
ppeter94: she ended up opening (and then closing) https://github.com/IQSS/dataverse/issues/6449 which has some details
14:18 pdurbin
"Just an update on this...the issue was due to my target group in AWS (that the ELB forwarded to) being configured to route on port 80 instead of port 443. I created a new target group that routes https on port 443 and registered the target also on port 443 and shibboleth works as expected."
14:19 ppeter94
thank you, I will look into it
14:19 donsizemore
@ppeter94 i've never aimed for idempotence. while it's "the ansible way," the dataverse installer is itself not idempotent and dataverse upgrades are not consistent in their patterns
14:20 ppeter94
I think that is not the case here: this is no https-behind-https
14:20 pdurbin
ppeter94: sure. Maybe as a troubleshooting step you could remove haproxy.
14:21 ppeter94
however, if you want to maintain your configuration in ansible, then you practically need idempotence. At least it is very murky without it.
14:24 ppeter94
@pdurbin Ok, I will try disabling haproxy and switch apache to https mode.
14:26 donsizemore
@ppeter94 semaphores around solr and postgres are fine, but if you want to re-run the dataverse installer you basically have to blow away glassfish and start over.
14:27 donsizemore
@ppeter94 i've primarily used dataverse-ansible in VMs for testing or for initial installation. after that, each upgrade or configuration change is unique
14:27 ppeter94
That is OK, I will look how I implemented it
14:28 donsizemore
@ppeter94 the above said, it's only my perspective, and pull requests are welcomed =)
14:28 ppeter94
The unique configuration changes make it a headache if there are multiple sysadmins
14:28 pkiraly
donsizemore, Are there any function which creates a "clean state" before running an integration test?
14:28 donsizemore
@pkiraly first they ran in docker, but sluggish stoage manifested as postgres deadlocks
14:29 donsizemore
@pkiraly so the decision was made to run them in AWS, which hopefully had more responsive storage. that was fine for a while. now the integration tests manifest postgres deadlocks again
14:30 donsizemore
@pkiraly if you'd like i can spin up your branch manually and grab you a server.log
14:31 pkiraly
@donsizemore, As I understand docker is set up only once, before the first test. So one test could insert some value into the database which might cause a problem for another test
14:32 pkiraly
@donsizemore, In unit test we have specific functions to clean the state, thus we can assure that each tests are independent, and their values are not dependent of unknown state of the environment
14:33 donsizemore
@pkiraly the current integration test flow starts from a clean slate each time.
14:34 donsizemore
@pkiraly there have been multiple issues opened regarding integration tests failures, usually revolving around postgres deadlocks
14:35 donsizemore
@pkiraly hence my suggestion to ignore those test failures from your PR unless you thought they were directly related to your code
14:36 pkiraly
@donsizemore, Yes, I am ignoring this time. I just would like to learn more about this integration tests. There might be false assumption in a later phase.
14:37 pdurbin
ppeter94: thanks, it would do my heart good to hear that it's working with fewer moving parts. :)
14:39 pkiraly
@donsizemore, I have another question: I see that adding integration tests doesn't increase code coverage numbers. Is there a report, or a plan to get a report on the coverage of these tests?
14:39 donsizemore
@pkiraly you're PR 6921, right? your unit tests succeeded: [WARNING] Tests run: 1117, Failures: 0, Errors: 0, Skipped: 6
14:39 donsizemore
@pkiraly https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/view/change-requests/job/PR-6921/8/
14:40 pkiraly
@donsizemore, yes, that's my PR
14:41 pkiraly
@donsizemore, I have created both unit tests and integration tests
14:41 donsizemore
@pkiraly you can see the console output here: https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/view/change-requests/job/PR-6921/8/console
14:42 donsizemore
and testing and logging output from your most recent build here https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/view/change-requests/job/PR-6921/8/execution/node/3/ws/target/
14:42 pkiraly
@donsizemore, Thanks!
14:43 donsizemore
@pkiraly for the record, i've been whining about PG deadlocks for a good while
14:43 pdurbin
a good long while
14:44 pdurbin
and so have I :)
14:53 pkiraly
@donsizemore, good luck!
14:54 donsizemore
@pkiraly if my pot of all-flash storage ever falls from the sky...
14:58 donsizemore
@ppeter94 there is something of a standard configuration in the dvinstall.zip of each release, but if you browse through the release upgrade instructions they're all different and sometimes installation-dependent
14:58 donsizemore
@ppeter94 i'm only a volunteer and it's never been a priority for me, but i would welcome your PR to see what you've done
16:08 nils`` joined #dataverse
16:57 ppeter94
@pdurbin I tried it with apache, but alas the result is the same
16:58 ppeter94
I had to do a major rework on the ssl.conf for apache to work, it is a bit of a mess now
16:59 ppeter94
so I will need some time to clean it up before I push it
17:00 pdurbin
ppeter94: still not working? Drat.
17:01 pdurbin
ppeter94: a completely different direction to try would be OIDC. It supports Shibboleth/SAML.
17:07 ppeter94
OIDC?
17:08 pdurbin
ppeter94: https://github.com/IQSS/dataverse/issues/6701
17:09 ppeter94
ok, I will look into it, but if shibboleth fails to transfer the variables to glassfish then it is moot anyways
17:09 ppeter94
I have to go, I will call in tomorrow, and also talk with the IDP guys before
17:10 pdurbin
Well, with OIDC, the variables come from the, uh, auth service, that you run.
17:10 pdurbin
It's a different protocol.
17:11 pdurbin
I've never touched it myself, apart from testing the setup from poikilotherm (who contributed the feature).
17:11 pdurbin
ppeter94: another thought is that you could test with samltest or whatever they call it these days.
17:12 donsizemore
@pdurbin @poikilotherm points out: "beware: the Dataverse side is not perfectly ready for this yet."
17:12 pdurbin
this thing https://samltest.id (which we mention in our docs)
17:12 pdurbin
ppeter94: but you're so close to having it working if /Session is looking good.
17:13 donsizemore
@pdurbin i'm sad to report that docker-aio doesn't "just work" with podman-cli
17:13 pdurbin
ppeter94: oh and please don't call in tomorrow. It's the weekend. And it's a holiday. See you Tuesday. :)
17:14 pdurbin
donsizemore: are we gonna end up with a podman-aio?
17:15 donsizemore
@pdurbin doubtful. podmain is preferable because it uses no daemon and doesn't need root permissions
17:15 pdurbin
I just saw a tweet from Dan Walsh that it's supported on Debian now.
17:16 pdurbin
accepted into Debian unstable: https://twitter.com/rhatdan/status/1262471247398612992
17:16 donsizemore
[dls
17:17 donsizemore
which requires sudo, which torpedoes the claim of unprivileged operation
17:18 pdurbin
When I did all that Dataverse on Openshift stuff we had to get the images to run not as root.
17:32 donsizemore
if there's enough interest i could make a podman-aio with payara5 for testing purposes but oliver's already done a ton in dataverse-k8s
17:37 pdurbin
Well, the main users of docker-aio at the moment are pameyer and pkiraly, as far as I know. What's driving your interest in podman? I heard something like Red Hat is going to ship it instead of docker.
17:51 donsizemore
i was mostly interested in its daemon-less root-less operation
17:53 pdurbin
gotcha
17:56 poikilotherm
Just checking in very quickly...
17:56 poikilotherm
Saw your podman talk @donsizemore and @pdurbin
17:56 poikilotherm
IIRC I added a few things about using it to the cloud guide
17:56 donsizemore
@poikilotherm more importantly, how's the house?
17:57 pdurbin
and the blueprints made in inkscape
17:58 donsizemore
there's a cloud guide?
17:58 poikilotherm
There you go https://dataverse-k8s.readthedocs.io/en/v4.19/images/build.html
17:58 donsizemore
ah, the k8s guide
17:59 poikilotherm
The house is still in progress...
18:00 poikilotherm
Made my way with building drywall from the roof to the basement
18:00 poikilotherm
donsizemore I'm writing that guide for a while now...
18:00 donsizemore
oh, i've seen it, i just considered it your k8s docs
18:05 poikilotherm
:-D
18:11 poikilotherm
Driving home now guys... Read you all on Monday.
18:12 pdurbin
Tuesday. Because holiday. :)
18:46 jri joined #dataverse
19:06 ppeter94
@pdurbin /Session was good even through haproxy
19:07 ppeter94
@pdurbin OK, I will try some things on my own on monday
19:08 ppeter94
Have a nice weekend!
19:08 pdurbin
You too!
19:08 pdurbin
I know it was good. You're so close.
20:10 donsizemore joined #dataverse
20:45 pdurbin
Ok, I'm heading out. Have a good long weekend. See you Tuesday.
20:45 pdurbin left #dataverse
