Time
S
Nick
Message
00:06 foobarbecue joined #dataverse
00:12 foobarbecue
hey, is there an easy way for me to make the demo-k3s instance listen to an external IP ? I know, it's not for deployment... at the moment I get a 404 on everything other than localhost (even 127.0.0.1)
02:00 foobarbecue joined #dataverse
04:44 foobarbecue joined #dataverse
07:32 Virgile joined #dataverse
12:18 donsizemore joined #dataverse
15:08 nightowl313 joined #dataverse
15:22 pdurbin joined #dataverse
15:23 pdurbin
poikilotherm: something about jacoco?
15:35 nightowl313
thanks to everyone who helped me with permissions/roles yesterday ... i spent hours testing different scenarios, and I think I actually understand them now! Have to give a little training on them this morning ... I created a diagram for the various steps if anyone wants to see it! =)
15:35 pdurbin
nightowl313: I'd love to see the diagram.
15:36 nightowl313
i'll send it to you ... let me know if you see any flaws! lol
15:37 pdurbin
I'll do my best.
15:37 poikilotherm
T_T for some reason I don't get the messages anymore.
15:38 poikilotherm
pdurbin I saw your message on iqlogbot
15:38 pdurbin
poikilotherm: what client are you using?
15:38 poikilotherm
It was just me not using JDK 8 after upgrading my Fedora box today.
15:38 pdurbin
ah, ok
15:39 pkiraly joined #dataverse
15:39 poikilotherm
pdurbin: Element, the standard Matrix client. Using the Matrix bridge to IRC . Meh.
15:40 poikilotherm left #dataverse
15:40 pdurbin
I keep meaning to try Matrix. It's what Mozilla switched to recently (from IRC ).
15:42 pkiraly
Hi, I try to install TwoRaven to a CentOS 7 which installs R 3.6. The problem is that the TwoRaven built on top of R modules. Some of them does not have a version for R 3.6, only 3.4 and 3.5. I never worked before with CentOS. Do you know a workaround?
15:44 poikilotherm joined #dataverse
15:45 pdurbin
pkiraly: hi! It's increasingly hard for us to support TwoRavens. Also, it's being rewritten. Did you see the talk by James Honaker about the new TwoRavens during the external tools session of the 2020 community meeting?
15:46 pkiraly
I did, but I do not remember ;-( I will rewatch it
15:47 pkiraly
@pdurbin, Thanks for the tip!
15:47 pdurbin
pkiraly: here you go: https://youtu.be/YH4I_kldmGI?t=1946
15:49 pkiraly
Great!
15:52 pdurbin
Basically, I wouldn't want you to spend a lot of time getting the old version of TwoRavens to work when there's a new version on the horizon. They would probably be happy to help you get the new version installed.
15:52 poikilotherm
Messages start to fly in again :-/
15:53 poikilotherm
Still some are lacking. Maybe the bridge is under heavy load :-(
15:53 pkiraly
Do you know they have an ETA for the release?
15:54 poiki joined #dataverse
15:54 pdurbin
pkiraly: I don't know. You should ask them.
15:54 pkiraly
OK
16:15 poiki
pdurbin I start to like https://github.com/anchore/grype :-)
16:16 pdurbin
I saw your slack.
16:18 pkiraly
pdurbin, I wrote to James, and meantime I found that the active development happens at https://github.com/TwoRavens/TwoRavens not on IQSS/TwoRavens which is referenced from the Dataverse documentation.
16:21 pdurbin
pkiraly: hmm. We should probably mention this in the guides. Would you be able to make a pull request? I'm thinking just a line that says "Please note that the next version of TwoRavens is being developed at [link]"
16:23 pkiraly
It is not a problem at this time, since this repo doesn't have a stable release. Once it will be published, we should update Dataverse documentation.
16:24 pkiraly
I can add that line of course, no problem. I just want to wait James' reply.
16:26 pdurbin
That makes sense. Meanwhile, did you see that Data Explorer 2 is available for beta testing?
16:30 pdurbin
Here was the announcement about Data Explorer 2: https://groups.google.com/g/dataverse-community/c/BvxsNdh6AIg/m/UE9UpfiDAgAJ
16:30 pkiraly
No I didn't. I have a backlog of Dataverse news...
16:31 pdurbin
good :)
17:47 foobarbecue joined #dataverse
17:48 foobarbecue
Allmost got a working setup based on prod-skel env1. Just need to get my dataverse-certificate secret set up. Is there an example of this?
17:48 foobarbecue
https://github.com/GlobalDataverseCommunityConsortium/dataverse-kubernetes/blob/8051b39c28967c0fe469aac7e1b57940e0dd2c67/personas/prod-skel/envs/env1/webapp/sidecars/ssl/patch.yaml refers to it but I don't see an example anywhere of how to set up that secret
17:49 foobarbecue
er, https://github.com/GlobalDataverseCommunityConsortium/dataverse-kubernetes/blob/develop/personas/prod-skel/envs/env1/webapp/sidecars/ssl/patch.yaml
17:52 foobarbecue
poikilotherm, you around?
17:56 Virgile joined #dataverse
18:01 poiki
Hi foobarbecue
18:02 foobarbecue
hi there! I was just asking: Allmost got a working setup based on prod-skel env1. Just need to get my dataverse-certificate secret set up. Is there an example of this? https://github.com/GlobalDataverseCommunityConsortium/dataverse-kubernetes/blob/develop/personas/prod-skel/envs/env1/webapp/sidecars/ssl/patch.yaml refers to it but I don't see an
18:02 foobarbecue
example anywhere of how to set up that secret
18:02 poiki
Kubernetes handles TLS certificates in Secrets as well as simple text values.
18:02 poiki
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/
18:04 poiki
So you'll need to create a secret with that name if you wanna use it.
18:04 poiki
Dunno if Let's Encrypt is an option for you...
18:04 foobarbecue
we already have .crt files
18:04 poiki
Traefik has builtin support for that
18:04 poiki
Ah nice
18:05 foobarbecue
ok great thanks, this guide looks like it'll get me going
18:05 poiki
Whooooops
18:05 poiki
Non it won't
18:05 foobarbecue
oh lol ok
18:05 poiki
Sorry I just gave you the wrong docs
18:05 poiki
That#s about the K8s internal CA
18:05 poiki
Intercluster communicatio
18:05 foobarbecue
ah
18:06 poiki
There you go https://kubernetes.io/docs/concepts/configuration/secret/#tls-secrets
18:07 poiki
I created dvcli-k8s to help with this. We don't have a Vault or other secret storage around
18:07 poiki
But we use Keepass
18:07 poiki
IIRC I wrote a line or two about that in the guide
18:08 poiki
We are collecting the secrets from a Keepass file with that little python tool when we need it
18:08 poiki
You could also use a few different other solutions to store your secrets
18:08 foobarbecue
I actually don't need https quite yet since we're just testing internally... I wonder if I can just reconfigure it for http first and get that working and then add the certs
18:09 poiki
of course
18:09 foobarbecue
right now my dataverse pod won't start, with: Unable to attach or mount volumes: unmounted volumes=[certificates], unattached volumes=[default-token-llh4k certificates heapdumps s3-secret db-secret files docroot doi-secret primefaces-upload config]: timed out waiting for the condition
18:09 poiki
just expose the http port and don't redirect
18:09 foobarbecue
should I remove the certificates volume mount from the pod yaml?
18:10 poiki
Well you could just comment the ssl sidecar in kustomization.yaml
18:10 foobarbecue
aha ok
18:10 poiki
But you need to take care of your service
18:11 poiki
Oh I just noticed I seem to have missed adding patch-svc.yaml :-D
18:13 poiki
Not sure if deploying doesn't fail then because of the missing patch
18:13 foobarbecue
yeah I already removed that
18:13 poiki
However, if you don't change the service, it will default to the upstream
18:13 poiki
So exposed on 8080 and 8009
18:14 poiki
You're on k3s right?
18:14 foobarbecue
yep
18:14 poiki
Did you add the ingress in your config?
18:14 poiki
It's not included in the prod-skel
18:15 poiki
Too many different options to do things :-D
18:15 poiki
But being on K3s, you should create the ingress object
18:15 poiki
https://github.com/GlobalDataverseCommunityConsortium/dataverse-kubernetes/blob/develop/personas/demo-k3s/ingress.yaml
18:17 poiki
pretty sure this will help you https://doc.traefik.io/traefik/v1.7/user-guide/kubernetes/#add-a-tls-certificate-to-the-ingress
18:18 poiki
Oh that's an old version...
18:21 poiki
OK looking at the current K3d docs, it should work like that... https://k3d.io/usage/guides/exposing_services/
18:21 poiki
They are using the usual ingress object
18:21 poiki
And that should support the syntax from the Traefik 1.7 docs
18:26 poiki
Ok this ingress thing isn't traefik specific - it's official Kubernetes API . So Traefik will support that no matter what ;-)
18:26 poiki
https://kubernetes.io/docs/concepts/services-networking/ingress/#tls
18:27 poiki
So you can basically add a HTTP only Ingress for now, using the one from the persona.
18:27 poiki
And it's easy to update that to a HTTPS variant
18:28 foobarbecue
ok great, I'm in a meeting now and will give it a try later today. If there's any chance of adding this info to docs or as example that would be helpful
18:28 poiki
Sre
18:28 poiki
Do you feel like adding it with a PR?
18:30 poiki
Not sure I'm helpful writing it in a way it's usable by others :-D
18:31 poiki
I'm outta here folks.
18:31 poiki
Have a good evening y'all
18:32 poiki
As messages seem still stuck in the bridge, I'm likely not seeing any pings...
18:32 poiki left #dataverse
19:57 pdurbin
donsizemore: for the centos 8 pull request, did you happen to look at vagrant, docker-aio or any other code that might be on centos 7?
19:59 donsizemore
oh. er. um, the title just sez "recommend?" (but i'll update those if you like)
20:00 donsizemore
i can send a follow-up commit
20:02 pdurbin
I mean, it's more work, obviously, to mess with that code. It could be done later, in a follow up pull request. Or all at once. Thoughts?
20:02 donsizemore
eh, i can just do it now. i'm not worried about compatibility - ansible has been triggering dataverse on centos 8 in vagrant and ec2 for months
20:03 donsizemore
i'm more interested in watching the fallout from pg driver move
20:03 pdurbin
Right. I've just lost track of whatever code is in the main repo. Obviously, I don't use it very much these days. :/ Thanks for updating it now, as part of that pull request.
20:04 pdurbin
I like having code for people to look at that matches what's in the guides.
20:04 donsizemore
i myself am more used to open source software ;)
20:04 pdurbin
It's like saying, "This actually works." :)
20:06 donsizemore
i may have to dig into the openshift stuff a bit
20:25 pdurbin
I think I deleted all the openshift stuff.
20:42 pdurbin
huh, the conf file is still there
21:07 nightowl313 joined #dataverse
21:24 nightowl313
hey all ... guess what .. have another question ... when we upload zip files to our dataverse using local storage, they extract to the individual files, but when we change to our s3 store it doesn't do so ... just remains zip file
21:25 pdurbin
Yeah. I'm not sure if it's documented (hopefully it is) but that's expected when you use S3 direct upload.
21:26 nightowl313
ah okay! good to know... wanted to make sure it wasn't something configured incorrectly ... no worries we won't advertise that ability!
21:26 nightowl313
thanks!
21:27 pdurbin
I can't find it in the guides. If you want, please go ahead and open an issue to document it.
21:28 nightowl313
okay will do ... thanks for all the quick help! We're doing a lot of testing as we get new dataverse/dataset requests and coming across things we didn't even think about before!
21:28 nightowl313
i feel like a pest!
21:29 nightowl313
you all are amazing!
21:29 pdurbin
Found it. In the dev guide.
21:29 pdurbin
"At present, one potential drawback for direct-upload is that files are only partially ‘ingested’, tabular and FITS files are processed, but zip files are not unzipped, and the file contents are not inspected to evaluate their mimetype."
21:29 pdurbin
https://guides.dataverse.org/en/5.1.1/developers/big-data-support.html#s3-direct-upload-and-download
21:30 pdurbin
The idea was that some day Big Data Support would grow up and move from the dev guide to somewhere else, installation guide or admin guide.
21:39 nightowl313
so we are just using the regular dv interface .. is it because we have direct upload enabled? we can turn it off
21:39 nightowl313
we are also seeing some issues with uploading a lot of files ... the upload stops after 10 or so files and then none of them upload
21:40 nightowl313
maybe this is a reason for having a different bucket for direct upload?
21:42 pdurbin
Oh! I thought you were using direct upload. With normal S3 files should be unzipped.
21:42 nightowl313
we do have direct upload enabled on the bucket that we are using ... but we are uploading these files through the regular dv interface
21:47 pdurbin
Ah but direct upload works (like magic) through the regular web interface.
21:48 nightowl313
so it's okay to have direct upload enabled on our main s3 bucket even though we are typically just using the regular interface? although i think we need the unzip thing to work
21:49 pdurbin
If you want unzip to work with direct upload, you should open an issue about that, please. :)
21:50 pdurbin
Given how things work right now, if you want unzip to work, it sounds like you need to turn off direct upload for that bucket or store or whatever.
21:50 pdurbin
I'm a little fuzzy on how to configure it.
21:50 pdurbin
I'm pretty sure we only have one bucket.
21:50 pdurbin
But we're starting to configure multiple stores.
21:51 pdurbin
With different file size upload limits, for example.
21:52 nightowl313
okay having trouble with the previewers now, too ... wonder if that is related ... yikes
21:56 pdurbin
Probably not but who knows. It's super dark here already. Heading out. See you all tomorrow.
21:56 pdurbin left #dataverse
22:06 nightowl313 joined #dataverse
22:11 nightowl313 joined #dataverse
22:31 dataverse-user joined #dataverse
22:35 dataverse-user
FWIW: While direct upload *should* work for all files, since it's new and more complex, it's probably better to configure two s3 stores that are the same (same bucket even) except for direct upload and size limits (file size, ingest size), and then use direct upload for people who need it (probably due to file size)
22:42 nightowl313
thanks, dataverse-user! i removed direct upload and things started working correctly again ... although it may have had to do with the fact that we enabled encryption on the bucket ... it seems to have started after that
22:43 nightowl313
but, I do like your idea of having 2 separate stores ... and one with direct upload for those that need it
22:49 dataverse-user
encryption on the bucket? https or something else?
22:50 dataverse-user
THe two issues I've seen are CORS policies not right (and 5.x needs the ETags header allowed which was not true for direct upload before) or a problem using http for DV and https for the bucket or vice versa
22:52 nightowl313
just the default aws encryption that you can add in the properties
22:54 nightowl313
we hadn't enabled that before and we enabled it a few days ago ... everything seemed to be fine before then, so I thought maybe there was some conflict ... turning off direct upload fixed things (encryption is still on) but that doesn't necessarily mean it was the cause .. going to try turning it back on on our test site
23:05 dataverse-user
hmm - haven't played with that - since direct upload requires presigned URLs to work, it's possible that normal upload would still work and there's something needed with presigned URLs that hasn't been done yet.
23:05 dataverse-user
I'm also going to head off into the dark, so mail may be better after this. (BTW, do you know if I can actually mail you yet?)
23:12 nightowl313
our main asu it dept wrote back and said that they are having difficulty with all yahoo email addresses and asked if they could see the original email response for the headers ... i can send you my gmail email address and you can email there until then ... thanks so much!!
23:19 nightowl313
oops i mean hotmail
23:20 nightowl313
=)
23:20 nightowl313
they are seeing all hotmail emails being blocked
