Time
S
Nick
Message
06:41 juancorr joined #dataverse
07:02 jri joined #dataverse
07:29 jri joined #dataverse
07:46 stefankasberger joined #dataverse
10:49 pdurbin
I stood up a little "preview of recent messages" area at http://chat.dataverse.org for before you log in. What do you think? :)
11:25 poikilotherm joined #dataverse
11:41 pdurbin
sivoais: I had to hack together a little Perl but I survived. :)
12:02 pdurbin
poikilotherm: I convinced Steve to re-open https://github.com/payara/Payara/issues/2296 :)
12:07 pdurbin
stefankasberger: Do you think we should add a job to https://jenkins.dataverse.org to test pyDataverse?
12:26 donsizemore joined #dataverse
12:35 MrK joined #dataverse
12:35 MrK
poikilotherm: Hey, have you managed to run dataverse on Payara?
12:42 poikilotherm
Hi :-)
12:42 poikilotherm
Nope, MrK, I haven't tried yet
12:42 poikilotherm
Err
12:42 poikilotherm
That's not true
12:43 poikilotherm
I tried but failed with stupid log stuff
12:43 poikilotherm
Lemme get those issue for you
12:43 MrK
I just tried with glassfish 5.0.1 and I think it will be impossible.
12:44 poikilotherm
https://github.com/IQSS/dataverse/issues/4172
12:44 poikilotherm
https://github.com/payara/Payara/issues/3506
12:44 poikilotherm
https://github.com/poikilotherm/dataverse/tree/4172-payara5
12:45 poikilotherm
5.0.1?
12:45 poikilotherm
Shouldn't this be Glassfish 5.1 from Jakarta/Eclipse?
12:45 poikilotherm
Beware, my tests are a bit old...
12:46 poikilotherm
The upstream issue with logs have been fixed since
12:46 poikilotherm
Not sure if the Docker image has been updated
12:47 MrK
poikilotherm: Yeah it's the same just rebranded.
12:47 poikilotherm
Ok logging has been fixed in Payara 5.191 and there is a docker image for that release
12:47 poikilotherm
https://hub.docker.com/r/payara/server-full
12:48 poikilotherm
I wouldn't be too sure about that... IIRC there has been some minor tweaks and screws fixed
12:48 poikilotherm
But anyway, if you ask me - go for Payara
12:48 poikilotherm
Eclipse is not ready
12:48 MrK
I'm just saying what's on their page :P https://projects.eclipse.org/projects/ee4j.glassfish/releases/5.1.0
12:49 poikilotherm
Many people at IQSS hoped for Eclipse Glassfish 5.1 being faster in releasing and fixing stuff, but it seems like the project has slowed down alot
12:49 pdurbin
I'm concerned about Eclipse Glassfish based on recently messages on their dev list.
12:49 pdurbin
recent*
12:50 pdurbin
I don't believe I've ever said that Eclipse Glassfish will be fast in releasing. Quite the opposite. :)
12:50 MrK
So you managed to get it running (Payara)? I stopped on those problems https://github.com/IQSS/dataverse/issues/4248#issuecomment-341571761 - And I figured out that the reason of validation error on main page is that previously if validation failed it failed silently now since they fixed it they broke out version.
12:51 MrK
I just wanted to upgrade glassfish 4 since eclipse-link is running on java 1.6 and I can't use streams when I'm playing around with entity.
12:52 poikilotherm
I never made it that far... My CPU was burned before I could test :-D
12:52 poikilotherm
(See the Payara issue above)
12:52 pdurbin
MrK: can I please co-assign you to https://github.com/IQSS/dataverse/issues/4172 so poikilotherm has some company? :)
12:53 poikilotherm
Yeah feels pretty lonely over there
12:53 pdurbin
donsizemore: can you please remind us if Akio is using Glassfish or Payara for his side app?
12:56 donsizemore
@pdurbin payara5
12:57 pdurbin
sweet
12:57 pdurbin
so there's some bad mojo in Dataverse... Payara 5 itself is fine
12:59 pdurbin
https://www.eclipse.org/lists/glassfish-dev/ has the archive of recent messages that concern me a bit about Eclipse Glassfish.
13:00 pdurbin
April 9th: Steve creates a 5.2.0 release: https://www.eclipse.org/lists/glassfish-dev/msg00050.html
13:00 pdurbin
April 16th: The 5.2.0 release is deleted: https://www.eclipse.org/lists/glassfish-dev/msg00057.html
13:01 donsizemore
@pdurbin to be fair, akio's webapp is 2.5MB of source and written from scratch this year
13:01 pdurbin
instead of over ten years of crusty code? :) mature, I mean. battle tested :)
13:04 donsizemore
seasoned
13:04 pdurbin
perfectly ripe, like a tasty pear
13:05 poikilotherm
Are we still talking about Dataverse? I am getting hungry again.
13:05 poikilotherm
;-)
13:05 pdurbin
I forget what we're talking about.
13:05 pdurbin
And I'm hacking on the "move dataset" GUI .
13:05 donsizemore
payara (not papaya) 5
13:05 poikilotherm
Sth about pears :-D
13:05 poikilotherm
*thumbs up*
13:06 poikilotherm
I am working on a presentation on RSE
13:08 pdurbin
any mention of Dataverse in your presentation? :)
13:10 pdurbin
donsizemore: what's the status of the ec2 spin up? Thanks for terminating those instances over the weekend. #amazonbill
13:11 donsizemore
um... still enabled? but i can kill it
13:12 pdurbin
enabled is good!
13:12 pdurbin
does termination happen automatically? with time? or after success or something?
13:17 donsizemore
@pdurbin no automatic termination just yet but we can make that happen if you like
13:17 donsizemore
@pdurbin so far all the "failures" are EC2 taking too long to spin up an instance — i bumped my local ec2 script to 60 seconds
13:18 pdurbin
gotcha
13:18 pdurbin
yeah, the script sometimes fails for me too when ec2 is slow to spin up
13:18 donsizemore
it gives your project a partly-cloudy icon :-/
13:19 pdurbin
we've been called worse than partly-cloudy :)
13:21 pdurbin
I think there are two things we could work on next. Termination or running the API test suite. Any preference?
13:25 pdurbin
Maybe a third thing. Add a second job for dataverse-metrics or some other Dataverse-related software.
13:32 donsizemore
well, jenkins tests the warfile build to start with. if you're going to terminate the ec2 instances automatically without allowing a developer to poke at them, we can just remove that post-build step
13:35 pdurbin
I think a developer will only want to poke if the API test suite fails. The main thing a developer will want is the server.log file. So maybe we can download and keep it somewhere if the API test suite fails? Or just leave the instance running.
13:38 pdurbin
Does that make sense?
13:49 donsizemore
absolutely
13:49 pdurbin
So which of the three? :)
13:51 donsizemore
i'll start with the API test suite?
13:51 donsizemore
need to finish up letsencrypt as well
13:52 pdurbin
I'm excited about https some day but the phoenix server never had it so it shouldn't block API test suite testing.
13:52 pdurbin
I do expect you'll find some gotchas though. You're going to have to scp a file over, or grab it from the code. You'll see. I'll help. :)
14:47 donsizemore joined #dataverse
14:59 pdurbin
donsizemore: this sush json file: https://github.com/IQSS/dataverse/commit/9608ed2
15:22 stefankasberger
@pdurbin: I will think it through, cause I am having planned to dive into jenkins by myself anyway regarding dataverse deployment.
15:36 pdurbin
stefankasberger: ok, to be clear, as a member of the Dataverse community, you are welcome to use https://jenkins.dataverse.org . That's what it's for. :) Please see http://guides.dataverse.org/en/4.13/developers/testing.html#continuous-integration
15:50 stefankasberger
thanks, thats a great option. will keep it in mind and come to you, if necessary.
15:50 pdurbin
sounds great, thanks
16:01 pdurbin
stefankasberger: one more thing if you're still around, speaking of CI.
16:01 stefankasberger
will leave in about 5min.
16:02 pdurbin
I'll be quick. :)
16:02 pdurbin
Do you have an account on Slava's CI pipeline thing yet? :)
16:02 stefankasberger
no.
16:02 pdurbin
ok
16:03 pdurbin
He was on vacation when I emailed him last week. Do you happen to know if he's around this week?
16:09 stefankasberger
no. will have to leave now. sorry. have a nice day.
16:09 pdurbin
you too, thanks!
17:07 donsizemore joined #dataverse
19:00 pdurbin
donsizemore: I hear Akio is about to call in. :)
19:02 donsizemore76 joined #dataverse
19:02 donsizemore76
@pdurbin hello =)
19:09 donsizemore76
@pdurbin you're muffled for some reason. enunciate like a chorister!
19:14 pdurbin
I blame the owl. Thanks for the feedback. I'll try. :)
19:14 pdurbin
Maybe I'll use this for notes.
19:15 pdurbin
I'm glad Jon brought up Pete's stuff. The rsync stuff.
19:17 pdurbin
http://guides.dataverse.org/en/4.13/developers/big-data-support.html
19:19 pdurbin
http://phoenix.dataverse.org/schemaspy/latest/tables/storagesite.html
19:20 pdurbin
{ "hostname": "dataverse.librascholar.edu", "name": "LibraScholar, USA", "primaryStorage": true, "transferProtocols": "rsync,posix,globus" }
19:22 pdurbin
the owl: https://www.owllabs.com/meeting-owl
19:22 donsizemore76
@pdurbin akio's current trsa-test interface, for reference: https://irss-trsatest.irss.unc.edu/trsa-web-2.0/
19:25 pdurbin
curl: (7) Failed to connect to irss-trsatest.irss.unc.edu port 443: Connection refused
19:27 donsizemore76
ah, good. i just noticed this early version displays API tokens, unauthenticated.
19:27 pdurbin
storageidentifier at http://phoenix.dataverse.org/schemaspy/latest/tables/dvobject.html is how Dataverse knows if it's on a filesystem or S3 or Swift.
19:30 pdurbin
curl https://dataverse.harvard.edu/api/datasets/:persistentId?persistentId=doi:10.7910/DVN/TJCLKP | jq '.data.latestVersion.files[0].dataFile.storageIdentifier'
19:30 pdurbin
"s3://dvn-cloud:16927bc8774-400f3e10b3fe"
19:30 pdurbin
donsizemore76: can you please show this to Jon?
19:37 pdurbin
Here's where "swift" is defined: https://github.com/IQSS/dataverse/blob/v4.13/src/main/java/edu/harvard/iq/dataverse/dataaccess/SwiftAccessIO.java#L86
19:37 pdurbin
storageIdentifier.startsWith(SWIFT_IDENTIFIER_PREFIX + "://")
19:38 pdurbin
"file://" is defined here: https://github.com/IQSS/dataverse/blob/v4.13/src/main/java/edu/harvard/iq/dataverse/dataaccess/FileAccessIO.java#L164
19:39 pdurbin
"s3" is definted here: https://github.com/IQSS/dataverse/blob/v4.13/src/main/java/edu/harvard/iq/dataverse/dataaccess/S3AccessIO.java#L109
20:17 pdurbin
Don, slides 6, 7, and 8 show files on two different storage drivers (file system and Swift). The columns have been renamed: https://osf.io/y4d2e/ via the "June 15" tab of https://projects.iq.harvard.edu/dcm2017/agenda
20:17 pdurbin
But like we were saying this only works well for reads.
20:17 pdurbin
Users don't have the ability to choose which storage is used.
20:18 pdurbin
It's controlled by the dataverse.files.storage-driver-id JVM option.
20:51 poikilotherm joined #dataverse
21:52 donsizemore joined #dataverse
21:55 dataverse-user joined #dataverse
21:58 dataverse-user
kamil10 can you contact me at blewis
21:59 blewis joined #dataverse
21:59 bglewis joined #dataverse
23:30 andrewSC joined #dataverse
23:40 andrewSC joined #dataverse
