Time
S
Nick
Message
02:31 jri joined #dataverse
03:25 jri joined #dataverse
08:50 juancorr joined #dataverse
10:00 jri joined #dataverse
12:16 jri joined #dataverse
12:42 donsizemore joined #dataverse
14:19 pdurbin joined #dataverse
17:18 pameyer joined #dataverse
17:18 pameyer
noticed a slight oddity - gdcc jenkins and my dev setup disagree on which ITs are failing
17:20 pdurbin
whelp, let's look at the gdcc jenkins first, I guess
17:20 pameyer
got a repeat run going now locally to see if it's a container glitch or not (always a possibility)
17:21 pdurbin
I'm looking at https://jenkins.dataverse.org/job/IQSS-dataverse-develop/ but we haven't had a blue (success) since April 1.
17:23 donsizemore
@pameyer @pdurbin the tests all run in EC2; jenkins just launches them
17:24 pdurbin
donsizemore: yep
17:24 donsizemore
@pameyer @pdurbin on Friday I was running integration tests on develop on Payara5 but couldn't get a consistent result from any two runs
17:24 pameyer
well, I haven't _heard_ about any aws stability/capacity issues....
17:25 pameyer
@donsizemore fresh VMs for each run?
17:25 donsizemore
@pameyer we got this behavior in Docker on the Jenkins server, and blamed slow storage (in Docker, in a VM , on shared vmware storage). I could consistently produced database deadlocks
17:26 pameyer
huh - I don't think I've ever seen a docker database deadlock
17:26 pameyer
but I've also never used vmware storage directly, so that might be it
17:27 donsizemore
@pameyer not Docker. integration tests in docker, on a VM , on a shared NFS datastore. so many layers of slowdown.
17:27 donsizemore
@pameyer in addition, the datastore is a netapp volume in an aggregate
17:28 donsizemore
@pameyer but that's historical at this point
17:28 pameyer
were your payara5 inconsistencies in VMs?
17:30 donsizemore
no, EC2
17:30 pameyer
I count those as VMs
17:33 pameyer
@pdurbin any ideas if that was before or after 4.20 release?
17:33 donsizemore
@pdurbin [ERROR] SiteMapUtilTest.testUpdateSiteMap:91 expected null, but was:<java.io .FileNotFoundException: /tmp/sitemap.xml (No such file or directory)>
17:34 pdurbin
pameyer: the 4.20 release happened when I was out of commission. So I don't know if the tests were passing or not. I do agree that tests are failing now.
17:34 pameyer
pdurbin: right, I'd forgotten about that
17:34 donsizemore
@pdurbin the file is present, and owned by jenkins, timestamped 1311 today
17:34 pameyer
glad to hear you're back in commission :)
17:34 donsizemore
an issue was opened then closed
17:35 pdurbin
pameyer: thanks :)
17:35 donsizemore
https://github.com/IQSS/dataverse/issues/6776
17:36 pameyer
looks to me like jenkins thinks FilesIT.testUningestFileViaApi is fine
17:37 pdurbin
donsizemore: interesting fix for that issue
17:37 pdurbin
pameyer: oh? I thought I saw that error in a few recent runs on develop
17:38 pameyer
I'm seeing it on 47ca3c36e4b2c2bb9954f75f554068d21264545f
17:39 pameyer
seems to be fairly consistent - 2x runs, 2x containers
17:39 pdurbin
pameyer: if you're willing to create an issue, please do. It's always appreciated.
17:39 donsizemore
WARNING: Unable to update sitemap! The staged file from a previous run already existed. Delete /tmp/sitemap.xml.staged and try again. Exception caught checking that XML is well formed: java.io .FileNotFoundException: /tmp/sitemap.xml (No such file or directory)
17:39 pameyer
and that fix does look interesting. not sure I'm following it, but I'll defer to the java experts
17:40 pameyer
pdurbin: issue incoming, figured it was worth checking in in case it was a known problem and I just didn't see a existing issue
17:41 pdurbin
We certainly aren't talking about failing tests. So thanks to both of you.
17:41 pdurbin
pameyer: any news on when we'll get your installation on the map? :)
17:42 pameyer
pdurbin: my list gets longer rather than shorter. auto-deposit of software releases is likely to converge first
17:43 pdurbin
ok, no worries :)
17:44 pdurbin
pameyer: we're working on updating stuff from glassfish to payara. Any plans to update docker-aio or docker-dcm?
17:44 donsizemore
@pdurbin I manually removed /tmp/sitemap.xml and https://jenkins.dataverse.org/job/IQSS-dataverse-develop/416/console looks good so far
17:45 pdurbin
donsizemore: nice!
17:45 pameyer
pdurbin: vague plans - should be a straighforward swap
17:45 donsizemore
I killed an "OdumInstitute" run this morning when archive.apache.org timed out; but I'm not sure I'm the culprit
17:45 pdurbin
ok, I'm working on docs, which is why I'm asking
17:46 pdurbin
pameyer: also, not long ago poikilotherm asked if we could delete docker-aio
17:46 donsizemore
@pameyer I've got a Payara Dockerfile for Akio's TRSA which I shamelessly stole from the wonderful @poikilotherm
17:47 pameyer
@donsizemore open source is a wonderful thing
17:48 pameyer
@pdurbin does he have a suggested replacement for integration tests and dcm?
17:48 pdurbin
pameyer: something call Testcontainers but it's a little hand-wavey right now.
17:49 pameyer
I'm pretty sure he's doing active dev work right now - I'm not really, for what that's worth
17:50 pdurbin
good to have that in mind
17:51 donsizemore
@pdurbin speaking of test failures. when do I switch the values in jenkins.yml over to payara5-land?
17:51 pdurbin
I'm focused on the docs at the moment but I'm aware that references to Glassfish are in a lot of places: Vagrant, docker-aio, docker-dcm, the old openshift stuff I worked on, etc.
17:52 pdurbin
donsizemore: excellent question. I would say not yet.
17:53 pdurbin
At some point all this "move to Payara" stuff needs to come together. I jumped on the docs issue to get it moving. It's a slog.
17:53 pameyer
it's both exciting and intimidating that the move off gf4 is happening
17:53 donsizemore
I tested Leonid's updated Python3 Payara5 installer, which works as advertised
17:54 pdurbin
I'm about the git rm the grizzly patch I made many years ago. Exciting. :)
17:54 pdurbin
donsizemore: did you try it on a Mac? That's where I'm stuck.
17:55 donsizemore
next I need to make GH PR builder report some appropriate status back to GH, then I was going to start an Ansible branch to see if I can just use the new installer instead of cobbling stuff in
17:55 donsizemore
@pdurbin I ran it on CentOS 8
17:55 pdurbin
ok
17:56 pdurbin
8, nice
17:59 pdurbin
donsizemore: there are a ton of little details in the guides that I'm tempted to bother you with but I'll wait until I make my pull request.
18:07 donsizemore
hey @pameyer look what just failed https://jenkins.dataverse.org/job/IQSS-dataverse-develop/416/testReport/junit/edu.harvard.iq.dataverse.api/FilesIT/testUningestFileViaApi/
18:08 donsizemore
@pdurbin you're welcome to bother me. checklist or google doc if you want?
18:08 pdurbin
probably a checklist in a pull request when I'm ready
18:12 pameyer
@donsizemore - here I was assuming I was doing it wrong ;)
18:13 pameyer
got a run going with the sleep statement uncommented, will let you know how that turns out
18:15 donsizemore
@pameyer the solution, of course, is to give all dataverse installations brand-new storage with SSD FlexCache
18:16 pameyer
:)
18:16 pdurbin
I'm supposed to get a new laptop soon. SSD hard drive. :)
18:18 donsizemore
I got one for my 8 year-old build box. It's like a new machine.
18:19 pameyer
it's suprising how many things that look like applicatino problems are actually storage problems
18:19 pdurbin
donsizemore: actually, here's a quick thing. I'm planning on just changing the init scripts without testing them.
18:19 pdurbin
pameyer: or network problems. Always blame the network. I used to do networking. :)
18:21 pameyer
the first time I heard "wiggle the wires around" as a legit troubleshooting suggestion, I spent probably 5 straight minutes swearing. now, it's just a shrug
18:21 pdurbin
Please, it's "reseat the cable." :)
18:22 pameyer
and the sleep statement doesn't help - FilesIT still fails, and AdminIT now also fails because ingest took too long
19:18 donsizemore
@pdurbin which init scripts?
19:20 pdurbin
donsizemore: the ones I changed in this pull request: https://github.com/IQSS/dataverse/pull/6816 :)
20:20 donsizemore
@pdurbin sir. we prefer small commits in small chunks.
20:21 pdurbin
heh
20:21 * donsizemore
ducks
20:21 donsizemore
looks good to me. did gustavo decide whether his jakarta faces patch is the one?
20:21 pdurbin
I think he's still deciding.
20:22 pdurbin
Let me go update the description to mention the patch.
20:23 pdurbin
done, thanks for the reminder
20:28 donsizemore
"We didn't NEED dialog, we had FACES!"
20:28 pdurbin
:)
20:29 pdurbin
I was so happy to delete the grizzly patch. Too bad we still need one.
21:43 pdurbin left #dataverse
