Time
S
Nick
Message
07:04 Virgile joined #dataverse
07:11 Virgile joined #dataverse
07:14 Virgile joined #dataverse
08:46 juancorr joined #dataverse
12:29 Youssef_Ouahalou joined #dataverse
12:57 donsizemore joined #dataverse
14:58 nightowl313 joined #dataverse
15:12 poikilotherm
donsizemore pdurbin: is there an API test for the ingest part of Dataverse? I'm not sure from the source files...
15:13 donsizemore
@poikilotherm I remember some API tests failing because the allotted ingest period was exceeded, so they're definitely in there
15:16 poikilotherm
I found "TabularIT" so far, but it is "@Ignored" and not to be found in a maven call that includes the other IT tests...
15:18 stefankasberger joined #dataverse
15:19 poikilotherm
Wondering if current API tests are sufficient to make sure that the JMS changes keep things working.
15:27 stefankasberger
hey guys!
15:31 poikilotherm
Hey stefankasberger :-)
15:48 stefankasberger joined #dataverse
15:49 pdurbin joined #dataverse
15:49 pdurbin
Youssef_Ouahalou: you're on the map! Congrats!
15:50 Youssef_Ouahalou
hello :)
15:50 Youssef_Ouahalou
Yes finally ! Thank you
15:50 pdurbin
poikilotherm: there's an API for uningest and one for reingest.
15:59 pdurbin
stefankasberger: thanks for the early Christmas present! This: https://github.com/AUSSDA/aussda_tests :)
16:25 poikilotherm
pdurbin those failing tests seem related to no files in the json output. where they ingested before?
16:27 pdurbin
Well, develop is passing. So, yes? :)
16:37 pdurbin
poikilotherm: do you think datasourcedefinition might affect ingest?
17:01 pdurbin
poikilotherm: having just played with it on my laptop, I can definitely say that ingest doesn't work in that branch.
17:10 stefankasberger
@pdurbin: you are welcome. hope it helps some folks. :) I am pretty happy with the outcome.
17:13 pdurbin
I'm just now singing donsizemore's praises in Slack about the automated testing infrastructure he built for us. Someday I'd like to add some browser based tests like yours. Slava has some too by the way: https://github.com/GlobalDataverseCommunityConsortium/dataverse-test-suite
17:13 pdurbin
Maybe you two can join forces. :)
20:16 pdurbin
poikilotherm: I haven't thought much about database pooling before your pull request. :)
20:22 poikilotherm
Pdurbin Me neither XD
20:23 pameyer joined #dataverse
20:23 poikilotherm
Some digging and reading revealed the truth...
20:24 pdurbin
So is the idea that we used to manually create a pool in the app server but now we are using a pool that's automatically created? I saw you wrote, "From various sources on the net I learned that @DataSourceDefinition is always creating an application server level pool."
20:25 poikilotherm
Yeah. Actually I didn't look at the appserver level code, if the usage of Pooling Data source has been harmful in the past. When we go with what the docs say, it might have been bad.
20:25 poikilotherm
But I also saw that Payara tries to identify the correct classes.
20:26 poikilotherm
Oh please don't skip quoting the rest. I looked at the payara code to verify
20:26 poikilotherm
This does actually work :-D
20:27 pdurbin
:)
20:27 poikilotherm
We might want to introduce more tweaks later. Like adding support to tune the pool size etc
20:27 pdurbin
That's what I was thinking. Tuning. Monitoring.
20:28 pdurbin
Here's an old issue we worked on: Connection Pool: Glassfish is running out of db pool connections in production. #2595 - https://github.com/IQSS/dataverse/issues/2595
20:29 pdurbin
So yeah, I imagine we'll want to be able to tune the new thing.
20:30 poikilotherm
I asked leonid about it but he said we're good.
20:30 pdurbin
oh good
20:31 poikilotherm
https://www.github.com/poikilotherm/dataverse/tree/7424-maildefinition/src%2Fmain%2Fjava%2Fedu%2Fharvard%2Fiq%2Fdataverse%2Futil%2FMailSessionProducer.java
20:32 poikilotherm
This is what I did for the mail resource. Enable mapping of all resources to the session config
20:33 poikilotherm
All of em configurable via MPCONFIG using dataverse.mail.system.* properties
20:34 pdurbin
ok, and you just append the names at https://javaee.github.io/javamail/docs/api/com/sun/mail/smtp/package-summary.html
20:34 poikilotherm
Mapped to mail.smtp.*
20:34 poikilotherm
Exactly :-)
20:35 pdurbin
So you have a plan for how to configure the database pool if we need to.
20:36 pdurbin
Right now it looks like it's hard coded to minPoolSize = 10, maxPoolSize = 200
20:37 poikilotherm
Yes Sir. Although we need a code change to enable. So maybe we better think of this early...
20:39 pdurbin
https://blog.payara.fish/an-intro-to-connection-pools-in-payara-server-5 is making me wonder if we could override what's in @DataSourceDefinition with web.xml, ejb-jar.xml or application.xml. I don't know which has precedence.
21:04 poikilotherm
https://github.com/IQSS/dataverse/pull/7422#issuecomment-736770694
21:04 poikilotherm
Pdurbin HELP. Kcondon seems to be hooked off and I have no idea what made him feel lost inside what I wrote in my explanation on the pooling in ... :-/
21:04 poikilotherm
Ah that link should have gone inside the message...
21:05 pdurbin
Yeah, I was just reading that. The drama. :)
21:06 pdurbin
You're both on Slack. Maybe you two can work it out there, away from prying eyes. :)
21:17 pdurbin
Away from the Helens of the world. [Simpsons reference]
21:22 poikilotherm
Ok reached out to him. Hope he sees it...
21:23 pdurbin
Cool.
21:23 poikilotherm
And thx 😊
21:23 pdurbin
Sounds like you're gonna hack on the branch more tomorrow.
21:23 pdurbin
I'm just glad the tests pass. That wasn't true this morning. :)
21:26 poikilotherm
I still wonder why this happened in ingest only....
21:27 pameyer
the db connection pool problems were a spin off of ingest problems?
21:27 pdurbin
meh, it's more like layers, I guess
21:28 pdurbin
tests were failing due to ingest being broken
21:28 pdurbin
so we switched to className = "org.postgresql.ds.PGSimpleDataSource"
21:29 pdurbin
but now we're thinking about "simple" vs the old pooling we had
21:30 pameyer
my knowledge of how ingest works is essentially zero, but it wouldn't seem to be something that would need a bunch of db traffic
21:31 pameyer
but people making guesses of things they don't know enough about to be useful is probably something the world could do with less of :)
21:31 pdurbin
Well, ingest puts jobs on a queue and that queue uses the database. Unless poikilotherm corrects me. :)
21:34 pameyer
cool - I hadn't known that
21:34 pameyer
but it does make some sense to re-use your db for a job queue, rather than needing another one
21:56 pdurbin left #dataverse
22:29 poikilotherm
Sorry pdurbin, but JMS is not using the database for the queue. All in memory
22:31 poikilotherm
And I once more repeat: using the simple data source is fine, as the appserver takes care of pooling.
22:57 pameyer
@poikilotherm so we're back to wondering why ingest cares about db connections?
