IQSS logo

IRC log for #dataverse, 2020-12-01

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

All times shown according to UTC.

Time S Nick Message
07:04 Virgile joined #dataverse
07:11 Virgile joined #dataverse
07:14 Virgile joined #dataverse
08:46 juancorr joined #dataverse
12:29 Youssef_Ouahalou joined #dataverse
12:57 donsizemore joined #dataverse
14:58 nightowl313 joined #dataverse
15:12 poikilotherm donsizemore pdurbin: is there an API test for the ingest part of Dataverse? I'm not sure from the source files...
15:13 donsizemore @poikilotherm I remember some API tests failing because the allotted ingest period was exceeded, so they're definitely in there
15:16 poikilotherm I found "TabularIT" so far, but it is "@Ignored" and not to be found in a maven call that includes the other IT tests...
15:18 stefankasberger joined #dataverse
15:19 poikilotherm Wondering if current API tests are sufficient to make sure that the JMS changes keep things working.
15:27 stefankasberger hey guys!
15:31 poikilotherm Hey stefankasberger :-)
15:48 stefankasberger joined #dataverse
15:49 pdurbin joined #dataverse
15:49 pdurbin Youssef_Ouahalou: you're on the map! Congrats!
15:50 Youssef_Ouahalou hello :)
15:50 Youssef_Ouahalou Yes finally ! Thank you
15:50 pdurbin poikilotherm: there's an API for uningest and one for reingest.
15:59 pdurbin stefankasberger: thanks for the early Christmas present! This: https://github.com/AUSSDA/aussda_tests :)
16:25 poikilotherm pdurbin those failing tests seem related to no files in the json output. where they ingested before?
16:27 pdurbin Well, develop is passing. So, yes? :)
16:37 pdurbin poikilotherm: do you think datasourcedefinition might affect ingest?
17:01 pdurbin poikilotherm: having just played with it on my laptop, I can definitely say that ingest doesn't work in that branch.
17:10 stefankasberger @pdurbin: you are welcome. hope it helps some folks. :) I am pretty happy with the outcome.
17:13 pdurbin I'm just now singing donsizemore's praises in Slack about the automated testing infrastructure he built for us. Someday I'd like to add some browser based tests like yours. Slava has some too by the way: https://github.com/GlobalDataverseCommunityConsortium/dataverse-test-suite
17:13 pdurbin Maybe you two can join forces. :)
20:16 pdurbin poikilotherm: I haven't thought much about database pooling before your pull request. :)
20:22 poikilotherm Pdurbin Me neither XD
20:23 pameyer joined #dataverse
20:23 poikilotherm Some digging and reading revealed the truth...
20:24 pdurbin So is the idea that we used to manually create a pool in the app server but now we are using a pool that's automatically created? I saw you wrote, "From various sources on the net I learned that @DataSourceDefinition is always creating an application server level pool."
20:25 poikilotherm Yeah. Actually I didn't look at the appserver level code, if the usage of Pooling Data source has been harmful in the past. When we go with what the docs say, it might have been bad.
20:25 poikilotherm But I also saw that Payara tries to identify the correct classes.
20:26 poikilotherm Oh please don't skip quoting the rest. I looked at the payara code to verify
20:26 poikilotherm This does actually work :-D
20:27 pdurbin :)
20:27 poikilotherm We might want to introduce more tweaks later. Like adding support to tune the pool size etc
20:27 pdurbin That's what I was thinking. Tuning. Monitoring.
20:28 pdurbin Here's an old issue we worked on: Connection Pool: Glassfish is running out of db pool connections in production. #2595 - https://github.com/IQSS/dataverse/issues/2595
20:29 pdurbin So yeah, I imagine we'll want to be able to tune the new thing.
20:30 poikilotherm I asked leonid about it but he said we're good.
20:30 pdurbin oh good
20:31 poikilotherm https://www.github.com/poikilotherm/dataverse/tree/7424-maildefinition/src%2Fmain%2Fjava%2Fedu%2Fharvard%2Fiq%2Fdataverse%2Futil%2FMailSessionProducer.java
20:32 poikilotherm This is what I did for the mail resource. Enable mapping of all resources to the session config
20:33 poikilotherm All of em configurable via MPCONFIG using dataverse.mail.system.* properties
20:34 pdurbin ok, and you just append the names at https://javaee.github.io/javamail/docs/api/com/sun/mail/smtp/package-summary.html
20:34 poikilotherm Mapped to mail.smtp.*
20:34 poikilotherm Exactly :-)
20:35 pdurbin So you have a plan for how to configure the database pool if we need to.
20:36 pdurbin Right now it looks like it's hard coded to minPoolSize = 10, maxPoolSize = 200
20:37 poikilotherm Yes Sir. Although we need a code change to enable. So maybe we better think of this early...
20:39 pdurbin https://blog.payara.fish/an-intro-to-connection-pools-in-payara-server-5 is making me wonder if we could override what's in @DataSourceDefinition with web.xml, ejb-jar.xml or application.xml. I don't know which has precedence.
21:04 poikilotherm https://github.com/IQSS/dataverse/pull/7422#issuecomment-736770694
21:04 poikilotherm Pdurbin HELP. Kcondon seems to be hooked off and I have no idea what made him feel lost inside what I wrote in my explanation on the pooling in ... :-/
21:04 poikilotherm Ah that link should have gone inside the message...
21:05 pdurbin Yeah, I was just reading that. The drama. :)
21:06 pdurbin You're both on Slack. Maybe you two can work it out there, away from prying eyes. :)
21:17 pdurbin Away from the Helens of the world. [Simpsons reference]
21:22 poikilotherm Ok reached out to him. Hope he sees it...
21:23 pdurbin Cool.
21:23 poikilotherm And thx 😊
21:23 pdurbin Sounds like you're gonna hack on the branch more tomorrow.
21:23 pdurbin I'm just glad the tests pass. That wasn't true this morning. :)
21:26 poikilotherm I still wonder why this happened in ingest only....
21:27 pameyer the db connection pool problems were a spin off of ingest problems?
21:27 pdurbin meh, it's more like layers, I guess
21:28 pdurbin tests were failing due to ingest being broken
21:28 pdurbin so we switched to className = "org.postgresql.ds.PGSimpleDataSource"
21:29 pdurbin but now we're thinking about "simple" vs the old pooling we had
21:30 pameyer my knowledge of how ingest works is essentially zero, but it wouldn't seem to be something that would need a bunch of db traffic
21:31 pameyer but people making guesses of things they don't know enough about to be useful is probably something the world could do with less of :)
21:31 pdurbin Well, ingest puts jobs on a queue and that queue uses the database. Unless poikilotherm corrects me. :)
21:34 pameyer cool - I hadn't known that
21:34 pameyer but it does make some sense to re-use your db for a job queue, rather than needing another one
21:56 pdurbin left #dataverse
22:29 poikilotherm Sorry pdurbin, but JMS is not using the database for the queue. All in memory
22:31 poikilotherm And I once more repeat: using the simple data source is fine, as the appserver takes care of pooling.
22:57 pameyer @poikilotherm so we're back to wondering why ingest cares about db connections?

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.