Time
S
Nick
Message
00:03 dataverse-user joined #dataverse
07:00 poikilotherm joined #dataverse
07:43 juancorr joined #dataverse
09:25 pdurbin
poikilotherm: morning. That 6142-flex-solr-schema branch you mentioned is local, right? I don't see it on your fork.
09:50 poikilotherm
Mornin!
09:51 poikilotherm
Yeah, I didn't push yet
09:51 poikilotherm
Almost done
09:51 poikilotherm
Wanna have a look? Can push a WIP
09:51 poikilotherm
Script done so far, wanted to test in docker aio too before doing a PR
09:52 pdurbin
No rush. I have plenty to do. And Joel is all set now.
09:52 poikilotherm
Yeah saw it in the logs
09:52 poikilotherm
My script would have helped him ;-)
09:52 pdurbin
Did you see his comment about terraform?
09:52 poikilotherm
Yah
09:52 poikilotherm
+e
11:54 yoh joined #dataverse
12:06 poikilotherm
I almost forgot how complicated docker-aio is... :-/
12:20 donsizemore joined #dataverse
12:36 pdurbin
Too complicated? :)
12:36 pdurbin
donsizemore: should we fill poikilotherm in on the plan for Jenkins? :) And good morning.
12:39 poikilotherm
pdurbin I just created https://github.com/IQSS/dataverse/pull/6146
12:39 poikilotherm
I need to add some docs for the new script
12:39 poikilotherm
But you might risk taking a look
12:41 pdurbin
What does cmb mean?
12:41 poikilotherm
custom metadata block
12:42 pdurbin
but "title" isn't a custom metadata block
12:42 pdurbin
er, "title" isn't in a custom metadata block
12:42 poikilotherm
Right. But the whole thing is about custom metadata, right?
12:43 poikilotherm
If you don't deploy your own, you don't need this, as the default deployed schema is sufficient
12:43 pdurbin
Well, we use the term "custom metadata block" to mean metadata blocks that anyone can create for whatever crazy use case they have. They're different than the... uh... out of the box metadata blocks. :)
12:44 poikilotherm
I can still rename it :-D
12:44 pdurbin
thanks
12:44 donsizemore
@pdurbin that's fine on jenkins. i need to do a few things for odum this morning first, then i'll start in on yesterday's list
12:45 pdurbin
donsizemore: sounds good. Thanks!
12:45 poikilotherm
pdurbin: I'm open to suggestions
12:45 poikilotherm
A clean and lean name would be cool ;-)
12:46 pdurbin
poikilotherm: I might make a pull request against your pull request.
12:46 poikilotherm
Go ahead
12:46 poikilotherm
Shall I add some docs to the metadata customization part first?
12:47 pdurbin
I'm thinking.
12:47 pdurbin
I think you should put this in code review as-is so I can talk about it at standup in a few hours.
12:47 pdurbin
I like the direction you're going.
12:48 poikilotherm
And add no docs about it?
12:48 pdurbin
But I want to put it on the radar of other people at standup.
12:49 poikilotherm
Good idea :-)
12:49 pdurbin
poikilotherm: if you could stub out a bit more in the dev guide it would be great. We have http://guides.dataverse.org/en/4.16/developers/tips.html#solr but maybe it's time for a dedicated page about Solr.
12:50 poikilotherm
Well I thought I should add some docs in the metadata customization guide
12:50 pdurbin
Yes, both would be great.
12:50 pdurbin
Developers need to understand the process.
12:50 pdurbin
And so do people who want to create their own blocks.
12:51 poikilotherm
Right
12:51 pdurbin
Does a dedicated page about Solr in the dev guide make sense?
12:51 poikilotherm
Maybe. But it sounds like a bigger issue
12:51 pdurbin
Sure.
12:51 poikilotherm
There's a lot of stuff that should be added about Solr for devs
12:51 pdurbin
How about for now you just add a TODO under that "tips" page.
12:52 poikilotherm
As a comment? Or visible?
12:52 pdurbin
Visibile. Same or similar to **TODO: describe usage of new script.**
12:52 poikilotherm
Right
12:52 pdurbin
just so we can see it in the diff
13:34 donsizemore
@pdurbin i don't seem to be able to close your dataverse-metrics addition cards at https://github.com/orgs/IQSS/projects/3#card-26069708 but you can close them
13:34 donsizemore
somebody threw a 503 doe
13:50 poikilotherm
Ok pdurbin I added the PR to Code Review
13:50 poikilotherm
Need to go now - kindergarten closing in few minutes :-D
13:50 poikilotherm
Cu on Monday
13:55 pdurbin
donsizemore: thanks! Hmm. Should I do something else instead of those cards?
14:06 donsizemore
@pdurbin whatever works for you; i just didn't find a way to mark them "done"
14:07 pdurbin
Maybe https://github.com/IQSS/dataverse-installations is a more logical place to track them.
14:10 pdurbin
donsizemore: can you close them now?
14:11 donsizemore
@pdurbin yes - done!
14:11 pdurbin
cool, so this works for you?
14:11 donsizemore
as long as i remember to keep the orgs tab open ;)
14:11 pdurbin
heh
14:12 pdurbin
just lemme know if you want to switch to something else
14:19 donsizemore
I just spoke to Akio about the Postgres deadlocks, I'd like to dig into that a little bit.
14:19 donsizemore
Improving any performance snafus between Glassfish and Postgres-9.6+ could benefit everybody
14:23 pdurbin
donsizemore: yes, absolutely. Thanks! Oh and I was saying to Tania and the rest of the design team that it would nice if we could spin up trsa-web next to Dataverse. Would it make sense to use dataverse-kubernetes for that?
14:26 donsizemore
sure. it's on dockerhub https://hub.docker.com/r/odumunc/trsa-web but you may want to build your own container to edit trsa.config
14:26 donsizemore
https://github.com/OdumInstitute/trsa-web/tree/jee8line is Akio's working branch
14:26 donsizemore
the Dockerfile should look familiar to @poikilotherm
14:27 pdurbin
Ok, cool. I'm just thinking that TRSA would be a lot more real to people here if we could click around and play with it the way Jon does in his demos.
14:28 donsizemore
it's easy to spin up. development hit a snag for the past couple weeks but he's back in the saddle
14:29 pdurbin
cool, I left Akio a review over at https://github.com/IQSS/dataverse/pull/6068
14:29 donsizemore
note that he carved out the ingest functionality so TRSA wants to talk to its corresponding fork of Dataverse: https://github.com/OdumInstitute/dataverse/tree/trsa-api
14:30 pdurbin
interesting
14:30 donsizemore
"In general, applications must be ready to handle deadlocks issue and retry the transaction in case of a failure." http://shiroyasha.io/deadlocks-in-postgresql.html
14:31 pdurbin
sounds like good advice :)
14:34 donsizemore
ima start by setting log_lock_waits = on in Odum's fork
14:36 pdurbin
sounds fine
14:36 pdurbin
Do you think this falls under https://github.com/IQSS/dataverse/issues/4201 ? (performance)
14:53 donsizemore
i think it's multiple begins; stepping on one anothers' toes (so more timing than performance)
14:55 pdurbin
begins?
14:58 donsizemore
postgres' begin; and commit; statements
14:58 pdurbin
gotcha
14:58 donsizemore
the default deadlock wait is 1sec
14:58 donsizemore
so slow storage would explain exactly what we're seeing
14:58 pdurbin
well, what I'm thinking is that this sprint is all about automated testing... is there a way to automate a test for database deadlocks?
15:03 donsizemore
run it on odum's vmware cluster ;)
15:03 pdurbin
heh
15:04 donsizemore
what does database expect in response from the database?
15:04 donsizemore
^^ dataverse
15:04 pdurbin
uh, data?
15:04 pdurbin
:)
15:05 donsizemore
can it catch a deadlock
15:05 pdurbin
oh, hmm
15:07 pdurbin
EJB: How To Catch javax.persistence.OptimisticLockException http://www.adam-bien.com/roller/abien/entry/ejb_how_to_catch_javax
15:07 pdurbin
The short answer is I don't know but maybe that blog post will help.
15:13 donsizemore
i'm starting with a log_lock_waits switch in dataverse-ansible
15:39 andrewSC joined #dataverse
15:44 pdurbin
perfect
15:45 pdurbin
and I mentioned and standup that you're digging in on database deadlocks
15:45 pdurbin
much appreciated
16:36 icarito[m] joined #dataverse
17:08 donsizemore joined #dataverse
17:09 donsizemore
@pdurbin are you interested in SQL errors during deploy?
17:23 rigelk joined #dataverse
17:30 pdurbin
Do you mean this? https://github.com/IQSS/dataverse/issues/3303
17:30 pdurbin
donsizemore: ^^
17:41 donsizemore
no, this one is a non-existent sequence
17:44 donsizemore
on a related note (to DB locks) kraffmiller just submitted #6128
17:55 joelmarkanderson joined #dataverse
17:55 pdurbin
donsizemore: I would say you're welcome to create an issue for the SQL error
17:56 joelmarkanderson
@pdurbin: howdy, i may have another question for you today from my researchers
17:57 pdurbin
joelmarkanderson: hit me
17:57 joelmarkanderson
they are getting "The following file already exists in the dataset"
17:57 joelmarkanderson
i assume this might be md5 collision?
17:58 joelmarkanderson
they claim it's not filename collision
17:58 joelmarkanderson
but i haven't received their file payload to verify yet
17:59 pdurbin
joelmarkanderson: yes, md5. There's at least one open issue about this. One sec.
18:02 donsizemore
@pdurbin !! https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-testSubset/27/
18:02 pdurbin
As a researcher, I need to publish a dataset that contains files with the same content, which are handled differently #4813
18:02 pdurbin
joelmarkanderson: https://github.com/IQSS/dataverse/issues/4813
18:02 donsizemore
(that's all of them)
18:02 * pdurbin
looks
18:03 pdurbin
donsizemore: is that blue I see? No more red? :)
18:04 donsizemore
yes though we got a deadlock with log_lock turned on
18:04 donsizemore
should all this just go in an issue? (or should the initial deploy SQL error go in its own issue)
18:07 pdurbin
donsizemore: maybe two new issues
18:07 pdurbin
joelmarkanderson: sorry talking to donsizemore and people on Slack also. Does that issue help? Also... I'd love for you to create a board for VTTI. :)
18:09 joelmarkanderson
@pdurbin: thanks, that helps me enough for today! you mean project board?
18:10 pdurbin
joelmarkanderson: yeah, it's a new thing I'm encouraging installations of Dataverse to do. If you look at https://github.com/orgs/IQSS/projects you'll see "OdumInstitute" created by donsizemore . And some others: Qualitative Data Repository, Scholars Portal Dataverse, UVa LibraData Dataverse, WZB Berlin Social Science Center, DataSPIRE - Sciences Po institutional Dataverse, DataverseNO
18:11 pdurbin
joelmarkanderson: for more context, please see "What if installations had a board?" at https://scholar.harvard.edu/pdurbin/blog/2019/jupyter-notebooks-and-crazy-ideas-for-dataverse :)
18:11 pdurbin
"Think hard about the next bug fix or feature that you want."
18:13 joelmarkanderson
hm
18:20 donsizemore
@pdurbin are you in the impact demo with mercè and gustavo?
18:21 pdurbin
What impact demo?
18:22 donsizemore
gustavo, tania and mercè are in it, sherry lake, piotr, some more folks from harvard
18:22 pdurbin
interesting
18:22 pdurbin
Jon is demo'ing?
18:26 donsizemore
just started his part. then i get drinkies!
18:26 pdurbin
is there a live stream?
18:27 donsizemore
it's a zoom
18:27 donsizemore
(and i'm sitting on the floor in jon's office)
18:27 pdurbin
oh, a zoom
18:37 donsizemore joined #dataverse
18:38 donsizemore
@pdurbin and jon and i just found a bug during the demo, but the guy at RENCI presenting accidentally skipped that step so i don't think the audience will see it
18:40 pdurbin
Is the demo being recorded?
18:40 donsizemore
i'll find out
19:33 pdurbin
thanks
20:40 donsizemore joined #dataverse
20:44 donsizemore
@pdurbin Ilya responds that Mercè was in control of the Zoom, so she would've needed to record it
20:44 donsizemore
@pdurbin but if you want the 25¢ tour, I have administrative access to most of the components demonstrated
20:50 pdurbin
heh, I'm good, time for margaritas by the pool?
20:54 pdurbin
time for me to head out, at least
20:54 pdurbin
have a good weekend, all!
20:54 pdurbin left #dataverse
