Time
S
Nick
Message
01:19 jri joined #dataverse
01:48 dataverse-user joined #dataverse
01:48 dataverse-user
hallo
01:49 dataverse-user
i am newbie in dataverse from indonesia
01:50 dataverse-user57 joined #dataverse
02:50 pdurbin
sorry, timezones :/
02:51 pdurbin
I wonder if this was a follow up on https://groups.google.com/d/msg/dataverse-dev/gluSLNay-7Q/218wUbVtAAAJ
04:19 jri joined #dataverse
07:26 jri joined #dataverse
07:35 jri joined #dataverse
08:06 poikilotherm joined #dataverse
08:08 poikilotherm
Mornin. Early Birds. Worms. Catching. All that stuff ;-)
09:21 jonas_ joined #dataverse
09:42 MrK joined #dataverse
09:59 poikilotherm
Hey MrK :-)
10:00 poikilotherm
Do I remember correctly that you guys tried to get Dataverse onto another application server
10:05 pdurbin
mornin' early birds
10:06 poikilotherm
That's not me. 12:00 here ;-)
10:06 poikilotherm
Did you have cup of coffee already?
10:09 pdurbin
No. Need coffee. I thought I'd reply on https://github.com/IQSS/dataverse/issues/5936 first quick.
10:10 poikilotherm
I have a thing or two here for you... ;-) Grab a coffee first, take your time, enjoy. Worms will be around for a while.
10:11 poikilotherm
(They nible code at the moment)
10:20 pdurbin
Ok, done replying. Making coffee. Reading what you write here. Please go ahead.
10:21 poikilotherm
Ok, first about my hacking from yesterday evening
10:21 pdurbin
jonas_: meeting in Berlin today: https://discourse.opensourcedesign.net/t/open-science-event-at-wikimedia-deutschland-june-12-2019/1107
10:21 pdurbin
poikilotherm: sorry, please go ahead
10:21 poikilotherm
I see issues #4172, #5907 and #5913
10:21 poikilotherm
There are some PRs for those around, too
10:22 poikilotherm
All of this seems to boil down to a common cause: updated libs handle viewParam differently than on GL 4.1
10:22 poikilotherm
GF 4.1
10:23 pdurbin
ok
10:23 poikilotherm
I wonder if it might make sense to create a new issue about this and post a price for the most squashed bugs
10:23 pdurbin
Heh, sure, go ahead.
10:23 poikilotherm
This is very likely to bite at many places
10:24 poikilotherm
Actually I am afraid we need to check each and every view around :-/
10:24 pdurbin
Do you think we'd see this bug on newer versions of Glassfish 4? 4.1.1 or 4.1.2 or whatever the latest in the 4.x was?
10:24 poikilotherm
It should be in there
10:24 pdurbin
ok
10:24 poikilotherm
At least to some degree. EL v3.0.1b05 should be included in those releases
10:25 poikilotherm
But IIRC there had been other troubles keeping you guys from those releases, right?
10:25 pdurbin
yeah
10:25 poikilotherm
So this might be hard to test / cross-check
10:25 pdurbin
yeah
10:28 pdurbin
What do you think the fix will look like? The "diff" in the pull request? And will that code work on both Glassfish 4.1 as well as version of Glassfish and Payara that are newer than that?
10:28 poikilotherm
I don't think so. This has been a big bummer bug in the past...
10:29 poikilotherm
I don't know if we can get around things with custom EL Resolvers
10:30 pdurbin
At least with custom EL Resolvers we'd be in control of the behavior.
10:30 poikilotherm
Maybe we can add another patch :-D
10:30 pdurbin
Regardless of the Glassfish version.
10:30 poikilotherm
We could replace the javax.el JAR like we do for welda and grizzly
10:30 pdurbin
Oh, that's a good point.
10:31 pdurbin
I'm not sure it's the best approach but it's an option.
10:31 poikilotherm
Hmm... Thinking... What about trying to reproduce the bug on GF 4.1 with an updated EL lib?
10:32 poikilotherm
(Which contains the fixes back from the good ol' days)
10:32 pdurbin
Good thought.
10:32 pdurbin
If it all boils down to one jar it's good to know which jar and which version.
10:32 poikilotherm
Lemme see if I can dangle together a small image...
10:33 pdurbin
ok
10:33 poikilotherm
But first, issue creating
10:33 pdurbin
ok
10:34 jonas4711 joined #dataverse
10:35 poikilotherm
Hmm on second thought it seems like there is only issue #5907
10:35 poikilotherm
Do you still want me to create a new, more generalized issue?
10:36 pdurbin
I almost closed #5907 yesterday. I couldn't decide what the scope should be. New issue, please.
10:38 poikilotherm
Yes Sir!
10:38 pdurbin
I can tell you what I want. I want code that works on old/current Glassfish and new Glassfish/Payara. Ideally without having to patch any jars. :)
10:39 pdurbin
poikilotherm: again, great find. Thanks for all the research on this bug.
10:39 poikilotherm
Yeah. Sounds great. Dunno if doable
10:40 poikilotherm
BTW - the setting for coercion in JSF has been unchanged since the first commit of Dataverse
10:40 pdurbin
Yeah, I noticed.
10:42 pdurbin
Dunno if you saw but I made a new, smaller pull request yesterday. With just default=root for a dataverse alias: https://github.com/IQSS/dataverse/pull/5933 . But do we want this fix?
10:42 pdurbin
It seems to do no harm on Glassfish 4.1.
10:43 poikilotherm
Yeah I saw it plus your comment.
10:43 poikilotherm
I wondered if it might be worth a try to insert some clever EL there to get the root dataverse name
10:43 poikilotherm
You told me that it could differ in installations...
10:44 poikilotherm
And it totally makes sense to use defaults to mitigate the problem... ;-)
10:45 pdurbin
It can. https://demo.dataverse.org/dataverse/demo is the root dataverse, for example. But I think we think it won't be a problem, that it's safe to put default=root in there.
10:46 pdurbin
I'm planning on closing the older pull request with two fixes in one commit: https://github.com/IQSS/dataverse/pull/5913 . No objections, right?
10:46 poikilotherm
Nope.
10:49 pdurbin
poikilotherm: done. And I'll probably close the issue I opened once you open yours.
10:50 poikilotherm
Check
10:50 pdurbin
Please feel free to use the screenshots from that issue, though. I like screenshots. :)
11:14 MrK
poikilotherm: Hi
11:15 MrK
In longer we will probably try spring :P
11:15 MrK
long term*
11:19 pdurbin
MrK: Spring? Really? :) But which app server?
11:22 MrK
pdurbin: I'm just talking long term :P, none default tomcat servlet container
11:22 pdurbin
ok, so tomcat
11:22 MrK
Yeah in spring you can change it on the fly so no problem with replacing.
11:22 pdurbin
sure
11:23 pdurbin
Well, I'm not sure if I know what you mean, actually. On the fly?
11:23 MrK
Em just that you can replace it and it won't cause loads of trouble.
11:23 pdurbin
sure, ok
11:24 pdurbin
In theory, we can get there with Java EE too.
11:24 pdurbin
So people can choose between Payara, Wildfly, and Liberty.
11:24 pdurbin
or whatever
11:24 pdurbin
but Dataverse is fairly dependent on Glassfish 4.1 right now :(
11:25 MrK
Yeah, but JEE is kinda weak for integration testing and app server is just troubles. Wanna change some dependency? gotta replace jar manualy, good luck with telling everyone that they gotta do it manually :P, in spring you just change dependency in pom.
11:28 pdurbin
I've never used Spring. But people obviously like it. Recently I started listening to https://soundcloud.com/a-bootiful-podcast
11:30 jonas4711 joined #dataverse
11:31 pdurbin
MrK: poikilotherm and I have been talking about IRC vs. newer stuff it you'd like to join in. I have a doc and a spreadsheet for you to read if you want.
11:31 MrK
sure
11:31 pdurbin
jonas4711: you seemed interested in this topic during the Dataverse DE call, I think.
11:32 pdurbin
The first doc is this, a copy and paste from an issue poikilotherm opened with comments by me. All are welcome to comment on this doc or the original issue: https://docs.google.com/document/d/18-4MrbSHYhcxvuFs1alAz0Opm_JTkRK2E0bafAxAHSI/edit
11:34 pdurbin
The spreadsheet is something I created later, called "Dataverse Community Chat Requirements" and the idea is that I have been the champion of certain features or requirements such as public logs. I'm fine with adding more requirements but each requirement needs a champion, I think: https://docs.google.com/spreadsheets/d/1lBm5HYhi0RFbfSYhsl2Is5z35PhOV2h1S8ZGIVoUSj8/edit?usp=sharing
11:38 pdurbin
Comments are welcome on that spreadsheet too, of course.
11:45 poikilotherm
Back from lunch...
11:46 poikilotherm
pdurbin, while I continue to write the issue, you might be interested in this for your other "side project":
11:46 poikilotherm
https://www.ontestautomation.com/tag/rest-assured/
11:46 poikilotherm
Meh
11:46 poikilotherm
https://www.ontestautomation.com/on-handling-processing-time-in-your-integration-tests-with-awaitility-and-on-using-the-gmail-api/
11:46 poikilotherm
https://github.com/awaitility/awaitility/
11:47 poikilotherm
I wondered yesterday how to improve from just using Thread.sleep. This seems pretty reasonable
11:51 pdurbin
poikilotherm: cool, thanks.
11:53 MrK
protip: I improved time to save the new dataverse by 6 times, 2.6 sec out of 3.1 sec are used to send an email, so i put it in another thread.
11:54 pdurbin
MrK: nice, any plans to make a pull request? :)
11:55 MrK
too much changes, I refactored whole DataversePage :P.
11:58 pdurbin
MrK: ok, I forget if I have your installation in our internal spreadsheet. That's where I track forks by their GitHub URL .
11:59 MrK
ahh
12:01 pdurbin
Ok, I already have https://github.com/CeON/dataverse there. Under the "unmapped" installations tab. :)
12:01 pdurbin
MrK: wanna be on the map? :)
12:02 MrK
sure :P
12:04 pdurbin
MrK: great! We'll be asking you for information like this: https://github.com/IQSS/dataverse-metrics/blob/v0.2.5/all-dataverse-installations.json
12:04 pdurbin
MrK: if you could follow this process it would be ideal: http://guides.dataverse.org/en/4.14/installation/config.html#putting-your-dataverse-installation-on-the-map-at-dataverse-org ... thanks!
12:07 MrK
pdurbin: oh so just like sending our json on email :D
12:08 MrK
also we have 3 "version" of repository so it will not be a single url/logo
12:08 MrK
versions*
12:09 pdurbin
MrK: you have three installations of Dataverse?
12:12 MrK
yeah, "general","sociological" and "crystallographic".
12:13 pdurbin
Nice! We'll be happy to add three dots on the map. :) NTU has two installations. A main one and one for their engineering school.
12:14 MrK
But our installations are not public yet, you need a password to access the site.
12:15 pdurbin
MrK: hmm, so we can't collect metrics from them. Or harvest from them via OAI-PMH.
12:18 MrK
maybe we will just talk about it when we are gonna go public :P?
12:25 pdurbin
MrK: sure that sounds fine. Also, how do you feel about "throwing over the wall" whatever Bundle.properties files you have in the form of a pull request at https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs ?
12:28 pdurbin
MrK: I forget if you know Slava. Probably you do. Everyone knows Slava. Recently he contributed a bunch of language translations even though they're somewhat dated, for Dataverse 4.9.4. This is totally fine.
12:33 MrK
Ah yeah recently we talked about it in the team, it is beeing prepared when I heard about it last time.
12:35 pdurbin
MrK: ok, so you do plan to contribute a Polish translation at some point. Want me to create an issue for this?
12:36 MrK
I think we had some issue already, and yeah we wanted to give out at least 4.9.4 polish translation.
12:37 pdurbin
I created one in Slava's repo. I can go create one in the new repo, now that it exists.
12:40 pdurbin
Or you can. But I'm happy to. :)
12:48 donsizemore joined #dataverse
12:48 pdurbin
donsizemore: good morning. Talk to me.
12:50 donsizemore
@pdurbin what would you like to talk about? (i'm still dealing with backup screwiness but will put in your docker cp for easily viewable server.logs this morning)
12:51 pdurbin
yum!
12:51 pdurbin
Looking forward to the docker cp.
12:51 donsizemore
@pdurbin also, andrey is low-key crushed that his DataverseML proposal didn't get selected so any feedback on that decision would put him at ease
12:51 pdurbin
Who crushed him? NSF?
12:53 MrK
Since I'm lazy I wouldn't mind issue created :P.
12:55 pdurbin
MrK: sure, here you go: https://github.com/GlobalDataverseCommunityConsortium/dataverse-language-packs/issues/26
13:02 pdurbin
donsizemore: wait, did we crush him? I'm looking at http://irclog.iq.harvard.edu/dataverse/2019-04-04#i_89952
13:04 donsizemore
@pdurbin perhaps "crushed" is a strong term, let's go with low-key anxiety over why it didn't get selected
13:05 pdurbin
I can't find his proposal.
13:05 donsizemore
@pdurbin i suggested that the DCM is geared more towards archivists while DataverseML would be of more interest to researchers
13:05 donsizemore
@pdurbin he did mention yesterday that he was pretty sure he submitted it
13:05 pdurbin
I don't see it.
13:05 pdurbin
Did he get an RT ticket number?
13:07 pdurbin
When I submitted my "crazy ideas" talk proposal I got an email with this link: https://help.hmdc.harvard.edu/Ticket/Display.html?id=275884
13:08 pdurbin
So he should have gotten an email like that.
13:08 donsizemore
i'll ask him when he gets in
13:08 pdurbin
ok, thanks
13:09 pdurbin
donsizemore: meanwhile, can I please bring you up to speed with Payara and https://github.com/IQSS/dataverse/issues/4172 ?
13:16 donsizemore
yes sir
13:17 donsizemore
oh. where would you want server.log to end up? beneath tests/ ? target/ ?
13:17 pdurbin
donsizemore: for now, anywhere that I can reach from a web browser without logging in.
13:18 donsizemore
you can see the entire workspace tree, so i'll dump it in tests
13:18 pdurbin
awesome, thanks!
13:18 donsizemore
or better yet conf/docker-aio/
13:18 pdurbin
whatever works
13:19 donsizemore
i added this line to the end of run-test-suite.sh
13:19 donsizemore
docker cp dv:/usr/local/glassfish4/glassfish/domains/domain1/logs/server.log conf/docker-aio/
13:19 donsizemore
so it should run immediately after the maven call. if you find that friendly i'll submit a PR
13:20 donsizemore
what's up with payara5? i saw lots of convo from @poikilotherm
13:20 pdurbin
donsizemore: for Payara, I'm wondering if I can co-assign you to that issue so that poor poikilotherm has a buddy. He did some great research into the wee hours last night on why the Dataverse JSF code (the front end) is so unhappy with (apparently) anything newer than the ancient Glassfish we require.
13:23 pdurbin
MrK: I'm happy to co assign you as well if you'd like to help us get off Glassfish 4.1. :)
13:25 donsizemore
i'm happy to be assigned for whatever use i may be
13:25 donsizemore
i'm no @poikilother
13:25 MrK
I'm not sure if we are gonna try getting of glassfish 4.1 to payara/glassfish 5 or try switching to spring later.
13:28 pdurbin
MrK: ok, but please make pull requests. :)
13:29 pdurbin
donsizemore: sure but if poikilotherm give you a URL to a jar we might want to patch like we do for weld and grizzly, I'm wondering if we should add it to dataverse-ansible as a option. To patch it or not, I mean. To test a theory we have.
13:30 donsizemore
@pdurbin absolutely. starting with the one ^^ /
13:31 donsizemore
with more coffee i may be able to type more accurately
13:38 juancorr joined #dataverse
13:41 pdurbin
donsizemore: great! I'll keep you posted on if we'd like that patch option or not. Thanks!
13:46 poikilotherm
I'm sorry guys, I am underperformer today... Having headaches :-(
13:46 poikilotherm
Maybe the weather :-/
14:11 pdurbin
poikilotherm: hope you feel better
14:26 jonas4711 joined #dataverse
14:31 Venki joined #dataverse
14:32 Venki
Hello guys
14:32 Venki
Good morning
14:33 Venki
Phil I would like to know the java code for getting all users in the dashboard
14:35 Venki
What is the code that is querying the dB? I would like to know whether the pagination is done at code level after all the user records are pulled from the database. or is it that we pull only 25 records from dB at a time.
14:36 Venki joined #dataverse
14:38 Venki joined #dataverse
14:39 pdurbin
Venki: hi! Sure, I can show you the code.
14:39 Venki
Thanks Phil
14:41 donsizemore
@pdurbin i made a note to dig out that jar and add a switch for it. where's my Round Tuit?
14:41 Venki joined #dataverse
14:42 pdurbin
Venki: here's the "next" button: https://github.com/IQSS/dataverse/blob/v4.14/src/main/webapp/dashboard-users-pager.xhtml#L63
14:43 pdurbin
Venki: which calls into runUserSearchWithPage at https://github.com/IQSS/dataverse/blob/v4.14/src/main/java/edu/harvard/iq/dataverse/dashboard/DashboardUsersPage.java#L72
14:43 pdurbin
donsizemore: ok, but I'm not sure if we need it yet
14:43 pdurbin
Venki: does that help?
14:44 Venki
I saw this but I am asking for the database query that was written to pull this?
14:44 Venki
Is it paginated at the dB query or we pull all records and do the pagination?
14:46 Venki joined #dataverse
14:47 Venki
I see that the UserListResult is called finally but not sure where is the dB code? Is it a stored proc?
14:49 pdurbin
Venki: I'm still digging. What problem are you having? Is it slow? How many users do you have?
14:49 Venki
Oh ok. Apologies for having not mentioned that
14:50 Venki
I am trying to solve the problem of sorting the users by ID. I found that you guys are using prime faces and it has a Data exporter function which allows to export the data from the table
14:51 Venki
If I include the prime face code for data exporter it exports only the current page and the not the whole table
14:51 Venki
also when I use the sort function of prime face it only sorts the first page with 25 records.
14:52 Venki
I see that the API also works the same way
14:52 Venki
We have only 450 records in the prod server and 75 records in the development
14:52 pdurbin
Venki: ok. Thanks. And I think I found the SQL query. getUserListCore at https://github.com/IQSS/dataverse/blob/v4.14/src/main/java/edu/harvard/iq/dataverse/UserServiceBean.java#L69
14:53 Venki
Oh that’s good.
14:54 pdurbin
I hope that's the right one. I think it is. I didn't write this code. :)
14:55 Venki
Ok my intention is to use that code to assign the value to the data table directly
14:55 Venki
Not sure whether it will work
14:57 Venki18 joined #dataverse
15:00 Venki18 joined #dataverse
15:00 Venki18
Thanks Phil
15:00 Venki18
Will test it out and let you know
15:01 pdurbin
Venki18: sounds good. Please feel free to open an issue about this.
15:01 Venki18
Sure will do that
15:02 Venki18
Phil just for your info, we did a crude way of getting google to index our CC-By-NC
15:04 Venki18
dataset.xhml file has a output field with jsonLd value. We have replaced the value of CC0 with CC-BY-NC and also the URL .
15:04 Venki18
Hope Google dataset can index this correctly..
15:06 Venki18
Thanks Phil for your help.
15:07 pdurbin
Venki18: interesting. :)
15:08 pdurbin
Venki18: speaking of your installation, did you see this tweet? I love it: https://twitter.com/suzyjstyles/status/1136865927171940352
15:08 pdurbin
donsizemore: the coloring fun ^^
15:10 donsizemore
@pdurbin i have my scented "Mr. Sketch" markers here on my desk!
15:11 donsizemore
(which a co-worker's daughter used for copious amounts of coloring yesterday)
15:11 pdurbin
nice
15:35 pdurbin
donsizemore: I just left a comment at https://github.com/IQSS/dataverse/pull/5933#issuecomment-501328898 about the potential jar to patch
15:36 pdurbin
The theory is that if we patch Glassfish 4.1 with javax.el-api-3.0-b05.jar (I hope that's the right jar, I'm pretty confused) that we will start to see the same problems with JSF that we've been seeing on newer versions of Glassfish or Payara. Does that make sense?
17:14 donsizemore
@pdurbin he wanted 3.0-b05? http://central.maven.org/maven2/javax/el/javax.el-api/3.0.0/
17:16 pdurbin
donsizemore: as an optional patch. This is more something I was thinking about playing with. To understand the source of the change. Which we think is that jar file. I guess. :)
17:16 pdurbin
lots of uncertainty here
17:17 pdurbin
Oh, I forgot to assign you to that issue.
17:17 pdurbin
done :)
17:18 pdurbin
ah, you opened https://github.com/IQSS/dataverse-ansible/issues/72
17:18 pdurbin
thanks :)
17:19 donsizemore
i ask because he mentioned 3.0-b05 but it hit 3.0.0 apparently in 2013
17:20 pdurbin
Hmm.
17:20 pdurbin
Yeah, I dunno.
17:20 pdurbin
I'm leaning heavily on his notes. :)
17:25 donsizemore
and i don't find a javax.el-api.jar beneath glassfish4/, only javax.el.jar. so this is an addition rather than a swap
17:26 pdurbin
Ok, I might be linking you to the wrong thing. :(
17:26 donsizemore
there's a javax.el jar, but when i unzip it i don't find a reference to any api
17:27 pdurbin
I'm running `/usr/local/glassfish4/bin/asadmin osgi lb` but I'm not even sure what I'm looking for.
17:28 pdurbin
I see this in the output:
17:28 donsizemore
there's a javax.el.jar and a javax.batch-api.jar, and i see those loaded
17:28 pdurbin
137|Resolved | 1|Expression Language 3.0 (3.0.0)
17:30 pdurbin
ok
17:30 pdurbin
Your guess is as good as mine.
17:30 pdurbin
And maybe it's not such a hot idea to do additional surgery on Glassfish 4.1 just to test something. :)
17:33 pdurbin
donsizemore: I'm pretty sure /usr/local/glassfish4/glassfish/modules/javax.el.jar is the one. If you unzip it you can find version=3.0.1-b03 in META-INF/maven/org.glassfish/javax.el/pom.properties
17:35 donsizemore
it's easy enough to test
17:35 pdurbin
If you don't mind, please go for it. :)
18:18 donsizemore
okay, i've got a branch that adds javax.el-api.jar to glassfish (but doesn't wipe out the much larger javax.el.jar)
18:18 pdurbin
and it's optional?
18:23 pdurbin
I'm confused that there are two jars.
18:35 jri joined #dataverse
18:41 pdurbin
donsizemore: can you please delete /tmp/sitemap.xml.staged ?
18:48 donsizemore
done. i'll add it to the tmp-cleaner
18:49 pdurbin
thanks!
19:57 jonas4711 joined #dataverse
21:10 jri joined #dataverse
22:13 dataverse-user joined #dataverse
22:19 pdurbin joined #dataverse
