IQSS logo

IRC log for #dataverse, 2019-06-11

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

All times shown according to UTC.

Time S Nick Message
07:13 juancorr joined #dataverse
07:16 jri joined #dataverse
10:10 juancorr joined #dataverse
10:52 poikilotherm joined #dataverse
11:03 poikilotherm Mornin' folks :-)
11:03 poikilotherm Sorry I wasn't in yesterday, it has been a holiday :-)
11:04 poikilotherm (Pentecost)
11:05 pdurbin Thanks for replying on https://github.com/IQSS/dataverse-kubernetes/issues/65
11:05 pdurbin I had no idea what to tell him. :)
11:05 poikilotherm :-D
11:06 poikilotherm Thanks for trying to support this community initiative :-)
11:06 poikilotherm pdurbin: do you know if files will be cached or similiar?
11:07 poikilotherm It would be totally easy when they are delivered freshly when changed on disk
11:08 pdurbin My main thought on dataverse logos is that I left a note about them here: http://guides.dataverse.org/en/4.14/installation/advanced.html#multiple-glassfish-servers
11:09 pdurbin When users upload a logo for their dataverse using the “theme” feature described in the Dataverse Management section of the User Guide, these logos are stored only on the Glassfish server the user happend to be on when uploading the logo. By default these logos are written to the directory /usr/local/glassfish4/glassfish​/domains/domain1/docroot/logos.
11:09 poikilotherm Meh.
11:09 pdurbin Is this a bug in Dataverse? Something we should improve?
11:10 poikilotherm Let's not call it a bug
11:10 pdurbin heh. ok
11:10 poikilotherm So people can upload an image via admin UI?
11:10 pdurbin Anyone who can create a dataverse.
11:10 poikilotherm Err...
11:10 pdurbin (When you create one you become an "admin" of it.)
11:10 poikilotherm Ah I misread
11:11 poikilotherm I only thought about navbar logo
11:11 poikilotherm Ok this definitly is bad.
11:11 pdurbin Let's call it a bug. :)
11:11 poikilotherm I remember we talked about logos being stored in dataverse.files.directory
11:11 poikilotherm Which seems to be wrong
11:12 poikilotherm As they are stored in the docroot
11:12 pdurbin I don't remember but iqlogbot might.
11:12 poikilotherm My memories might have been tampered with
11:12 poikilotherm No guarantees ;-)
11:13 poikilotherm Wouldn't it make sense to store the logos within the filespace? Either local or object store?
11:13 poikilotherm That way multiple servers wouldn't need to sync
11:16 poikilotherm I wasn't aware of the problem about Dataverse logos, I just had the logos in my mind.
11:16 poikilotherm Would you be so kind to add a comment to dataverse-kubernetes#65?
11:16 pdurbin I can at least copy and paste what I wrote in the guide. :)
11:17 poikilotherm Some more sugar would be appreciated :-)
11:17 poikilotherm Perhaps some icing
11:17 pdurbin The problem is that breakfast is starting to happen here. :)
11:17 poikilotherm Like "be aware", "danger: uploads ahead"
11:17 poikilotherm Take your time :-)
11:18 poikilotherm And enjoy :-)
11:18 poikilotherm (Timezones. Who ever invented those should be sued.)
11:21 pdurbin Left a brief comment. Now coffee.
11:23 poikilotherm "Brief". Man, that's 20-25 lines... When this is "brief", what is "detailed" for you? :-O
11:26 pdurbin poikilotherm: oh, here's yet another doc about chat that you can comment on if you'd like: https://docs.google.com/spreadsheets/d/1lBm5HYhi0RFbfSYhsl2Is5z35PhOV2h1S8ZGIVoUSj8/edit?usp=sharing :) "Dataverse Community Chat Requirements"
11:55 donsizemore joined #dataverse
12:28 pdurbin andrewSC bjonnh bricas_ candy`_ donsizemore dzho icarito[m] JonathanNeal jri juancorr pmauduit poikilotherm rigelk sivoais: do any of you know enough PHP to help with https://groups.google.com/d/msg/dataverse-dev/gluSLNay-7Q/218wUbVtAAAJ ?
12:30 pmauduit seems that it is more of a curl concern than php, maybe it requires a proxy configuration ?
12:31 pdurbin Well, the error is triggered for any API endpoint that doesn't exist. For example: https://demo.dataverse.org/api/doesNotExist
12:31 pmauduit it might also require to set the Host header, and maybe the script is using a too old version of libcurl which does not implement SNI (but it would be a very old version then), so it is kindof unlikely, but we never know :)
12:31 pdurbin So I don't *think* it's a proxy thing.
12:32 jri It isn't just a path URI issue ?
12:32 jri http://guides.dataverse.org/en/latest/api/native-api.html#id32
12:33 jri is this doc there /datasets/ avec the DV name
12:33 poikilotherm Well there is a lack of info regarding the Dataverse version...
12:33 poikilotherm And a testable URL would be cool
12:34 poikilotherm So we can check if the problem is on the Dataverse installation side
12:34 poikilotherm And of course this should be tried with cURL on CLI to avoid possible problems with PHP
12:34 poikilotherm Sometimes there are strange cURL problems with PHP wrapper around it.
12:35 pdurbin jri: ah, good catch. I think you're right that the person doesn't have "/datasets/" in the path. Do you want to write to this person? I'm happy to write if that's easier for you. :)
12:35 poikilotherm And IIRC on windows PHP ships curl on its own, which might also be a problem
12:38 poikilotherm Meh. pdurbin: ORCID came back to me regarding production credentials...
12:38 poikilotherm They want UI changes :-(
12:38 pdurbin Can you please tell them to open an issue? :)
12:38 poikilotherm LOL
12:39 poikilotherm Are you aware of someone with a production instance + ORCID login?
12:39 jri pdurbin: I not sure to quickly remember my google account and password to answer and unfortunately I don't have many time right now. if it does not bother you I would like you to answer him ?
12:39 pdurbin jri: I'll write him quick and give you credit. :)
12:40 jri Ok thanks pdurbin :) Let us know if it wasn't the problem
12:41 pdurbin jri: will do, thanks again!
12:42 pdurbin poikilotherm: yes, ORCID login is used in production at https://dataverse.harvard.edu and probably some others.
12:42 pdurbin poikilotherm: all 44 installations are in all-dataverse-installations.json at https://github.com/IQSS/dataverse-metrics if you'd like to poke around and find others using ORCID login. :)
12:43 poikilotherm Well I will first let her know that you guys are running this and she can take a look at the audit of your installation... ;-)
12:44 poikilotherm Thx
12:47 pdurbin Sounds good. Pete is too but his installation isn't on the map yet.
12:49 poikilotherm pdurbin: shall I put you in CC?
12:49 pdurbin poikilotherm: please don't. You could forward it to support@dataverse.org if you want.
12:49 poikilotherm Alright
12:50 pdurbin donsizemore: jenkins jenkins jenkins
12:53 donsizemore @pdurbin good morning. where did we leave off?
12:53 pdurbin donsizemore: I think I owe you some troubleshooting BUT I also wanted to back up and explain my thinking AND ask if we can merge one of my open pull requests.
12:54 donsizemore i owe you more troubleshooting because i was in the datacenter most of yesterday afternoon
12:55 pdurbin the best place on a hot day
13:00 pdurbin donsizemore: any objection to merging https://github.com/IQSS/dataverse-jenkins/pull/9 ?
13:00 donsizemore on my boxen here, i can re-run the same job on the same repo and hit a different test failure each time
13:01 donsizemore @pdurbin done =)
13:02 pdurbin donsizemore: thanks! And yes, the flakiness of the api test suite is what I want to talk about. This issue, basically: https://github.com/IQSS/dataverse/issues/5827
13:03 pdurbin I was on vacation when the last release came out but I hear we had a heck of a time making the phoenix server happy, to get the test passing, before release. A lot of "sleep" lines were added if I'm not mistaken.
13:05 pdurbin Obviously, adding "sleep" isn't great but I think we might need to add some more to get the api test suite passing for the IQSS-dataverse-api-test-suite-docker-aio-develop job at https://jenkins.dataverse.org (also, I'm happy to talk naming conventions for jobs when you're ready).
13:06 pdurbin donsizemore: I'm wondering how best to add the "sleep" statements. Want me to experiment in /home/pdurbin? Or do you want to hack around a bit?
13:12 poikilotherm Hey guys...
13:13 poikilotherm I just gave REST assured a look
13:13 poikilotherm Are you aware that 4.0.0 has been released lately?
13:13 poikilotherm And at least 3.3.0 in January?
13:14 poikilotherm I stumbled over the imports names, using com.jayway, not io.restassured
13:14 poikilotherm Maybe your timing issues are related to some old bug?
13:14 poikilotherm (within rest assured)
13:16 poikilotherm So at least one should try to get on 2.9.0 which is the latest version using the old imports
13:18 donsizemore @pdurbin i too suspected timing issues, but you may know better than i where to add sleep statements
13:19 pdurbin donsizemore: ok, I'll see if I can make a pull request or two. Thanks.
13:20 pdurbin poikilotherm: I think the big difference is that with phoenix the rest assured tests are executed over the wire from the jenkins server to the phoenix server. Now donsizemore and I are trying to execute the same tests using docker-aio so there may be less latency? That's my theory anyway. And no, I didn't know version of is out. Thanks.
13:20 pdurbin of rest assured
13:21 poikilotherm I am not very deep into REST tests, but the test cases are somewhat strange.
13:21 poikilotherm Most of 'em create something
13:21 pdurbin yeah
13:21 poikilotherm And after that, they read back
13:21 poikilotherm Try things
13:21 poikilotherm etc
13:22 pdurbin yeah
13:22 poikilotherm So in case of delays, this will break
13:22 pdurbin yeah
13:22 poikilotherm But are tests relying on having more or less no delay not broken by design?
13:22 poikilotherm At least in a REST world, you have to except faulty connections, etc
13:22 pdurbin the design is bad, I agree
13:23 pdurbin it's my fault
13:23 pdurbin please help :)
13:23 poikilotherm Sure :-)
13:23 poikilotherm Already trying to be helpfull ;-)
13:24 pdurbin some actions like ingest will always take a little while
13:24 poikilotherm And I don't blame you - happy that you dived into this in the first place!
13:24 pdurbin We basically had zero tests for DVN 3.
13:24 pdurbin So I'm actually very happy with what we have. :)
13:24 poikilotherm Yeah :-
13:25 poikilotherm :-)
13:25 pdurbin Even if the tests are poorly designed. :)
13:25 poikilotherm Well, those can be refactored
13:25 pdurbin Yes.
13:25 poikilotherm I'd expect REST assured to have some example for this common use case
13:25 pdurbin You saw my thoughts on automated testing, right? https://groups.google.com/d/msg/dataverse-dev/ISot5k4VjZQ/t-hzPk8tAwAJ
13:25 * poikilotherm takes a shouvel, a lamp and puts his helmet on
13:26 poikilotherm Let's go looking for some gold!
13:26 pdurbin donsizemore and I have been working on step 1 of my goals
13:26 poikilotherm I think I saw this, but it got lost on track...
13:27 pdurbin I only listed 4 goals. I agree with your goal but I think it's number 5 or 6 in terms of my priorities.
13:27 pdurbin But everyone is welcome to have their own priorities. :)
13:27 poikilotherm Well, when you are in bad luck, you will not solve step 1 without this :-/
13:28 poikilotherm As stated before, I am not all into these tests, but it makes sense for me that on cramed boxes, VMs, etc etc tests might fail due to timing.
13:28 pdurbin I'm feeling pretty good that donsizemore and I can achieve step 1 with some sleep statements. That's my hope anyway.
13:28 poikilotherm Sound great
13:29 poikilotherm Maybe you should do that step and reconsider looking into refactoring before adding test for merge requests?
13:30 poikilotherm Because before adding more tests in step 4, one might be interested in learning how to write more reliable tests... ;-)
13:30 pdurbin If someone would like to volunteer to refactor tests instead of adding sleep statements that would of course be better but https://github.com/IQSS/dataverse/issues/5827 isn't in our sprint. I'm just hacking on it on the side because I believe so strongly that we must improve our automated testing situation.
13:30 poikilotherm Ok, good to know
13:31 pdurbin This is the issue that's in the sprint that I'm working on: https://github.com/IQSS/dataverse/issues/5235 . New pull request at https://github.com/IQSS/dataverse/pull/5931
13:35 poikilotherm pdurbin I like your stuff in https://github.com/IQSS/dataverse-sample-data
13:35 poikilotherm Shall I create a K8s job for this?
13:35 pdurbin poikilotherm: sure! Just please be aware that it's very much a moving target.
13:36 pdurbin donsizemore: I was going to ask if you want this in dataverse-ansible too.
13:37 donsizemore @pdurbin okay, i'm back. a co-worker's daughter is here today coloring at my desk, and another co-worker was asking about the DCM schedule
13:37 donsizemore @pdurbin yes, i started on an api-test-suite branch for dataverse-ansible but i'd have to review it to tell you the status
13:38 pdurbin donsizemore: you must have her color smiling brain cells from the PDF hosted by the NTU Dataverse installation: https://twitter.com/suzyjstyles/status/1136865927171940352
13:38 poikilotherm pdurbin I just created https://github.com/IQSS/dataverse-kubernetes/issues/66
13:38 poikilotherm Is there an upstream issue I should reference?
13:38 donsizemore oh, sample-data. yes it'd be better than my kludgy readdir scripts
13:38 poikilotherm (Despite #5235)
13:39 pdurbin poikilotherm: looks great. You can reference https://github.com/IQSS/dataverse/issues/5235 since that's the one I'll put into code review when I'm sick of hacking on the code.
13:39 poikilotherm Ok, but from what I can see, this is only a doc change?
13:39 pdurbin donsizemore: I'm so sorry for asking you to do all that sample data work. I'm still hoping to use some of it.
13:39 poikilotherm Is there an issue for the creation/work on the sample data repo?
13:40 pdurbin poikilotherm: well, there's code in the new dataverse-sample-data repo.
13:40 poikilotherm Alright
13:40 pdurbin poikilotherm: what matters at standup is how the work is represented as a card on our kanban board: https://github.com/orgs/IQSS/projects/2
13:41 poikilotherm Not much time left for the Sprint... ;-9
13:41 pdurbin In 90 minutes or so I'll ask Mike, our front end guy, to see if he can get the code in the new repo working.
13:42 pdurbin We never finish all the stuff in our sprint.
13:44 pdurbin I think a separate repo makes sense. I'll find out what others think soon. :)
13:45 pdurbin poikilotherm: did you see the recent discussion on https://github.com/IQSS/dataverse/pull/5913 ? I'd still love for you to help me get Dataverse running on Payara 5. That's another side project that isn't in our sprint. :)
13:46 poikilotherm Yeah I saw that.
13:46 poikilotherm (On my mobile)
13:46 poikilotherm Thx for pinging
13:46 poikilotherm Wanted to look at the code
13:47 pdurbin 5907-omnifaces-page0-00fb3d0 (that pull request) is deployed to http://ec2-34-207-192-222.compute-1.amazonaws.com:8080 if anyone would like to poke at Dataverse running on Payara 5. All kinds of stuff is broken, I fear. JSF stuff. :(
13:48 poikilotherm I will give it a whirl in a second. Trying out my new payara dev build :-D
13:48 poikilotherm I really hope for them to release their new docker image soon
13:48 pdurbin nice!
13:48 poikilotherm Much smaller...
13:49 poikilotherm And Payara is eating my RAM for breakfast, lunch and dinner all at the same time.
13:50 pdurbin poikilotherm: if it helps, I see 5.192 at https://github.com/AdamBien/docklands/blob/0c517f58c43cb4a5d921cf0d5c6d09b9f2c31a93/payara-micro-s2i/Dockerfile
14:00 poikilotherm pdurbin: I am not all into JSF, but those #{} are EL expressions, right?
14:01 poikilotherm E. g. <f:viewParam name="page" value="#{SearchIncludeFragment.page}"/>
14:09 jonas4711 joined #dataverse
14:15 pdurbin poikilotherm: yes. "expression language": https://docs.oracle.com/javaee/7/tutorial/jsf-el.htm
14:15 poikilotherm Right
14:15 poikilotherm I have a suggestion
14:15 poikilotherm Err.
14:15 pdurbin I'm not at all into JSF either, for the record. :)
14:16 poikilotherm Suspicion
14:16 poikilotherm Lemme get you some line numbers
14:16 pdurbin ok
14:17 poikilotherm https://github.com/IQSS/dataverse/blob/24d4209e372bdfb8681d5e4dc7d6ab6f39fdeca9/src/main/webapp/search-include-pager.xhtml#L43
14:17 poikilotherm That foreach is creating the pagination links
14:17 poikilotherm I wondered where within the code is some sort of calculus for the page attribute
14:18 poikilotherm And the only place, where the getPage() of SearchIncludeFragment is not used, is here
14:18 poikilotherm In all other places I could find, there is always some Math:max() to ensure this is 1
14:18 poikilotherm And not 0
14:18 poikilotherm Now here we go
14:19 poikilotherm Looking at https://www.codejava.net/java-ee/jstl/jstl-core-tag-foreach
14:19 pdurbin "Begin index of the iteration. Iteration begins at the value mentioned in this attribute value. (if items specified) First item has index of 0."
14:20 poikilotherm There is no items specified!
14:20 pdurbin poikilotherm: can you fix it, doctor?
14:21 poikilotherm Actually I am not sure.
14:21 poikilotherm As there is no "items", why is this working anyway?
14:21 pdurbin /shrug
14:21 pdurbin no idea how any of it works :)
14:21 poikilotherm Is this valid code?
14:21 poikilotherm What happens, when you are not providing items
14:21 poikilotherm Actually, this is a "foreach"
14:22 poikilotherm Not providing items only makes limited sense
14:22 poikilotherm But I see that you guys want to have sth. done a few times (you calculate it)
14:22 poikilotherm Is there a for?
14:26 pdurbin If there's a for we don't seem to use it.
14:27 poikilotherm Quick googling tells me, there is none, at least not "for"
14:27 poikilotherm Reading more...
14:27 * poikilotherm is processing input
14:28 pdurbin poikilotherm: I'm about to bike (through the rain) to work but here's something you might want to tell the German SRE community about: https://discourse.opensourcedesign.net/t/open-science-event-at-wikimedia-deutschland-june-12-2019/1107 . Or RSE? Yes, I mean this, I think: https://www.de-rse.org
14:28 pdurbin It's tomorrow in Berlin.
14:34 poikilotherm Ok, I think I found it.
14:34 poikilotherm foreach can be used with begin and end
14:35 poikilotherm But you will need to provide a var argument
14:35 poikilotherm that will contain the current value
14:35 poikilotherm <c:forEach var="col" begin="0" end="255">
14:35 poikilotherm <span style="color:rgb(${col},${col},${col})">${col}</span>
14:35 poikilotherm </c:forEach>
14:36 poikilotherm Current versions seem to be more strict... Maybe in 4.1, the index value started with 1?
14:36 poikilotherm There is .index and .count
14:36 poikilotherm .index is 0 based, .count 1 based
14:38 poikilotherm pdurbin: how shall I get a fix in your branch?
14:47 poikilotherm https://github.com/IQSS/dataverse/pull/5913#discussion_r292496657
14:48 poikilotherm I need to pick up kids
14:48 poikilotherm pdurbin: this is not yet tested.
14:48 poikilotherm And obviously you would need to revert your commit adding the logic for page = 0
14:48 poikilotherm If this mitigates the problem
14:49 poikilotherm Could you please test this? A big thank you in advance
14:49 poikilotherm See you guys tomorrow
14:54 jonas4711 joined #dataverse
15:07 pdurbin I will take a look. Thanks!!
15:10 pdurbin donsizemore: what should we do about all these files in /tmp? Can we put off working on https://github.com/IQSS/dataverse/issues/5885 and have a script that blows them away that I can run as "pdurbin" for now?
15:17 pdurbin donsizemore: I just created https://github.com/IQSS/dataverse-jenkins/issues/10
18:50 pdurbin donsizemore: still there?
18:54 donsizemore @pdurbin yes =) back, rather.
18:54 donsizemore i'm happy to drop that in the crontab. we'll want to rm -rf /tmp/dv as well
18:55 donsizemore unless that portion of the build script cleans up after itself
18:55 donsizemore those files will be a non-issue once jenkins is doing all the work
18:56 pdurbin Sure, but they're blocking me right now. :)
18:56 donsizemore they're nuked
18:56 pdurbin Thanks! Lemme try another run of the api test suite.
19:01 donsizemore @pdurbin i was about to ask and see if you wanted me to fire off my script
19:02 pdurbin Sorry, which script?
19:02 * pdurbin has a lot of scripts
19:04 donsizemore i did a declare -a TMPFILES=("sitemap.xml" "sushi_sample_logs.json" "dvjsonld.json") with a for loop
19:10 pdurbin fancier than my version :)
19:11 donsizemore so your tests are passing, mine are failing, same source, same box?
19:13 pdurbin Well, no, I'm getting failures in /home/pdurbin on your Jenkins too. But I wonder if it's from something we just merged: FileTypeDetectionIT.testRedetectMimeType
19:27 pdurbin Oh, I need to go add that test to phoenix.
19:30 pdurbin donsizemore: is there a smart way to pull server.log out of the docker image and dump it into https://jenkins.dataverse.org/job/IQSS-dataverse-api-test-suite-docker-aio-develop/ws/ ? And maybe rename it to server.log.txt so I can easily see it in from my browser.
19:36 donsizemore we could wedge a 'docker cp' into the workflow? assuming the container name is always dv0 or some such
19:42 pdurbin sounds fine
19:46 pdurbin Want me to open an issue?
21:07 donsizemore i jotted down a note to do it but you're welcome to open an issue. proper github etiquette and all that
21:17 poikilotherm joined #dataverse
21:18 jri joined #dataverse
21:21 poikilotherm pdurbin: riiiinnnngggg
21:21 poikilotherm Still around?
21:50 jonas4711 joined #dataverse
21:59 jonas_ joined #dataverse
22:42 poikilotherm pdurbin: taking some notes while digging. https://iffmd.fz-juelich.de/s/SJFijspC4#
22:58 pdurbin poikilotherm: "You are not allowed to access this page. Maybe try logging in?"
22:59 poikilotherm Oh forgot to switch perms
22:59 poikilotherm Try agai
23:00 pdurbin Heh. "string madness"
23:00 pdurbin Thank you for digging in on this!
23:02 poikilotherm It all looks like we have been on unmaintained Glassfish 4.1 for way too long
23:03 poikilotherm This stuff was hot in 2014/2015
23:03 poikilotherm Is your AWS instance still running?
23:06 pdurbin poikilotherm: yep: http://ec2-34-207-192-222.compute-1.amazonaws.com:8080 ... and still running payara5-foreach-e4946bc
23:06 poikilotherm err login?
23:07 pdurbin dataverseAdmin/admin1
23:07 poikilotherm Ah, dataverse-ansible, I forgot
23:07 poikilotherm TADA
23:07 * poikilotherm raises a fist from the grave like a zombie
23:08 poikilotherm I dug my way out
23:08 poikilotherm http://ec2-34-207-192-222.compute-1.amazonaws.com:8080/dataverse.xhtml?page=1
23:08 pdurbin Yeah, admin1 comes from https://github.com/IQSS/dataverse-ansible/blob/d1dcc4cd83db536de38bdc1c4ae04abb2741eed5/defaults/main.yml#L23
23:09 pdurbin Not sure what you mean. Yes, one can pass page=1 as a query parameter.
23:09 poikilotherm Please compare the results :-D
23:09 poikilotherm Once without parameter, once with parameter
23:10 pdurbin Sure, yes.
23:10 pdurbin But now what?
23:10 pdurbin uh oh, dinner
23:10 poikilotherm Enjoy!
23:10 poikilotherm I am outta here... Get some sleep
23:11 poikilotherm Plz take a look at my findings later / tomorrow
23:12 pdurbin What findings?
23:12 pdurbin I'm confused.
23:12 poikilotherm https://iffmd.fz-juelich.de/s/SJFijspC4#Error-500-because-page--0
23:18 poikilotherm pdurbin I will refactor this a bit tomorrow and add it as a comment to https://github.com/IQSS/dataverse/pull/5913
23:19 poikilotherm Maybe we should change my commit message :-D
23:19 poikilotherm Good night
23:43 pdurbin Oh, those findings. Cool.

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.