IQSS logo

IRC log for #dataverse, 2019-05-13

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

All times shown according to UTC.

Time S Nick Message
03:17 jri joined #dataverse
07:13 jri joined #dataverse
07:37 juancorr joined #dataverse
08:54 MrK joined #dataverse
09:41 pdurbin joined #dataverse
10:03 poikilotherm joined #dataverse
10:10 pdurbin poikilotherm: interesting that Slava might switch to your Docker images: https://github.com/IQSS/dataverse-docker/issues/18#issuecomment-491715460
10:11 pdurbin MrK: mornin'
10:11 poikilotherm Morning pdurbin
10:11 poikilotherm He wrote me an emai l: _)
10:11 pdurbin Ah, ok.
10:11 pdurbin probably makes sense to join forces :)
10:12 poikilotherm Yeah ;-)
10:12 poikilotherm He wanted to take a look and see how it goes
10:13 pdurbin Does he have any particular needs, such as running multiple Glassfish servers?
10:14 poikilotherm He also asked about running Selenium based integration tests on these deployments
10:14 pdurbin nice
10:14 poikilotherm If I got it right, he wanted those to ensure that the deployment works as expected
10:14 poikilotherm I told him that I would be happy to run Selenium tests on this, but those should come or be commited to upstream
10:15 poikilotherm Because those should also help devs and QA
10:15 poikilotherm He didn't tell me about special needs
10:15 pdurbin Yes, it would be great for those Selenium tests to be contributed upstream.
10:16 poikilotherm Oh  there is a "from" missing above... "come from"
10:17 pdurbin They could replace old selenium tests like this one that hasn't been touched or run in four years: https://github.com/IQSS/dataverse/blob/v4.14/tests/create_account.py
10:22 poikilotherm Yeah. IMHO such things should be implemented in Selenium ruled by JUnit
10:23 poikilotherm (Maybe automated / supported with things like Arquillian)
10:24 pdurbin Ah, instead of Python.
10:24 poikilotherm Stay within the same eco-system, you know ;-)
10:25 pdurbin Couldn't hurt. But Python is fine too.
10:58 poikilotherm Oi pdurbin: any preferences on where to put some notes about the dataverse kubernetes installation adventure?
10:59 poikilotherm I am not sure if it fits inside "Advanced installation"
10:59 poikilotherm I dunno if it might be the right time to refactor this part
11:00 pdurbin It's definitely fine to refactor. Do you plan to put https://github.com/IQSS/dataverse/issues/4665 through QA?
11:01 poikilotherm Uff. Dunno. I thought about creating a new issue
11:01 pdurbin sure, a new issue is fine
11:01 poikilotherm Maybe some day we will need to abort 4665 ;-)
11:01 poikilotherm The original owner seems not to be responsive...
11:02 pdurbin Want me to unassign you?
11:03 poikilotherm What do you prefer?
11:03 poikilotherm Is there a standard process at IQSS how to deal with stale issues?
11:03 pdurbin I prefer for issues to be closed, eventually. :)
11:04 pdurbin Not really. I have a couple labels I sometimes throw on issues. "Status: Still interested?" and "Vote to close: pdurbin" :)
11:04 poikilotherm :-D
11:06 poikilotherm If you want, I can come up with a branch for the issue and add some docs
11:06 poikilotherm Then it might hurt less to close the issue :-d
11:07 pdurbin I believe the creator of that issue is in production already at http://data.mel.cgiar.org
11:07 pdurbin 42nd installation of Dataverse: https://groups.google.com/d/msg/dataverse-community/_1qOQqQRXEA/-7SmtE_jAQAJ
11:08 pdurbin running 4.8.5
11:08 poikilotherm Yeah
11:10 poikilotherm So this might be a plan how to get it resolved - finally?
11:10 pdurbin I just left a comment and added the "still interested?" label.
11:11 pdurbin If your pull request makes it clear that your docker images are community supported I'd say we can put that issue through QA.
11:15 poikilotherm Alright
11:15 poikilotherm I'll give it a shot
11:15 pdurbin \o/
11:19 MrK pdurbin: hi
11:22 MrK poikilotherm: In our version we already integrated Arquillian but it was not without troubles
11:23 poikilotherm Oh nice :-)
11:23 poikilotherm What kind of troubles?
11:23 pdurbin MrK since you and poikilotherm are so interested in Slack, I thought I'd ask for feedback from you two (and anyone else who is interested) on this new "Slack is bad for open source" idea I've come up with: https://github.com/good-labs/slopi-communication . What do you think? :)
11:27 MrK I think it's good initiative.
11:28 pdurbin Thanks. So far I think Jupyter is winning as most SLOPI: https://github.com/good-labs/slopi-communication/issues/2
11:30 pdurbin MrK: can you help us add your Arquillian tests to https://jenkins.dataverse.org ?
11:32 MrK so far they are very limited and I'm not happy with them, the limitations are for example - you cannot test ViewScope beans so far since you need primefaces context, every class needs their own container deploy so it will be very slow (issue for this is waiting for 9?? years).
11:33 MrK you mean creating the pipeline or :p?
11:34 pdurbin MrK: I mean making a pull request for https://github.com/IQSS/dataverse-jenkins
11:37 MrK Would it make sense? Our dataverses are kinda different now :P.
11:38 pdurbin Well, do your tests work on "vanilla" Dataverse? Or do they require your fork?
11:40 MrK I mean adding Arquillian itself requires changes, and we have 1 test class for our functionality.
11:43 pdurbin Interesting. Do you recommend Arquillian?
11:45 MrK Hmm I can't say I recommend since if I could I would just switch to spring which was created in mind about integration testing :D, but I don't think there are alternatives. But still as I mentioned before there are still problems with it.
11:46 pdurbin Ok. What is the functionality that's only in your fork? The display order for files feature?
11:46 MrK lemme check the board
11:47 pdurbin thanks
11:49 MrK we moved the settings to property file, multiple licenses for file (WIP nearly done), banners and global messages for dataverse, possibility to search by range in advanced search, new permission and role, bug fixes.
11:50 pdurbin Wow, those all sound great and non controversial. Do you plan to make pull requests upstream for any of those?
11:55 MrK Well that's the problem, we were unable to merge constantly and also make pr's since we are time binded, atm we are I think very apart from each other. I can make pr, but I would not be able to change the things you guys would want :<, so it would be just "handing over" the code.
11:57 pdurbin Ok, it sounds like there are multiple problems. Upstream releases too often? Is that what you're saying? You're under time pressure and can't wait for upstream to merge the features you need? Is that right?
12:01 pdurbin I believe there are around 7 forks in production (out of 43 installations on our map), maybe more. It's hard to tell sometimes. You aren't alone. :) We'd like to work with all the forks and get the features merged upstream. :)
12:01 poikilotherm When I understood MrK correct, they are just limited in time to get their additions inline with upstream wishes, toughts, views and opinions.
12:02 MrK We were unable to wait for PR if we wanted to add some new feature and there was also a risk that since test coverage is low there would be new bugs if we merged, and since it could be not our code we would have to find out and understand other's people code, so it was also just safer.
12:03 MrK Yeah we have the contract with UE so from what I heard they are strict.
12:03 pdurbin Sure, but longer term it's probably safer to stay close to upstream. I'm thinking about security fixes, especially. :)
12:04 poikilotherm But MrK how do you deal with upstream changes?
12:04 poikilotherm pdurbin: yeah, same question same time :-d
12:04 MrK We just don't merge :D
12:05 poikilotherm Typical fork and merge issue of OSS ;-)
12:06 MrK Yeah it's getting hard when there are not only 2 sides but more and they all work kinda differently.
12:15 pdurbin herding cats :)
12:34 poikilotherm pdurbin do you mind if I change that entire prep part?
12:34 poikilotherm file:///home/obertuch/Entwicklung/dataverse/doc/​sphinx-guides/build/html/installation/prep.html
12:34 poikilotherm whoops
12:34 poikilotherm wrong link
12:34 poikilotherm http://guides.dataverse.org/en/latest/installation/prep.html
12:35 pdurbin poikilotherm: be bold! As Wikipedia says. :)
12:35 poikilotherm :-D
12:35 poikilotherm I think of a responsive diagram with the components clickable, providing more info
12:35 pdurbin sounds fancy
12:36 pdurbin go for it
12:36 poikilotherm I saw https://onesignal.com/blog/rust-at-onesignal/ and that triggered me :-D
12:37 pdurbin very nice
13:01 sivoais joined #dataverse
13:09 donsizemore joined #dataverse
13:21 pdurbin donsizemore: mornin. Can I show you something I posted on AskCI the other day?
13:36 donsizemore @pdurbin shore. just scrolled through this morning's IRC chatter
13:59 pdurbin_m joined #dataverse
14:00 pdurbin_m donsizemore: thanks, how does this look? https://ask.cyberinfrastructure.org/t/what-sort-of-approaches-work-well-for-providing-encrypted-storage-for-research-computing-environments/105/2
14:03 donsizemore @pdurbin looks great to me. i've implemented most of that stack, just waiting on the SAFE and Notary services to solidify
14:04 pdurbin_m awesome
14:05 pdurbin_m How are the API tests treating you?
14:06 donsizemore oh, i kicked them off on friday, looks like they were still running when i left
14:08 pdurbin_m cool
14:18 donsizemore whatever i did, i killed it. ima do a clean run
14:31 pdurbin_m thanks
14:34 donsizemore @pdurbin_m i should be able to use the FAKE DOI provider, yes?
14:47 pdurbin donsizemore: sure. FAKE is fine.
14:49 donsizemore @pdurbin asking because modifying Datacite creds is the only step in the README i skipped
14:49 pdurbin ah
14:49 pdurbin phoenix uses FAKE
14:50 pdurbin FAKE is good for what ails you.
14:51 donsizemore fake it 'til you make it, i always say.
14:51 pdurbin heh
14:57 donsizemore @pdurbin so, for me using FAKE, the spruce dataset publish fails "due to an error contacting datacite"
14:57 donsizemore @pdurbin why i asked about FAKE
14:59 pdurbin the spruce dataset is created using curl?
17:40 donsizemore joined #dataverse
18:08 pdurbin or with java? (rest assured)
18:19 donsizemore @pdurbin i updated the zipfile. today i hit my first 500 at edu.harvard.iq.dataverse.api.DatasetsI​T.testPrivateUrl(DatasetsIT.java:948); everything after that call returns a 500
18:20 donsizemore Exception caught trying to instantiate PrivateUrlRedirectData: PrivateUrlUser cannot be null
18:45 pdurbin good times
18:51 donsizemore joined #dataverse
19:33 pdurbin donsizemore: should I spin up an ec2 instance with ssh keys for both of us so we can look at the same server.log etc?
19:34 donsizemore @pdurbin or i'm happy to. i imagine yours will succeed =p
19:35 pdurbin https://github.com/donsizemore.keys is empty so maybe you should spin it up and add https://github.com/pdurbin.keys please
19:36 pdurbin We'd spin up a centos 7 ec2 instance and then install docker on it? Would that be the plan?
19:36 pdurbin I don't think I've ever installed docker on anything other than a laptop.
19:41 donsizemore i was just going to build develop then do a git clone? we can run it in docker if you like
19:42 pdurbin ok, so just run `mvn test` with the IT tests. sounds perfect. fewer moving parts :)
19:42 pdurbin sorry this is so hard
19:42 pdurbin while we wait for the ec2 instance to spin up, can I pick your brain about sample data?
19:42 donsizemore not hard for me, just iterative until i get a smiley face
19:44 pdurbin it's sort of a suggestion, I guess, for sample data
19:46 donsizemore running on EC2 now. will check results (and possibly nuke VM after the gym?)
19:46 pdurbin sounds good
19:47 pdurbin Harvard IT Summit tomorrow so I may not be very available.
19:48 donsizemore may want to do it in docker, the prep_IT script sets the burrito et al
19:48 donsizemore enjoy the summit!
19:48 pdurbin sure, and thanks. do you want to hear about the sample data thing quick? :)
19:48 pdurbin I don't want to eat up your gym time. :)
19:49 donsizemore sure thing
19:51 pdurbin The news is the Stefan from this channel gave a little talk about his pyDataverse package last week.
19:51 pdurbin And he mentioned a format he invented. Once sec, I'll pull up the notes.
19:52 pdurbin the notes: https://groups.google.com/d/msg/dataverse-community/op753PRmoUY/mm7GAGoaAgAJ
19:52 pdurbin "A central functionality for this will be to create a data model for the data and metadata used by Dataverse which then can be used to import from and export into other sources and to directly upload into the Dataverse API. One output format is what I call a "DVTree" (Dataverse Tree), a predefined structure of directories and files with defined naming conventions, and content, so you can export and import
19:52 pdurbin from this local file hierarchy and directory structure into the python objects."
19:52 pdurbin So here's my suggestion... what if we use his DVTree format for sample data?
20:02 pdurbin Does that make sense? What do you think?
21:37 donsizemore joined #dataverse
21:38 donsizemore @pdurbin I really like the DVtree concept
21:38 pdurbin great!
21:38 pdurbin I brought all this up at a design meeting this morning.
21:38 pdurbin If we adopt DVtree, Mike can use the sample data on his laptop, where he doesn't use ansible.
21:39 donsizemore @pdurbin I was thinking(sizzling) at the gym. I could either a) add an ec2-create flag so it stands up docker-aio and runs tests (simpler) or b) add a flag so's it mimics what prep_IT et al do and run the tests locally (longer)
21:40 donsizemore but for now, ima head home and spread fungicide on my iris. i'll take a look at the DVtree sampledata in the morning?
21:40 pdurbin I'm confused. Spin up an ec2 instance and run docker-aio on the ec2 instance? Why not just run the tests from jenkins against the ec2 instance? Let's talk about this later. I hope your eye is ok. O.o
21:42 donsizemore @pdurbin docker-aio would be a short-term get-it-working. running tests directly needs some of prep_IT carved out (burrito, api key) but would be the better solution
21:42 donsizemore gonna head out. have a great evening!
21:42 pdurbin you too
21:43 * pdurbin starts making burritos

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.