Time
S
Nick
Message
03:17 jri joined #dataverse
07:13 jri joined #dataverse
07:37 juancorr joined #dataverse
08:54 MrK joined #dataverse
09:41 pdurbin joined #dataverse
10:03 poikilotherm joined #dataverse
10:10 pdurbin
poikilotherm: interesting that Slava might switch to your Docker images: https://github.com/IQSS/dataverse-docker/issues/18#issuecomment-491715460
10:11 pdurbin
MrK: mornin'
10:11 poikilotherm
Morning pdurbin
10:11 poikilotherm
He wrote me an emai l: _)
10:11 pdurbin
Ah, ok.
10:11 pdurbin
probably makes sense to join forces :)
10:12 poikilotherm
Yeah ;-)
10:12 poikilotherm
He wanted to take a look and see how it goes
10:13 pdurbin
Does he have any particular needs, such as running multiple Glassfish servers?
10:14 poikilotherm
He also asked about running Selenium based integration tests on these deployments
10:14 pdurbin
nice
10:14 poikilotherm
If I got it right, he wanted those to ensure that the deployment works as expected
10:14 poikilotherm
I told him that I would be happy to run Selenium tests on this, but those should come or be commited to upstream
10:15 poikilotherm
Because those should also help devs and QA
10:15 poikilotherm
He didn't tell me about special needs
10:15 pdurbin
Yes, it would be great for those Selenium tests to be contributed upstream.
10:16 poikilotherm
Oh there is a "from" missing above... "come from"
10:17 pdurbin
They could replace old selenium tests like this one that hasn't been touched or run in four years: https://github.com/IQSS/dataverse/blob/v4.14/tests/create_account.py
10:22 poikilotherm
Yeah. IMHO such things should be implemented in Selenium ruled by JUnit
10:23 poikilotherm
(Maybe automated / supported with things like Arquillian)
10:24 pdurbin
Ah, instead of Python.
10:24 poikilotherm
Stay within the same eco-system, you know ;-)
10:25 pdurbin
Couldn't hurt. But Python is fine too.
10:58 poikilotherm
Oi pdurbin: any preferences on where to put some notes about the dataverse kubernetes installation adventure?
10:59 poikilotherm
I am not sure if it fits inside "Advanced installation"
10:59 poikilotherm
I dunno if it might be the right time to refactor this part
11:00 pdurbin
It's definitely fine to refactor. Do you plan to put https://github.com/IQSS/dataverse/issues/4665 through QA?
11:01 poikilotherm
Uff. Dunno. I thought about creating a new issue
11:01 pdurbin
sure, a new issue is fine
11:01 poikilotherm
Maybe some day we will need to abort 4665 ;-)
11:01 poikilotherm
The original owner seems not to be responsive...
11:02 pdurbin
Want me to unassign you?
11:03 poikilotherm
What do you prefer?
11:03 poikilotherm
Is there a standard process at IQSS how to deal with stale issues?
11:03 pdurbin
I prefer for issues to be closed, eventually. :)
11:04 pdurbin
Not really. I have a couple labels I sometimes throw on issues. "Status: Still interested?" and "Vote to close: pdurbin" :)
11:04 poikilotherm
:-D
11:06 poikilotherm
If you want, I can come up with a branch for the issue and add some docs
11:06 poikilotherm
Then it might hurt less to close the issue :-d
11:07 pdurbin
I believe the creator of that issue is in production already at http://data.mel.cgiar.org
11:07 pdurbin
42nd installation of Dataverse: https://groups.google.com/d/msg/dataverse-community/_1qOQqQRXEA/-7SmtE_jAQAJ
11:08 pdurbin
running 4.8.5
11:08 poikilotherm
Yeah
11:10 poikilotherm
So this might be a plan how to get it resolved - finally?
11:10 pdurbin
I just left a comment and added the "still interested?" label.
11:11 pdurbin
If your pull request makes it clear that your docker images are community supported I'd say we can put that issue through QA.
11:15 poikilotherm
Alright
11:15 poikilotherm
I'll give it a shot
11:15 pdurbin
\o/
11:19 MrK
pdurbin: hi
11:22 MrK
poikilotherm: In our version we already integrated Arquillian but it was not without troubles
11:23 poikilotherm
Oh nice :-)
11:23 poikilotherm
What kind of troubles?
11:23 pdurbin
MrK since you and poikilotherm are so interested in Slack, I thought I'd ask for feedback from you two (and anyone else who is interested) on this new "Slack is bad for open source" idea I've come up with: https://github.com/good-labs/slopi-communication . What do you think? :)
11:27 MrK
I think it's good initiative.
11:28 pdurbin
Thanks. So far I think Jupyter is winning as most SLOPI: https://github.com/good-labs/slopi-communication/issues/2
11:30 pdurbin
MrK: can you help us add your Arquillian tests to https://jenkins.dataverse.org ?
11:32 MrK
so far they are very limited and I'm not happy with them, the limitations are for example - you cannot test ViewScope beans so far since you need primefaces context, every class needs their own container deploy so it will be very slow (issue for this is waiting for 9?? years).
11:33 MrK
you mean creating the pipeline or :p?
11:34 pdurbin
MrK: I mean making a pull request for https://github.com/IQSS/dataverse-jenkins
11:37 MrK
Would it make sense? Our dataverses are kinda different now :P.
11:38 pdurbin
Well, do your tests work on "vanilla" Dataverse? Or do they require your fork?
11:40 MrK
I mean adding Arquillian itself requires changes, and we have 1 test class for our functionality.
11:43 pdurbin
Interesting. Do you recommend Arquillian?
11:45 MrK
Hmm I can't say I recommend since if I could I would just switch to spring which was created in mind about integration testing :D, but I don't think there are alternatives. But still as I mentioned before there are still problems with it.
11:46 pdurbin
Ok. What is the functionality that's only in your fork? The display order for files feature?
11:46 MrK
lemme check the board
11:47 pdurbin
thanks
11:49 MrK
we moved the settings to property file, multiple licenses for file (WIP nearly done), banners and global messages for dataverse, possibility to search by range in advanced search, new permission and role, bug fixes.
11:50 pdurbin
Wow, those all sound great and non controversial. Do you plan to make pull requests upstream for any of those?
11:55 MrK
Well that's the problem, we were unable to merge constantly and also make pr's since we are time binded, atm we are I think very apart from each other. I can make pr, but I would not be able to change the things you guys would want :<, so it would be just "handing over" the code.
11:57 pdurbin
Ok, it sounds like there are multiple problems. Upstream releases too often? Is that what you're saying? You're under time pressure and can't wait for upstream to merge the features you need? Is that right?
12:01 pdurbin
I believe there are around 7 forks in production (out of 43 installations on our map), maybe more. It's hard to tell sometimes. You aren't alone. :) We'd like to work with all the forks and get the features merged upstream. :)
12:01 poikilotherm
When I understood MrK correct, they are just limited in time to get their additions inline with upstream wishes, toughts, views and opinions.
12:02 MrK
We were unable to wait for PR if we wanted to add some new feature and there was also a risk that since test coverage is low there would be new bugs if we merged, and since it could be not our code we would have to find out and understand other's people code, so it was also just safer.
12:03 MrK
Yeah we have the contract with UE so from what I heard they are strict.
12:03 pdurbin
Sure, but longer term it's probably safer to stay close to upstream. I'm thinking about security fixes, especially. :)
12:04 poikilotherm
But MrK how do you deal with upstream changes?
12:04 poikilotherm
pdurbin: yeah, same question same time :-d
12:04 MrK
We just don't merge :D
12:05 poikilotherm
Typical fork and merge issue of OSS ;-)
12:06 MrK
Yeah it's getting hard when there are not only 2 sides but more and they all work kinda differently.
12:15 pdurbin
herding cats :)
12:34 poikilotherm
pdurbin do you mind if I change that entire prep part?
12:34 poikilotherm
file:///home/obertuch/Entwicklung/dataverse/doc/sphinx-guides/build/html/installation/prep.html
12:34 poikilotherm
whoops
12:34 poikilotherm
wrong link
12:34 poikilotherm
http://guides.dataverse.org/en/latest/installation/prep.html
12:35 pdurbin
poikilotherm: be bold! As Wikipedia says. :)
12:35 poikilotherm
:-D
12:35 poikilotherm
I think of a responsive diagram with the components clickable, providing more info
12:35 pdurbin
sounds fancy
12:36 pdurbin
go for it
12:36 poikilotherm
I saw https://onesignal.com/blog/rust-at-onesignal/ and that triggered me :-D
12:37 pdurbin
very nice
13:01 sivoais joined #dataverse
13:09 donsizemore joined #dataverse
13:21 pdurbin
donsizemore: mornin. Can I show you something I posted on AskCI the other day?
13:36 donsizemore
@pdurbin shore. just scrolled through this morning's IRC chatter
13:59 pdurbin_m joined #dataverse
14:00 pdurbin_m
donsizemore: thanks, how does this look? https://ask.cyberinfrastructure.org/t/what-sort-of-approaches-work-well-for-providing-encrypted-storage-for-research-computing-environments/105/2
14:03 donsizemore
@pdurbin looks great to me. i've implemented most of that stack, just waiting on the SAFE and Notary services to solidify
14:04 pdurbin_m
awesome
14:05 pdurbin_m
How are the API tests treating you?
14:06 donsizemore
oh, i kicked them off on friday, looks like they were still running when i left
14:08 pdurbin_m
cool
14:18 donsizemore
whatever i did, i killed it. ima do a clean run
14:31 pdurbin_m
thanks
14:34 donsizemore
@pdurbin_m i should be able to use the FAKE DOI provider, yes?
14:47 pdurbin
donsizemore: sure. FAKE is fine.
14:49 donsizemore
@pdurbin asking because modifying Datacite creds is the only step in the README i skipped
14:49 pdurbin
ah
14:49 pdurbin
phoenix uses FAKE
14:50 pdurbin
FAKE is good for what ails you.
14:51 donsizemore
fake it 'til you make it, i always say.
14:51 pdurbin
heh
14:57 donsizemore
@pdurbin so, for me using FAKE, the spruce dataset publish fails "due to an error contacting datacite"
14:57 donsizemore
@pdurbin why i asked about FAKE
14:59 pdurbin
the spruce dataset is created using curl?
17:40 donsizemore joined #dataverse
18:08 pdurbin
or with java? (rest assured)
18:19 donsizemore
@pdurbin i updated the zipfile. today i hit my first 500 at edu.harvard.iq.dataverse.api.DatasetsIT.testPrivateUrl(DatasetsIT.java:948); everything after that call returns a 500
18:20 donsizemore
Exception caught trying to instantiate PrivateUrlRedirectData: PrivateUrlUser cannot be null
18:45 pdurbin
good times
18:51 donsizemore joined #dataverse
19:33 pdurbin
donsizemore: should I spin up an ec2 instance with ssh keys for both of us so we can look at the same server.log etc?
19:34 donsizemore
@pdurbin or i'm happy to. i imagine yours will succeed =p
19:35 pdurbin
https://github.com/donsizemore.keys is empty so maybe you should spin it up and add https://github.com/pdurbin.keys please
19:36 pdurbin
We'd spin up a centos 7 ec2 instance and then install docker on it? Would that be the plan?
19:36 pdurbin
I don't think I've ever installed docker on anything other than a laptop.
19:41 donsizemore
i was just going to build develop then do a git clone? we can run it in docker if you like
19:42 pdurbin
ok, so just run `mvn test` with the IT tests. sounds perfect. fewer moving parts :)
19:42 pdurbin
sorry this is so hard
19:42 pdurbin
while we wait for the ec2 instance to spin up, can I pick your brain about sample data?
19:42 donsizemore
not hard for me, just iterative until i get a smiley face
19:44 pdurbin
it's sort of a suggestion, I guess, for sample data
19:46 donsizemore
running on EC2 now. will check results (and possibly nuke VM after the gym?)
19:46 pdurbin
sounds good
19:47 pdurbin
Harvard IT Summit tomorrow so I may not be very available.
19:48 donsizemore
may want to do it in docker, the prep_IT script sets the burrito et al
19:48 donsizemore
enjoy the summit!
19:48 pdurbin
sure, and thanks. do you want to hear about the sample data thing quick? :)
19:48 pdurbin
I don't want to eat up your gym time. :)
19:49 donsizemore
sure thing
19:51 pdurbin
The news is the Stefan from this channel gave a little talk about his pyDataverse package last week.
19:51 pdurbin
And he mentioned a format he invented. Once sec, I'll pull up the notes.
19:52 pdurbin
the notes: https://groups.google.com/d/msg/dataverse-community/op753PRmoUY/mm7GAGoaAgAJ
19:52 pdurbin
"A central functionality for this will be to create a data model for the data and metadata used by Dataverse which then can be used to import from and export into other sources and to directly upload into the Dataverse API . One output format is what I call a "DVTree" (Dataverse Tree), a predefined structure of directories and files with defined naming conventions, and content, so you can export and import
19:52 pdurbin
from this local file hierarchy and directory structure into the python objects."
19:52 pdurbin
So here's my suggestion... what if we use his DVTree format for sample data?
20:02 pdurbin
Does that make sense? What do you think?
21:37 donsizemore joined #dataverse
21:38 donsizemore
@pdurbin I really like the DVtree concept
21:38 pdurbin
great!
21:38 pdurbin
I brought all this up at a design meeting this morning.
21:38 pdurbin
If we adopt DVtree, Mike can use the sample data on his laptop, where he doesn't use ansible.
21:39 donsizemore
@pdurbin I was thinking(sizzling) at the gym. I could either a) add an ec2-create flag so it stands up docker-aio and runs tests (simpler) or b) add a flag so's it mimics what prep_IT et al do and run the tests locally (longer)
21:40 donsizemore
but for now, ima head home and spread fungicide on my iris. i'll take a look at the DVtree sampledata in the morning?
21:40 pdurbin
I'm confused. Spin up an ec2 instance and run docker-aio on the ec2 instance? Why not just run the tests from jenkins against the ec2 instance? Let's talk about this later. I hope your eye is ok. O.o
21:42 donsizemore
@pdurbin docker-aio would be a short-term get-it-working. running tests directly needs some of prep_IT carved out (burrito, api key) but would be the better solution
21:42 donsizemore
gonna head out. have a great evening!
21:42 pdurbin
you too
21:43 * pdurbin
starts making burritos
