Time
S
Nick
Message
08:07 juancorr joined #dataverse
08:21 jri joined #dataverse
08:23 jri joined #dataverse
10:22 jri joined #dataverse
10:50 skasberger joined #dataverse
10:55 jri joined #dataverse
12:04 pdurbin joined #dataverse
12:17 poikilotherm joined #dataverse
12:17 poikilotherm
Morning pdurbin_m
12:18 poikilotherm
I see you are already commenting on PRs :-D
12:18 pdurbin
Yeah but I need coffee.
12:19 * poikilotherm
hands over more coffee
12:19 pdurbin
If the release note had been in there I would have moved the Solr PR to QA. bricas should be happy. :)
12:20 pdurbin
I'm glad the deprecations warnings are only at startup. Thanks.
12:20 poikilotherm
LOL
12:20 poikilotherm
I wanted to give Danny to opportunity to jump ok
12:20 poikilotherm
s/ok/in
12:21 poikilotherm
s/to/the/
12:21 pdurbin
You don't need to wordsmith the release note stub very much. You can just something in doc/release-notes/[issue]-solr-upgrade.
12:21 pdurbin
just throw something
12:21 poikilotherm
About coffee: there is a startup in Germany called "earlybird coffee" https://www.gruenderszene.de/datenbank/uploads/company_normal/earlybird-coffee.jpg
12:21 pdurbin
cute bird
12:22 poikilotherm
You would like me to push sth?
12:30 poikilotherm
Took the liberty to "just do it"
12:42 pdurbin
The release note looks good. Thanks! Moved it to QA.
12:50 pdurbin
And I created this issue: https://github.com/IQSS/dataverse-ansible/issues/146
12:54 pdurbin
skasberger: wow, this sounds like a big upgrade: https://twitter.com/TheAUSSDA/status/1226806161401417729
12:58 poikilotherm
Go skasberger go! :-D
13:05 pkiraly joined #dataverse
13:30 Benjamin_Peuch joined #dataverse
13:38 pdurbin
poikilotherm: your "in focus" column is looking good: https://github.com/orgs/IQSS/projects/4#column-5769388
13:38 poikilotherm
Thx
13:38 poikilotherm
How come?
14:17 pdurbin
- Provide documentation of production reuse
14:17 pdurbin
- Upgrade to Dataverse 4.19
14:18 pdurbin
- Add log persistance for Make Data Count
14:18 pdurbin
- Test to use upstream support Solr images (>=7.5)
14:18 pdurbin
poikilotherm: I just mean I'm seeing a lot of stuff I like. ^^ :)
14:18 Benjamin_Peuch
Hello everybody.
14:18 poikilotherm
Heh.
14:18 poikilotherm
Busy man over here
14:18 poikilotherm
Hi Benjamin_Peuch
14:19 Benjamin_Peuch
Hi hi poikilotherm.
14:19 Benjamin_Peuch
Hey pdurbin, I was wondering if you knew of a more user- (understand: non-computer technician user) -friendly way of importing datasets into Dataverse than with DVUploader or through APIs as an admin.
14:20 Benjamin_Peuch
Or if this was perhaps in development so that it's integrated to the basic Dataverse UI.
14:20 pdurbin
Benjamin_Peuch: hmm. In DVN 3.x we had a superuser "import" feature in the GUI .
14:20 donsizemore joined #dataverse
14:20 Benjamin_Peuch
But it's limited to the Superuser, then? It's not for everybody?
14:21 skasberger
Yeah, from 4.6.2 to 4.18.1 I hope. :)
14:21 Benjamin_Peuch
Because we are realizing that we might have a lot of situations in the future where users who know nothing about command lines have to import datasets that already received a DOI or a Handle.
14:21 poikilotherm
Benjamin_Peuch: would CLI be ok? This could be sth for dvcli
14:21 skasberger
But I hope it will be finished in one week, but you never know.
14:22 pdurbin
Benjamin_Peuch: see "Import Utilities" at http://guides.dataverse.org/en/3.6.2/dataverse-user-main.html#utilities for how it worked in DVN 3.x. This feature has not (yet) been ported to Datavers 4.x.
14:23 Benjamin_Peuch
Thanks, pdurbin. I found this documentation while browsing the Google Group looking for an answer but I didn't look at in depth. I'll do that now.
14:23 Benjamin_Peuch
Well poikilotherm, we were hoping not to force our users to use CLI because we fear most of them are not familiar with it.
14:24 Benjamin_Peuch
We like the current Dataverse UI and we want to make it so that our users can do as much as possible with it. Though I understand some cases might require extra manipulations.
14:24 pdurbin
Benjamin_Peuch: if you'd like a GUI "import dataset" feature like this for Dataverse 4.x, please create an issue.
14:25 poikilotherm
Benjamin_Peuch: OK. When nothing works for you, you could still do a lightweight JavaScript web app talking to the APO
14:25 poikilotherm
API
14:25 pdurbin
We should probably have external tools at the installation level that superusers can use.
14:26 Benjamin_Peuch
That was going to be my next question (what can we do in the meantime before such a feature is integrated in Dataverse's GUI ). Thanks pdurbin. :)
14:26 Benjamin_Peuch
I mean poikilotherm. Sorry about that.
14:26 Benjamin_Peuch
Thanks, too, pdurbin. I'll just make sure this hasn't already been filed in an issue, in which case I'll bump it.
14:27 poikilotherm
Well I guess IQSS might be not unhappy with an external tool developed by community for this feature instead of investing resources...
14:27 pdurbin
Benjamin_Peuch: sure, you're welcome to look but I don't believe an issue like this exists.
14:27 pdurbin
Benjamin_Peuch: have you seen https://github.com/IQSS/dataverse-ddi-converter-tool ? It feels related to me.
14:27 pdurbin
I'm sure skasberger knows about it.
14:34 pdurbin
Benjamin_Peuch: what format do you want to use when importing? XML ? JSON ?
14:38 Benjamin_Peuch
pdurbin: Thanks for pointing me to the pyDataverse module. I'm looking into it.
14:38 pdurbin
sure
14:38 Benjamin_Peuch
pdurbin: Ideally DDI, but it's not like it matters a lot, does it?
14:38 pdurbin
I don't think it matters much. It's a one time conversion.
14:38 Benjamin_Peuch
Whatever works I'd say. We've had a few problems with XML -DDI files during tests here and, unsurprisingly, using JSON instead often solved the issue.
14:39 pdurbin
gotcha
14:39 pdurbin
Well, for what it's worth, we used DDI to migrated datasets from DVN 3.x to Dataverse 4. Everybody did. It was the supported migration process. Still is. There's one DVN 3.x installation left on the map. :)
14:49 pdurbin
poikilotherm: did you see https://github.com/poikilotherm/dvcli/issues/1 ?
14:51 poikilotherm
Yeah! :-)
14:51 poikilotherm
Thanks for opening
14:51 poikilotherm
Interesting ideas
14:53 pdurbin
Sure. I don't know if this is helpful or just adds confusion but I have a bunch of scripts squirreled away at https://github.com/IQSS/dataverse/tree/v4.19/scripts/database/homebrew
14:55 Benjamin_Peuch
I have to run, but thanks a lot for the help, pdurbin and poikilotherm! I'll investigate and see what I can find among existing tools. And if I can't find what I'm looking for, I'll make sure to open an issue on GitHub.
14:55 Benjamin_Peuch
Have a nice day. :)
14:55 pdurbin
o/
14:56 poikilotherm
Cu Benjamin_Peuch
14:56 pdurbin
poikilotherm: and I have a bunch of scripts on my laptop with dubious names like glassfish-rm-generated :)
14:56 poikilotherm
pdurbin: nice. Also I don't know if we should operate on the DB directly...
14:56 poikilotherm
Aren't there API endpoints for this?
14:57 donsizemore
@poikilotherm if you need to rm -rf generated/ glassfish won't wake up enough to respond to the endpoint
14:57 poikilotherm
OK so dvcli needs to be sudo compatible
14:57 pdurbin
If there were an API for rm -rf generated, I wouldn't have it here: https://github.com/IQSS/dataverse/blob/v4.19/scripts/deploy/phoenix.dataverse.org/rebuild#L9
14:57 poikilotherm
I meant API for things like role creation
14:58 donsizemore
it would be a polite thing for dataverse to do at shutdown
14:58 pdurbin
I'd be afraid to have all my dev scripts on a production system. I'd probably drop the database or something. Whoops!
14:59 poikilotherm
You have a create-role script in homebrew that talks to postgres, creating a role in the DB
14:59 poikilotherm
That's why I was looking into a framework. We can create plugins.
14:59 poikilotherm
So you can install more command by installing a pip package
15:00 poikilotherm
Maybe we should try to keep the core really minimum in terms of dependencies and danger
15:00 pdurbin
We laugh in the face of danger.
15:00 poikilotherm
:-D
15:02 pdurbin
Ah, the first #LoveDataWeek and #LoveData20 tweets from a Dataverse installation. It's a nice tradition: https://twitter.com/scholarsportal/status/1226881781300973568
15:40 pkiraly
pdurbin: I continued the Native API documentation update: https://github.com/IQSS/dataverse/pull/6632
15:43 pdurbin
pkiraly: great! Are you still on that branch? Can I ask you to make a couple more edits?
15:44 pkiraly
pdurbin, sure
15:45 pdurbin
pkiraly: do you see two cases of "export ID=xxxxxx" ? Here, for example: https://github.com/IQSS/dataverse/blob/develop/doc/sphinx-guides/source/api/native-api.rst#report-the-data-file-size-of-a-dataset
15:45 pdurbin
and here: https://github.com/IQSS/dataverse/blob/develop/doc/sphinx-guides/source/api/native-api.rst#get-the-size-of-downloading-all-the-files-of-a-dataset-version
15:45 pdurbin
And "VERSIONID=x.x" in the second example.
15:46 pdurbin
pkiraly: can you please change these to real numbers? 42 or 1.1 or whatever? Does that make sense?
15:46 pdurbin
because just having xxxxxx is confusing, in my opinion
15:47 pdurbin
the whole point of the examples is to make the examples concrete, real world
15:51 pkiraly
pdurbin: It's strange, in my version all these don't exist. I've created my branch from develop, these must be newer additions
15:54 pkiraly
pdurbin: I forget to add that I've created the branch on Friday, these changes must have been added after I created the branch
15:57 pkiraly
yes, these have been added recently: https://github.com/IQSS/dataverse/commit/8669089ef59acca06304f088797f1217b599d5a9#diff-472e268f05a2e75eb8b32c148f80c2c0 . The commit has been created last week, but it is not clear how to check when the pull was merged.
15:59 pkiraly
pdurbin: when these changes will be accepted, I'll continue with the files API , and then I can fix these two new sections as well. Is it OK for you?
16:02 pdurbin
pkiraly: yes, that's fine! Let me take a quick look at your pull request and move it to QA if it looks ok.
16:03 pdurbin
wow, this is a pretty big one :)
16:04 pdurbin
I *love* seeing stuff like this go away: DELETE http://$SERVER/api/datasets/$id/versions/:draft?key=$apiKey
16:04 pdurbin
full curl examples are much better
16:07 pdurbin
pkiraly: looks good, I just moved it over to QA with a note. Thanks!!
16:11 pkiraly
pdurbin: the ticket is a big one. It talk about changing all examples in the file. And there are about 50 of them. I split it to 4 pull requests, this one is the third. I can give smaller chunks, e.g. one PR for each section. Is that OK, or do you have a different suggestion?
16:30 pdurbin
Whoops, he's gone. This chunk is big enough. No bigger, please. :)
16:31 pdurbin
bricas poikilotherm: at standup we decided to open a new issue just about the Solr upgrade. We still need to figure out a solution for https://github.com/IQSS/dataverse/issues/6599 . Suggestions are welcome.
17:45 donsizemore
@pdurbin want me to stand up DV with solr-7.7.2 somewhere semi-permanent?
18:37 pdurbin
donsizemore: sure! Maybe the Payara 5 box?
18:50 dataverse-user1 joined #dataverse
18:54 donsizemore
I also have a number of test Dateversia on which we may beat
18:55 donsizemore
^^ up which we may beat
18:56 pdurbin
let us beat upon them
19:02 donsizemore
@poikilotherm @pdurbin just to verifiy the solr-solrj version bump in pom.xml doesn't require anything glassfish-wise for a solr upgrade?
19:04 jri joined #dataverse
19:06 poikilotherm
Uh I didn't look that one up... Didn't check on the deps
19:06 poikilotherm
Good catch
19:06 pdurbin
well, we'd need to build a new war file, right?
19:06 poikilotherm
On the one hand it will just bundle any deps in the WAR
19:07 poikilotherm
And as long as we don't make them scope provided, they would be bundeld
19:07 poikilotherm
But I didn't monitor if there are any changes that affect other components
19:08 poikilotherm
pdurbin yes it will be in war for 4.20
19:08 poikilotherm
But maybe better take a look at the diff of the dep tree
19:10 pdurbin
What are we worried about? Or... what problem are we trying to solve? :)
19:17 poikilotherm
Nothing. Just me and my paranoia
19:18 poikilotherm
Better safe than sorry
19:18 * pdurbin
hands poikilotherm anti-paranoia pills
19:19 poikilotherm
If you guys are cool with the risk and don't tell Slava, I'm good. Don't blame me :-D
19:19 * pdurbin
fires up Skype
19:20 poikilotherm
Man, don't do that. Since Tromso you better watch out. Slava is great, but I'm a bit afraid of him...
19:20 donsizemore
@pdurbin @poikilotherm http://payara5.odum.unc.edu/ is now running DV-4.19 and Solr-7.7.2
19:21 pdurbin
donsizemore: that reminds me, can we get Swagger YAML out of it?
19:21 poikilotherm
Nice! I ran it in multiple locks clusters and on my stage/test env and didn't run into errors. But it did not see real data and no users
19:22 poikilotherm
s/locks/local/
19:22 pdurbin
Wow! OpenAPI/Swagger output! Nice! http://payara5.odum.unc.edu/openapi
19:22 poikilotherm
It's so nice to see so many shiny things on Payara making Phil happy
19:23 poikilotherm
pdurbin BTW I just received an email about dvcli from a postdoc at IQSS.
19:24 pdurbin
orly
19:25 pkiraly joined #dataverse
19:25 poikilotherm
Ana T.. Do you think we might try to raise her here? She was asking for invitations for any kind of meetups
19:25 poikilotherm
Even wants to help developing it
19:27 sj joined #dataverse
19:50 an-tris joined #dataverse
19:50 an-tris
hello?
19:51 pdurbin
poikilotherm: there she is
19:51 an-tris
hi!
19:51 pdurbin
an-tris: hi! Sorry https://chat.dataverse.org isn't more friendly. It's my fault.
19:52 pdurbin
This one might be easier to use: http://webchat.freenode.net/?channels=#dataverse
19:53 pdurbin
Since you're on a Mac, you could try Adium. https://adium.im . I put some screenshots at https://github.com/IQSS/chat.dataverse.org/issues/2
19:53 an-tris
haha, ok, this reminds me of chat rooms of the 90-ies
19:53 pdurbin
Yes! IRC is from the late 80s. :)
19:53 pdurbin
When all the good movies came out!
19:54 pdurbin
an-tris: anyway, it's already 9pm for poikilotherm so he might be getting his kids to bed or something. :)
19:54 an-tris
it's ok now, we can stay here
19:55 an-tris
ah ok, I liked the idea of DV-CLI
19:55 pdurbin
Me too. Did you see my Google doc?
19:56 pdurbin
I linked it from https://github.com/poikilotherm/dvcli/issues/1
19:56 an-tris
if there are meetings on that, i'd love to join :-D
19:56 pdurbin
Sure, what sort of use cases do you have in mind?
19:57 poikilotherm
Hi an-tris . Indeed, as pdurbin mentioned, it's already late here... I'm at the construction site, so forgive me slow replies
19:57 pdurbin
poikilotherm: your construction is taking longer than Dataverse 5. ;)
19:58 poikilotherm
Nice to meet you and thanks for offering to jump on the Waggon :-)
19:58 pdurbin
poikilotherm: you should watch "Building reproducible workflows for earth sciences - Responding to reproducibility challenges from physics to social sciences" by an-tris at https://dataverse.org/dataversetv
19:59 pdurbin
actually, all of you should watch it :)
19:59 an-tris
nice to meet you too
20:00 an-tris
haha, that one is soon going to be out-of-date :)
20:00 an-tris
I'll need to have another presentation :))
20:02 an-tris
so, dvcli would be focus on dataverse development? not on what is currently facilitated with pyDataverse?
20:04 poikilotherm
My idea is to create a tool maintained with a core focusing on users and admins most relevant tasks, but be extendable by anyone via plugins to add more features as needed
20:05 poikilotherm
I found the click framework for python which looked pretty promising for that
20:06 poikilotherm
Taking inspirations from kubectl and many other famous cli tools on how to structure this, but open to everything
20:07 an-tris
ah yeah, click is great
20:08 poikilotherm
I don't want to compete with anyone, but ease of use should be a main goal for this project. Easy install, small number of deps, and good ux for admins and normal users
20:08 poikilotherm
Currently there are DVuploader and a lot of other scripts.
20:08 an-tris
I was working on a client at my last job, at cern, this was our client: https://github.com/cernanalysispreservation/cap-client and there is a youtube tutorial: https://www.youtube.com/watch?v=G4ikK2gv-fw&t=4s
20:09 poikilotherm
Wow that reads amazing
20:09 poikilotherm
Building such a thing would definitely be really cool
20:10 poikilotherm
My hope is that we might kick off a simple to use command structure that allows easy splitting of development, so we can maximize the use of resources
20:11 an-tris
thanks! yeah, i agree, I was meaning to put together a project proposal on dv-cli, after having worked with the DV API
20:11 poikilotherm
:-D
20:11 poikilotherm
What a nice coincidence!
20:12 poikilotherm
skasberger is doing such a great job with pyDataverse, it would be so cool to have it at your fingertips
20:12 poikilotherm
With bash completion :-D
20:13 poikilotherm
I do have crazy ideas like autocompletion for metadata etc on my mind...
20:13 an-tris
that would be really cool and useful
20:14 an-tris
I am aware that it's late in Europe, and I don't want to keep you busy
20:14 an-tris
but we should follow-up!
20:18 poikilotherm
Great!
20:19 poikilotherm
I'm very glad to meet someone with experience in designing this. People knowing how users tend to use cli tools are rare in the scientific community...
20:22 an-tris
great! I hope I will be able to help
20:23 pdurbin
I knew you two would get along. :)
20:46 pdurbin
donsizemore: if it's quick... https://services.dataverse.harvard.edu/static/swagger-ui/index.html?url=/miniverse/metrics/v1/swagger.yaml is working but https://services.dataverse.harvard.edu/static/swagger-ui/index.html?url=http://payara5.odum.unc.edu/openapi is not. "Can't read from server. It may not have the appropriate access-control-origin settings." Do you think enabling CORS would help?
21:35 stefankasberger joined #dataverse
21:40 donsizemore
@poikilotherm @pdurbin I just compiled 4.19 with your solr-solrj bump, deployed it to payara5 and enabled https
21:40 donsizemore
@pdurbin i also enabled CORS in the process. i hadn't made certs yet b/c it was going to be i think payara5.dataverse.org but today i made an executive decision
21:41 donsizemore
@pdurbin so you want https://payara5.odum.unc.edu/openapi
21:42 donsizemore
@pdurbin now i get "fetching resource list: https://payara5.odum.unc.edu/openapi; Please wait."
21:48 pdurbin
donsizemore: yeah, same.
21:48 donsizemore
ssl.conf say 'Header set Access-Control-Allow-Origin "*"'
21:51 pdurbin
Yeah. Payara spits out YAML from that /openapi endpoint. But does swagger-ui work with YAML? I don't know. It definitely works with JSON . I'm looking at docs at https://swagger.io/docs/open-source-tools/swagger-ui/usage/installation/
21:51 pdurbin
Those docs say this: SWAGGER_JSON=/foo/swagger.json
21:52 pdurbin
"We can also automatically generate client libraries for your API in many languages and explore other possibilities like automated testing. Swagger does this by asking your API to return a YAML or JSON that contains a detailed description of your entire API." https://swagger.io/docs/specification/2-0/what-is-swagger/
21:53 pdurbin
So it *should* work with YAML , according to that.
21:56 pdurbin
oh hey, I think I got it working
21:59 pdurbin
I went to https://petstore.swagger.io and replaced https://petstore.swagger.io/v2/swagger.json with https://payara5.odum.unc.edu/openapi and clicked "Explore". Boom! There are all the (documented and undocumented) Dataverse API endpoints.
21:59 donsizemore
ooh - more undocumented endpoints?
22:00 pdurbin
shh!
22:00 pdurbin
stefankasberger will hear!
22:03 pdurbin
Here's a screenshot: https://github.com/IQSS/dataverse/issues/5794#issuecomment-584379823
22:10 donsizemore
so, i take it you don't want me to ask a student worker to make a list of undocumented endpoints? (and work on a PR to document them?)
22:10 pdurbin
heh
22:10 pdurbin
Now they're all documented for free, right? :)
22:16 donsizemore
@pdurbin should this be? https://payara5.odum.unc.edu/file.xhtml?fileId=55&version=1.0
22:17 pdurbin
donsizemore: this works if I change http to https - https://qualitativedatarepository.github.io/dataverse-previewers/previewers/ImagePreview.html?fileid=55&siteUrl=https://payara5.odum.unc.edu&datasetid=54&datasetversion=1.0
22:18 pdurbin
nice organ
22:18 pdurbin
The fix should be to change it to https in dataverse.siteUrl (the JVM option)
22:19 donsizemore
me fix siteUrl
22:19 pdurbin
or host the previewers yourself on http
22:19 donsizemore
found that just as you posted =)
22:19 pdurbin
ok, but we should both go home
22:19 donsizemore
have a great evening!
22:21 pdurbin
you too :)
