Time
S
Nick
Message
04:55 poikilotherm joined #dataverse
06:45 poikilotherm
pdurbin: good morning. I've been thinking. What would you say if I suggest creating an official fork of Dataverse, lets call it "dataverse-next". Give it a separate repo under IQSS and have a place to go to for everything we need to be changed for production ready usage on Payara 5. Incorporate any changes that happen in 4, but adapt for usage on Payara if necessary.
06:45 pkiraly joined #dataverse
06:46 poikilotherm
Mornin' pkiraly :-)
06:46 pkiraly
@poikilotherm, Hi!
06:49 poikilotherm
pdurbin: this would allow us to track issues regarding the move more easily. Working with experts and others on such a dedicated project might be easier than letting them dig deep into ~6k issues...
06:51 pkiraly
Do you happen to know if is there a file limit in Dataverse (given the standard setup)? I am aware that there is a parameter I can set, and the API doc said that one of the API call has 2 GB limit.
06:52 poikilotherm
file limit meaning a default size limit or number of files?
06:53 pkiraly
file size
06:54 poikilotherm
Lemme check that
06:55 poikilotherm
Ok installation/config.html#maxfileuploadsizeinbytes says there is no limit by default. Let me check the codebase if the docs are correct :-D
06:59 pkiraly
I've checked it, and it just check if the file size is larger than that. But somewhere I have heard that Glassfish itself might have some implicit limitation.
07:00 poikilotherm
You might run into troubles with your reverse proxy if you are using one
07:00 poikilotherm
Timeouts
07:00 poikilotherm
I'm not aware of other issues when using Glassfish only
07:01 poikilotherm
pdurbin will be around in ~2.5h, he may know some other limitation
07:01 poikilotherm
You could test it :-D
07:01 juancorr joined #dataverse
07:02 poikilotherm
Firing up a test instance is easy with dataverse-kubernetes
07:02 pkiraly
Thanks! My colleague just received a question from a user, who has large files. (However it was not clarified know how large are those files.)
07:03 poikilotherm
Codebase says there is no limit. Haven't found the place where filesize checks happen though
07:03 poikilotherm
Sooner or later, timeouts will occur
07:03 poikilotherm
There is DCM for a reason :-D
07:03 pkiraly
sure. Thanks! (I have to leave now)
07:23 poikilotherm
OMG - pdurbin is already closing issues... Its 3:23 over there...
09:52 pdurbin
poikilotherm: it's a great idea but instead of "dataverse-next" can we call it something like "iqss-jakartaee-template"? It could be a more modern version of https://github.com/IQSS/iqss-javaee-template
10:13 poikilotherm
Hey pdurbin :-)
10:13 poikilotherm
Sorry, but I think we are talking about different problems and solutions to them
10:26 pdurbin
poikilotherm: how about you give me push access to https://github.com/poikilotherm/dataverse and we work on a branch in there?
10:27 poikilotherm
Sure. No problem. I was thinking that making an official repo where more people could contribute in an official manner would be a stronger signal
10:27 pdurbin
But maybe we should make a little progress first. :)
10:28 poikilotherm
Yeah, that was what I wanted to do :-D
10:28 pdurbin
You can call the branch whatever you want. "next" is fine.
10:28 poikilotherm
I thought about creating a repo, maybe even for now just with the issue tracker enabled
10:28 poikilotherm
To collect things we want to see done before we go into production
10:29 poikilotherm
One could use a milestone for that
10:29 poikilotherm
Hmm or we could use a project
10:29 poikilotherm
I could not edit a milestone in IQSS/dataverse
10:29 poikilotherm
But I can participate in a project
10:30 poikilotherm
Dunno
10:30 pdurbin
I'm trying to figure out your requirements.
10:30 pdurbin
- Separate issue tracker?
10:30 poikilotherm
Yeah, that would be cool. I cannot create an issue in IQSS/dataverse and give it a tag, e.g. "payara" to distinct it
10:30 poikilotherm
Ok, that could be solved by some bot
10:31 pdurbin
How about we create a project under https://github.com/orgs/IQSS/projects ?
10:31 poikilotherm
I would like to see a filter "give me all issues about payara"
10:31 poikilotherm
Yeah, that might work (as mentioned above :-D )
10:31 pdurbin
Want me to make it?
10:31 poikilotherm
Can we move fast enough?
10:32 poikilotherm
I would really like to see freedom to experiment etc
10:32 poikilotherm
I have no push access to any branch in IQSS/dataverse
10:32 poikilotherm
And most certainly our changes are far away from going into develop anytime soon
10:33 poikilotherm
How shall we do QA?
10:33 poikilotherm
Most likely we need to do breaking stuff and simply cannot merge into develop without breaking v4
10:34 poikilotherm
That's some reasons why I thought about creating a new repo. So things don't get messed up
10:35 poikilotherm
I dunno if kcondon would do QA for our stuff
10:35 poikilotherm
And what he would like to see
10:36 pdurbin
I think a new repo under IQSS would confuse people.
10:37 pdurbin
Unless we call it "esrevatad" or something.
10:37 poikilotherm
LOL
10:38 poikilotherm
That's why I came up with sth like IQSS/dataverse-next or IQSS/dataverse5
10:38 poikilotherm
Most certainly most people won't notice, just as they don't know about the other repos
10:38 poikilotherm
Like dataverse-ansible or dataverse-aws
10:38 poikilotherm
If you don't use it, its kinda invisible
10:38 pdurbin
Well, the active repos rise to the top. And knowing you... :)
10:39 poikilotherm
LOL
10:39 poikilotherm
So would using a separate repo make sense for you when we come up with some other name?
10:40 pdurbin
I'm thinking.
10:41 poikilotherm
We could up an idea we chatted about a while ago... Rename things to Dataverse...
10:41 poikilotherm
move java packages
10:41 poikilotherm
use another org
10:41 pdurbin
I'm thinking about the risk of just giving you push access to the IQSS repo. Most teams would probably just do this work in a branch.
10:43 pdurbin
I have my one on one with Danny in a few hours. Should I ask him?
10:44 poikilotherm
Sure! Good idea!
10:44 pdurbin
:)
10:44 poikilotherm
Shall we write some summary?
10:44 pdurbin
Nah, that's ok.
10:44 pdurbin
Should we go ahead and make a project?
10:44 pdurbin
I could show him the project.
10:45 pdurbin
You know about "notes" right? They're lighter weight than issues.
10:45 pdurbin
And can be converted into issues.
10:45 poikilotherm
I dunno if using a branch and a project is fine. We need to think about a process for QA
10:45 poikilotherm
Yeah
10:46 poikilotherm
But ok, if we create a branch "next" or similar, we could create a separate "process"
10:46 pdurbin
You could make your pull requests into the "next" branch.
10:46 poikilotherm
Maybe we should add "next" or similar to the pull request to that branch so it can be seen in a glimpse what this is about
10:46 poikilotherm
YEah
10:47 poikilotherm
That was in my head
10:47 poikilotherm
Does Github allow for per-branch rules?
10:47 poikilotherm
Gitlab does...
10:47 pdurbin
Who can push to a branch, you mean?
10:48 poikilotherm
Yeah. It would be nice to have that option about pushing to next depending on how we shape that new process
10:48 poikilotherm
So I could accept PRs for that branch, but not for develop
10:49 poikilotherm
(just thinking about options)
10:49 poikilotherm
If you guys feel better without giving me push access, that's totally fine
10:50 pdurbin
We could even write about "next" in a future version of http://guides.dataverse.org/en/4.16/developers/version-control.html#branches
10:50 poikilotherm
Sure
10:51 pdurbin
A third infinite branch. That some day we'll use for Quarkus once the move the Payara is complete. :)
10:52 poikilotherm
Hihihi
10:52 pdurbin
And after that, quantum computing.
10:53 poikilotherm
Muaahahahah
10:53 pdurbin
We're cutting 4.17 soon. We could try to get "next" in the docs quick.
10:54 poikilotherm
Did I wake you today with my early message?
10:54 poikilotherm
Yeah
10:54 poikilotherm
I saw the issue...
10:54 poikilotherm
I was hoping to get my scribejava lib merged...
10:54 poikilotherm
So it ships with 4.17
10:55 pdurbin
I'm pretty unreachable when I'm sleeping.
10:55 pdurbin
Is your scribejava pull request ready for more review?
10:56 poikilotherm
I had no chance to test with external services yet :-(
10:56 poikilotherm
It has been reviewed but needs testing
10:56 pdurbin
Can we spin up an external service within dataverse-kubernetes?
10:56 poikilotherm
You can't spin up Google, Github or ORCID :-D
10:57 pdurbin
Well, ORCID might be open source. I don't know. :)
10:57 poikilotherm
I was hoping to add better mocks and tests with WireMock and the like, but that stuff needs more refactoring
10:57 poikilotherm
Testing views is not so easy being stuck on old JSF versions :-D
10:58 pdurbin
yeah
10:58 pdurbin
How can I help?
10:59 poikilotherm
We could spin up it in EC2 and test the Google and Github OAuth2 functionality
10:59 poikilotherm
IIRC we just need to tell kcondon if it works or not
10:59 poikilotherm
(and if not debug and refactor)
11:00 poikilotherm
Because he is not very able to test it with current toolsets
11:00 pdurbin
He spins up branches on EC2 already. You can ask for that.
11:00 poikilotherm
Ok but we should do the test before him, so we catch errors ;-)
11:00 poikilotherm
Don't push workload to him if not necessary :-D
11:01 pdurbin
Yes, of course.
11:02 pdurbin
Can we add Google or GitHub auth to https://github.com/IQSS/dataverse-ansible ? You have to go get a key or a secret or whatever. I don't now if this can be automated.
11:02 poikilotherm
Nope, this cannot be automated with OAuth2 :-(
11:02 poikilotherm
There is an option in OIDC for that
11:03 pdurbin
bummer
11:05 pdurbin
Maybe we could ask alejandratenorio to test it. Before she makes a pull request for https://github.com/IQSS/dataverse/issues/6155
11:08 poikilotherm
Ok let me finish my Jenkinsfile for kubeval, then I will test with support of dataverse-kubernetes#64 if I can register a localhost URL at Google
11:08 pdurbin
localhost URL works with GitHub, last I tried it
11:08 pdurbin
I don't know about Google.
11:32 pdurbin
poikilotherm: hey should I add a Dataverse event to https://fosdem.org/2020/fringe/ ? Are you going?
11:33 poikilotherm
Depends on me going to Lissabon and/or Tromso
11:33 poikilotherm
I will get killed if I visit the three of 'em
11:34 pdurbin
I hear that.
12:05 poikilotherm
Man Jenkins really sucks at build matrixes from Jenkinsfiles
12:06 poikilotherm
Declarative hasn't this construct, only usable from scripted pipeline.
12:06 poikilotherm
And guess what: scripted needs to do this in different node definitions, so you cannot mix results from earlier stages
12:06 * pdurbin
doesn't know a lot about Jenkins
12:08 poikilotherm
https://issues.jenkins-ci.org/browse/JENKINS-40986
12:09 pdurbin
poikilotherm: you're in deep. I'm not even sure what you're doing exactly. :) On a related note, I just requested a review from you for https://github.com/IQSS/dataverse-jenkins/pull/16
13:04 juancorr joined #dataverse
13:05 donsizemore joined #dataverse
13:12 poikilotherm
Alright, done with my kubeval stuff :-)
13:12 poikilotherm
Lives in a Jenkins pipeline now
13:16 poikilotherm
https://github.com/IQSS/dataverse-kubernetes/blob/master/test/kubeval/Jenkinsfile
13:16 poikilotherm
Including GitHub status reports :-)
13:17 poikilotherm
Morning donsizemore :-)
13:17 poikilotherm
Did you have a chance to look at my https://github.com/IQSS/dataverse-kubernetes/pull/87
13:18 * poikilotherm
crosses fingers
13:24 pdurbin
poikilotherm: which of your pull requests do you want me to review first?
13:24 poikilotherm
In K8s or upstream?
13:24 pdurbin
anywhere
13:24 poikilotherm
He.
13:25 poikilotherm
Could you test https://github.com/orgs/IQSS/projects/4#card-23657411
13:25 pdurbin
poikilotherm: I have a column for you on my board. Maybe you should have a column for me on your board.
13:25 poikilotherm
Oh good idea :-)
13:27 poikilotherm
https://github.com/orgs/IQSS/projects/4#column-6549146
13:29 pkiraly joined #dataverse
13:32 pdurbin
Ok, I just left a review.
13:32 pdurbin
donsizemore: good morning!
13:33 pdurbin
pkiraly: hi! I owe you a pull request into your pull request. :)
13:37 pkiraly
@pdurbin, or you can share here what you expect
13:40 pkiraly
... or even in the issue (for me it is much quicker than playing with pull request)
13:43 poikilotherm
pdurbin: what to do about the i18n comments?
13:43 poikilotherm
Is this beyond scope?
13:44 pdurbin
Do you know if that English appears in the UI for the end user?
13:45 poikilotherm
I have no idea :-D Ideally it should not appear, as the login would have failed then...
13:45 pdurbin
Can you test it?
13:45 poikilotherm
I didn't add those, just reused them
13:45 poikilotherm
Hui...
13:45 poikilotherm
That might be tough
13:45 poikilotherm
Would need to simulate a network outage or similar
13:45 poikilotherm
Manually testing this is hard
13:45 poikilotherm
This would be easier with WireMock
13:46 poikilotherm
And an e2e test with selenium/...
13:46 poikilotherm
It doesn't look like anybody will sacrifice much time for my PR #87. I will just merge it :-D
13:47 poikilotherm
If things don't work: it wasn't me. It's dataversebot fault.
13:50 pdurbin
Just don't blame iqlogbot
13:50 poikilotherm
Hehehe.
13:53 poikilotherm
BTW pdurbin did you notice my fancy pro/contra list in the README that you need to expand first?
13:53 pdurbin
no, please link me
13:53 poikilotherm
https://github.com/IQSS/dataverse-kubernetes#development-usage
13:53 poikilotherm
Click on "if you think..."
14:04 pdurbin
pkiraly: ok, please look at https://github.com/pkiraly/dataverse/pull/1
14:04 pdurbin
poikilotherm: one on one time. I'll ask about push access.
14:04 * poikilotherm
thumbs up
14:12 pkiraly
pdurbin, I have some comments.
14:13 pkiraly
1. do you want to add the exports to all examples? I left it, because one of the previous examples already had it
14:13 Alejandra69 joined #dataverse
14:13 pkiraly
I am fine with repetition, but I don't know it is was the intention
14:14 pkiraly
2. curl http://$SERVER/api/datasets/$id?key=$apiKey is not equivalent with curl -H X-Dataverse-key:$API_TOKEN $SERVER_URL/api/datasets/:persistentId?persistentId=$PERSISTENT_ID
14:15 pkiraly
as far as I understant the first one expects the dataset identifier in the database, and not the DOI
14:16 pkiraly
so it is a different API call
14:16 pkiraly
understant -> understand
14:54 poikilotherm joined #dataverse
15:01 pdurbin
poikilotherm: ok, I just added you to https://github.com/orgs/IQSS/teams/dataverse-write . Have fun! Don't break anything. :)
15:01 pdurbin
pkiraly: hi! Thanks for taking a look at my pull request to your pull request. I have standup in 15 minutes. Do you have time now?
15:02 pkiraly
yes
15:04 pdurbin
Alejandra69: hi! I have a question for you too. :)
15:05 pdurbin
pkiraly: I do agree that all the exports are repetitive but my goal was to have each endpoint doc "stand alone" if that makes sense.
15:05 pdurbin
Part of the challege here is that the api endpoint doc you are trying to improve is a tricky one.
15:06 pdurbin
If you want, I could suggest a different API endpoint doc to start with.
15:06 pdurbin
Or even an undocumented API . :)
15:07 pdurbin
pkiraly: please let me know when you're ready for more on 2. :)
15:10 pkiraly
@pdurbin: I guess I am not able to decide alone which are the outdated/unwanted API calls. So I am happy to go and replace the format of the API calls wuth curl and fully expanded form, and do the 2. (API change) when we are both available on the chat. Is that acceptable for you?
15:11 pkiraly
wuth -> with
15:14 pdurbin
How about this... You can merge my pull request into yours if you're happy with it. Then we'll move it to QA. Then I'll point you to the next API endpoint to document. Would that be ok?
15:20 pkiraly
OK, but are you sure that you want to remove documentation of the id-based API call?
15:27 Alejandra69
@pdurbin I'm confused, to make a pull request I must clone the project (developer) locally, create a branch and finally commit. Is that right?
15:27 Alejandra69
@pdurbin I'm confused, to do a pull request I must clone the project (developer) locally, create a branch and finally commit. Is that right?
15:28 pkiraly
@Alejandra69, the convenient way is to fork the project, into your own Github account
15:29 pkiraly
@Alejandra69, here is a detailed description: https://help.github.com/en/articles/fork-a-repo
15:29 Alejandra69
Thank you
15:32 pdurbin
Alejandra69: hi, yes, like pkiraly said, you need to fork "dataverse" so that it appears under your GitHub username instead of IQSS.
15:34 pdurbin
pkiraly: I can work some more on that "Get JSON Representation of a Dataset" doc if you want. I was in a hurry to try to show you what I meant. Again, "Get JSON Representation of a Dataset" is challenging and thorny. I think I'd like to point you at a different doc to work on first if that's ok.
15:44 pkiraly
@pdurbin, OK, please do. I'll work on it tomorrow.
15:49 pdurbin
pkiraly: thanks! Here's a fresh issue for you! https://github.com/IQSS/dataverse/issues/6190
15:49 pkiraly
great! ;-)
15:50 pdurbin
pkiraly: you should have gotten an email to join https://github.com/orgs/IQSS/teams/dataverse-readonly . Once you join, I can assign the issue to you.
15:57 pkiraly
I got it, and accepted the challange!
16:00 pdurbin
pkiraly: assigned! Thanks!
16:01 pdurbin
donsizemore: full curl command coming ^^
16:13 pkiraly
@pdurbin, Bye, and have a nice day!
16:14 pdurbin
you too!
17:26 pdurbin
Alejandra69: are you all set? Anything I can do to help?
17:55 Alejandra69
@pdurbin It's All right, we've done the PR.
17:55 Alejandra69
Thank's
18:10 pdurbin
Alejandra69: I just took a quick look. Do you want to add some docs? :) In oauth2.rst I mean.
18:25 Alejandra69
@pdurbin, Yeah, right. How do I edit the oauth2.rst file?
18:26 pdurbin
Alejandra69: if you want I could add a TODO or FIXME to it in your branch so that you see where it is.
18:31 Alejandra69
@pdurbin, sure, it's done.
18:33 pdurbin
Alejandra69: Danny is taking a look.
18:34 pdurbin
He's in my office now. :)
19:12 pdurbin
Alejandra69: I'm already talking about your new pull request: https://groups.google.com/d/msg/dataverse-community/rCsRC4Vvqjw/eg_8hLg1BAAJ :)
19:13 pdurbin
donsizemore: Are you around? I'm on your 15_ansible_role branch. Do I just run the ec2 create script?
19:23 pdurbin
donsizemore: with no args I get an error so I'm suggesting this fix: https://github.com/OdumInstitute/dataverse-jenkins/pull/1
19:38 donsizemore
@pdurbin hey, still busy with Odum stuff. what I pushed worked for me but at this point it may be tomorrow
19:39 pdurbin
donsizemore: no worries, I just keep saying at standup that I'm going to test your branch and I wanted to actually do it. :)
19:39 donsizemore
@pdurbin i'm not sure that i ever ran it with no arguments, so...
19:39 pdurbin
Gotcha. I can try again *with* args if you tell me what to use. :)
19:40 donsizemore
I did this: $ec2/ec2-create-instance.sh -b 15_ansible_role -r https://github.com/OdumInstitute/dataverse-jenkins.git -g ~/group_vars/dataverse-jenkins.yml
19:40 donsizemore
all i did in group_vars was change the admin password
19:42 pdurbin
lemme give that a shot
19:42 pdurbin
I don't have a group_vars directory.
19:43 donsizemore
i just made it locally for my admin password
19:43 pdurbin
oh
19:43 pdurbin
can I skip that part?
19:43 donsizemore
absolutely
19:44 pdurbin
ok. deploying to c4.large
19:44 pdurbin
I feel like we're moving up in the world. Bigger apartment.
19:45 donsizemore
deluxe, even?
19:45 pdurbin
our first apartments in Boston were in basements
19:45 pdurbin
we had a lot of sticker shock moving from the Midwest
19:46 donsizemore
yeah, if you ran that version of ec2 without the branch and repo flags and the PR wasn't merged, you didn't get the new stuff
19:46 donsizemore
i'm hearing there is concern that the vegan restaurants in NC won't be as good as Boston's
19:55 pdurbin
I see Jenkins! http://ec2-3-208-27-81.compute-1.amazonaws.com:8080/
19:56 pdurbin
hmm, I know a vegan or two
19:57 pdurbin
donsizemore: should I resolve the merge conflicts and merge this thing?
20:02 donsizemore
sounds good to me. there are merge conflicts already?
20:04 Alejandra31 joined #dataverse
20:07 pdurbin
yeah. config.xml
20:08 donsizemore
sowwy
20:08 pdurbin
which version do you want?
20:08 donsizemore
let me look at them
20:08 pdurbin
the one in master or the one in your branch?
20:08 pdurbin
ok
20:10 donsizemore
my branch
20:10 donsizemore
which allows IQSS-Dataverse-Develop to import cleanly (in its current state)
20:11 pdurbin
you're the boss
20:13 pdurbin
that's weird... master merged cleanly locally
20:16 pdurbin
donsizemore: can you please merge https://github.com/OdumInstitute/dataverse-jenkins/pull/2 if you're happy with it?
21:47 pdurbin joined #dataverse
