Time
S
Nick
Message
06:46 poikilotherm joined #dataverse
07:43 pkiraly joined #dataverse
09:23 pdurbin
pkiraly: you're becoming a regular. :) And I just looked at the "who's who" spreadsheet in the topic. Now that your installation of Dataverse is on the map, I'll add a link to it. :)
09:24 pdurbin
poikilotherm: I'm looking forward to getting your installation on the map. :)
09:28 pkiraly
@pdurbin, I just created the pull request. Now I have to leave for the Climate Strike demonstration (https://fridaysforfuture.de/allefuersklima/) , but later I'll be back. I have a question to consider meantime: $SERVER or $SERVER_URL?
09:28 pkiraly
I was not aware that we are already on the map ;-)
09:29 pdurbin
pkiraly: yep. Announced in July: https://groups.google.com/d/msg/dataverse-community/dX2pZDNPzU0/KQnwqNMIBQAJ
09:34 pdurbin
pkiraly: $SERVER_URL. I just left you a review on https://github.com/IQSS/dataverse/pull/6194 :)
09:35 pdurbin
Have fun at the demonstration. :)
09:57 poikilotherm
Good morning guys :-)
09:57 pdurbin
Mornin'.
09:58 poikilotherm
pdurbin I hope I wasn't too pushy on https://github.com/IQSS/dataverse/pull/6192
09:59 pdurbin
poikilotherm: here you were a little pushy: https://github.com/IQSS/dataverse/pull/6146#issuecomment-530901855 :)
10:00 poikilotherm
Ok, so my change request about not including code which is available from upstream was ok?
10:00 poikilotherm
About #6146: I didn't mean to be pushy :-( Maybe its a language/ culture issue?
10:01 pdurbin
Maybe. "Pushy" was the word used at standup back then. :)
10:01 pdurbin
Not by me. :)
10:01 pdurbin
But I nodded. :)
10:01 poikilotherm
Guess you know me best :-D
10:02 pdurbin
Maybe someday we'll have a code of conduct and we'll say, "Don't be pushy." :)
10:03 poikilotherm
Yeah. I really try not to be pushy. I meant to ask gentle and serville, as I know you guys are all swamped
10:03 pdurbin
But yeah, I'm glad you pointed out to alejandratenorio that upgrading ScribeJava will mean getting a Microsoft provider for free. I didn't know that. Maybe she didn't. Did you see she was in IRC yesterday? Hopefully we can talk more to her.
10:03 poikilotherm
Yeah, I read the logs :-)
10:04 poikilotherm
Hope she's coming back so we can talk
10:04 pdurbin
Ah, so you saw the good news... What are you going to call your branch? :)
10:04 poikilotherm
I saw it the moment you added me to dataverse-write :-D
10:04 poikilotherm
(Does that mean I haven't been too pushy altogether?)
10:05 poikilotherm
GitHub send me a notification
10:05 pdurbin
Heh, you're fine. But it may be a cultural thing. I dunno. bencomp was pushy too. I doubt he'd mind me saying that. He still follows me on Twitter. :)
10:06 pdurbin
Please push and pull us in healthy directions. We want to keep growing. :)
10:06 poikilotherm
Yes Sir!
10:07 poikilotherm
Challenge accepted :-D
10:07 pdurbin
So what's the name of the branch gonna be?
10:07 poikilotherm
Are you a Star Trek fan?
10:07 poikilotherm
We could call it "next generation"
10:07 poikilotherm
Or "deep space nine"
10:07 poikilotherm
(Just kidding)
10:08 pdurbin
I was more of a Next Generation guy. I only watched a bit of Deep Space Nine (and the original series).
10:08 pdurbin
Danny was asking why to call it "next" instead of "payara". Any thoughts on this?
10:09 poikilotherm
borg-cube? "We are Dataverse. Resistance is futile. Your life as it has been is over. From this time forward, we will service you in research data management"
10:10 poikilotherm
Well I thought it might be usefull for other breaking stuff, too
10:10 poikilotherm
You once told me that DV 3 was on GF 3, DV 4 on GF 4
10:10 pdurbin
yeah
10:10 poikilotherm
So I thought you might stick with that
10:10 poikilotherm
And use DV 5 on Payara 5
10:11 poikilotherm
This is a good idea anyway, is we will do breaking changes
10:11 poikilotherm
Which always should have a major release version
10:11 pdurbin
That's fine but eventually I want Dataverse to run on OpenLiberty and TomEE and down the road... Quarkus.
10:11 poikilotherm
Yeah
10:11 poikilotherm
So Payara might be to special
10:12 pdurbin
I think there's already a milestone for Dataverse 5.
10:12 poikilotherm
So I would use sth like "next" or "next-generation" or even shorter "ng"
10:12 poikilotherm
That milestone has been there for quite a while
10:12 poikilotherm
And people talk about DataTags and such, but there is not much movement
10:12 pdurbin
It says "5.0 - DataTags and Support for Sensitive Data": https://github.com/IQSS/dataverse/milestone/56
10:13 pdurbin
And under that milestone is "Glassfish 5 Upgrade" flagged with "Too Large to Estimate" :( T-T
10:15 pdurbin
But you'll help us get off Glassfish 4, right? :)
10:16 poikilotherm
Yes :-D
10:18 pdurbin
"use Payara for any branch" sounds awesome: https://github.com/IQSS/dataverse-kubernetes/pull/88
10:19 poikilotherm
It works quite well. Altough I will try to reduce memory usage of the dev container.
10:19 poikilotherm
By default it will eat your RAM for breakfast
10:19 pdurbin
yum
10:20 poikilotherm
The container is in production mode, so it wil allocate 2G heap space on startup
10:20 pdurbin
To spin it up on AWS I should follow https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/aws-kops.md , right?
10:20 poikilotherm
That's fine for real use cases, but a lot on a small dev platform :-D
10:20 poikilotherm
More or less. You will need a K8s cluster on AWS first
10:21 poikilotherm
After that you can use Skaffold, but we need to test how to push the images over there
10:21 poikilotherm
I don't want them uploaded to Docker Hub, as it would clutter tags there
10:21 pdurbin
Oh. How do I get a k8s cluster on AWS?
10:21 poikilotherm
Different options. You can use kops, kubeadmin, ...
10:22 pdurbin
Ok, so "Use a tool to deploy an EC2 based K8s cluster" will give me that.
10:22 poikilotherm
Aye
10:22 pdurbin
How many EC2 instances will it spin up?
10:22 poikilotherm
You can configure that. IIRC default is 1
10:22 poikilotherm
Per K8s service
10:23 pdurbin
perfect
10:23 pdurbin
oh, per service? how many services are there?
10:23 poikilotherm
Gimme a sec, I go looking
10:24 poikilotherm
https://kubernetes.io/docs/setup/production-environment/tools/kops/
10:25 pdurbin
I'm a little lost in those docs. Is it 3 services? 30?
10:27 poikilotherm
It looks like you will have 1 master node instance and 2 pod node instances by defaul
10:27 poikilotherm
https://asciinema.org/a/97298
10:27 pdurbin
cool, thanks
10:29 poikilotherm
:-)
10:34 pdurbin
Should we create a project board for the new branch?
10:35 poikilotherm
Sure :-)
10:35 poikilotherm
So do we have a name?
10:36 poikilotherm
I would prefer have a more general name, so other could do stuff in there, too.
10:37 poikilotherm
Naming it Payara sounds too limited to me, but I'll use whatever seems appropriate to you
10:37 pdurbin
I'm fine with whatever. glassfish4begone
10:40 pdurbin
thanks-for-the-fish
10:49 poikilotherm
LOL
10:49 poikilotherm
I like that one
10:49 poikilotherm
I do like Douglas Adams
10:50 poikilotherm
Let's call it 42. No one will ever know what this is about, but that's the answer :-D
10:50 pdurbin
I'm fine with 42. For both the branch and the project?
10:51 poikilotherm
Ok joking aside. Let's be realistic. A speaking name would be a good choice.
10:52 poikilotherm
Would you like to stress about that this is primarily about the move away from GF?
10:52 poikilotherm
Or would you like to see other contributions, too
10:53 pdurbin
What I want is MicroProfile APIs, while keeping the Java EE APIs, of course. I also don't want to feel stuck on any particular dependency.
10:53 poikilotherm
Ok.
10:54 poikilotherm
Then just use sth. serious as "appsrv-move"
10:54 pdurbin
I feel stuck on Glassfish 4.1.
10:55 poikilotherm
That should be general enough without focusing too much on payara
10:55 poikilotherm
And I like it short and quick
10:55 pdurbin
We could call it "microprofile".
10:55 poikilotherm
But that would just be a subset of things
10:55 poikilotherm
There's a lot of work ahead
10:56 poikilotherm
BTW I would really like to know about Gustavos findings. We should invite him to the board and collect ideas over there
10:57 pdurbin
Absolutely. In Slack he's been talking about Payara and TomEE.
10:58 pdurbin
I noticed he joined this channel: http://irclog.iq.harvard.edu/dataverse/2019-09-18#i_105834
10:58 poikilotherm
Yeah, I saw that, too :-D
10:59 poikilotherm
Dunno why he did not stay
11:02 poikilotherm
Do you want to create the branch and project?
11:02 poikilotherm
I'm working on tests for #5997
11:03 poikilotherm
Great. Found the first NPE when signing in via Google
11:04 pdurbin
Sure. Want to call the branch "microprofile" and the project "MicroProfile Support"? Or do you prefer "payara" and "Payara Support"?
11:04 poikilotherm
Hehehe. Doing a 360 here :-D
11:05 poikilotherm
[12:54] <poikilotherm> Then just use sth. serious as "appsrv-move"
11:05 poikilotherm
But "payara" is fine, too
11:05 pdurbin
appsrv-move doesn't bring me joy
11:06 pdurbin
I'll create the project. You can create the branch when you're ready. Maybe fix those NPEs first. :)
11:06 poikilotherm
Oh, you want joy... Then use "badass-fish".
11:07 poikilotherm
https://blog.payara.fish/hubfs/Payara_Blog_Images/Be_badass_go_Micro.jpg
11:13 poikilotherm
BTW pdurbin: did Danny mention when he wants to cut 4.17?
11:14 poikilotherm
Oh and should I add a link to the mentioned ScribeJava Client to my comment? Commented on my mobile yesterday evening while glueing vapor retarder foil in my roof...
11:17 pdurbin
poikilotherm: ok, I created a project called "Dataverse on Payara". Please look at the description at https://github.com/orgs/IQSS/projects and let me know what I'm forgetting, especially in terms of benefits.
11:23 poikilotherm
I added a few points... :_D
11:24 pdurbin
Good point about Hazelcast. I almost put that in there. But maybe we should use the name of the standard instead.
11:25 poikilotherm
Done
11:26 pdurbin
Better but let me go find the JSR.
11:27 pdurbin
JSR 107: JCache is what I was thinking of: https://github.com/IQSS/dataverse/issues/2100
11:30 poikilotherm
Aye. JCache. But that's not what has been on my mind
11:30 poikilotherm
I thought about things like distributed timers, locks etc
11:31 poikilotherm
Is this also part of JCache?
11:32 poikilotherm
I doubt that after reading https://blog.payara.fish/glassfish-to-payara-server-migration-hazelcast-as-a-coherence-active-cache-alternative
11:33 poikilotherm
https://docs.payara.fish/documentation/payara-server/hazelcast/
11:33 poikilotherm
JCache is one part of the "Domain Data Grid"
11:35 pdurbin
When you're done editing, please let me know and I'll add JCache to the list.
11:36 poikilotherm
Done :-D
11:36 poikilotherm
Go add JCache :-)
11:37 poikilotherm
Reads like you already have a specific use case in your mind?
11:37 poikilotherm
I'm curious to learn about that :-)
11:37 pdurbin
Please see that issue. :)
11:38 *Â poikilotherm
goes reading
11:40 pdurbin
and please watch https://www.youtube.com/watch?v=FscLn-9crfE#t=12m21s
11:42 poikilotherm
So you would like to cache... what? DB results? Index results?
11:43 pdurbin
db results
11:43 pdurbin
I'd like to play with it at least. :)
11:43 pdurbin
I'm done adding JCache stuff.
11:43 pdurbin
poikilotherm: do you want to email dataverse-dev about the new board?
11:48 pdurbin
We could add to both of these threads:
11:48 poikilotherm
Maybe. Definitly not know :-D
11:48 poikilotherm
-k
11:48 pdurbin
- founder of Payara wants to help Dataverse run on Payara: https://groups.google.com/d/msg/dataverse-dev/07dyNe5BvJA/GxsZLPEJBwAJ
11:48 pdurbin
- Future use of Glassfish and/or Payara: https://groups.google.com/d/msg/dataverse-dev/L6rNXbbZG0o/UFVfI0_gAQAJ
11:50 poikilotherm
pdurbin do you think I should refactor http://github.com/poikilotherm/dataverse/blob/5991-oidc-provider/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/AbstractOAuth2AuthenticationProvider.java#L134-L134
11:51 poikilotherm
My auth with Google failed and the error message reads like sth failed with ORCID, which is obviously wrong
12:00 pdurbin
poikilotherm: yes, definitely.
12:00 pdurbin
good catch
12:00 pdurbin
Also, I added those two threads from dataverse-dev to the "Dataverse on Payara" project description.
12:01 pdurbin
poikilotherm: you could even report it as a bug if you can replicate it in the develop branch.
12:04 pkiraly
@pdurbin, I am back. I accepted your suggestions, and added the note. What's the next step? I could go forward, and do the same at the next section. This would involve a new ticket, new branch, new pull request.
12:13 pkiraly
I've created the ticket: https://github.com/IQSS/dataverse/issues/6195
12:19 poikilotherm
pdurbin: looks like I added this myself. I should add a few message to the bundle then instead. Shall I attack those places you mentioned in the PR, too?
12:21 pdurbin
pkiraly: I just clicked "Approve" to move your pull request to QA. Thank you!!
12:21 donsizemore joined #dataverse
12:23 pkiraly
here is the next one: https://github.com/IQSS/dataverse/pull/6196
12:23 pdurbin
pkiraly: I don't think /api/info/server does anything useful, unfortunately. Check out https://dataverse.harvard.edu/api/info/server
12:23 pdurbin
poikilotherm: you can't replicate that bug in develop? Phew.
12:23 pkiraly
it says: "This is useful when a Dataverse system is composed of multiple Java EE servers behind a load balancer"
12:24 poikilotherm
Nope. I added check if the auth service returns the scope we requested
12:24 poikilotherm
+a
12:24 poikilotherm
And used that info message which in develop is only present in the ORCID provider
12:24 pkiraly
@pdurbin, is it OK to continue with api/info/settings/:DatasetPublishPopupCustomText?
12:27 pdurbin
pkiraly: yes, please!
12:27 pdurbin
poikilotherm: so we need a switch statement?
12:28 poikilotherm
Maybe. But it would be better to extend the providers and use the interface already present
12:28 poikilotherm
-interface +abstract base class
12:31 poikilotherm
Oh oh
12:31 poikilotherm
[dataverse-5fcf65f655-kskps dataverse] Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.ex​ceptions.DatabaseException
12:31 poikilotherm
[dataverse-5fcf65f655-kskps dataverse] Internal Exception: org.postgresql.util.PSQLException: ERROR: value too long for type character varying(64)
12:31 poikilotherm
[dataverse-5fcf65f655-kskps dataverse] Error Code: 0
12:31 poikilotherm
[dataverse-5fcf65f655-kskps dataverse] Call: INSERT INTO OAUTH2TOKENDATA (ACCESSTOKEN, EXPIRYDATE, OAUTHPROVIDERID, RAWRESPONSE, REFRESHTOKEN, SCOPE, TOKENTYPE, USER_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
12:34 pdurbin
yikes
12:34 poikilotherm
Wow, Google auth doesn't seem to be in use very much, does it?
12:35 poikilotherm
I bet my shoes on that this blew up due to the scope string of Google auth provider
12:35 pdurbin
donsizemore has Google auth enabled at https://dataverse.unc.edu (good morning). And it's enabled at https://dataverse.harvard.edu too.
12:35 poikilotherm
http://github.com/poikilotherm/dataverse/blob/5991-oidc-provider/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/impl/GoogleOAuth2AP.java#L25-L25
12:36 poikilotherm
And scope is limited to 64 chars
12:36 poikilotherm
http://github.com/poikilotherm/dataverse/blob/5991-oidc-provider/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/OAuth2TokenData.java#L61-L62
12:36 poikilotherm
That Google scopes have 96 char
12:36 pdurbin
poikilotherm: I just created some columns on the new "Dataverse on Payara" board: https://github.com/orgs/IQSS/projects/16 . donsizemore, you're the only one with cards in "Done". Thank you! :)
12:37 poikilotherm
I want one in Done too :-D Let me merge that Payara stuff :-D
12:37 pdurbin
poikilotherm: please drag to "Doing" anything you're working on. :)
12:38 pdurbin
donsizemore: annoyingly, to read the (long) description of the new "Dataverse on Payara" board, you have to go to https://github.com/orgs/IQSS/projects . poikilotherm and I would love feedback!
12:38 pdurbin
(from everybody here)
12:57 pdurbin
pkiraly: I have "to do" columns for donsizemore and poikilotherm at https://github.com/orgs/IQSS/projects/3 . Would you like me to make one for you? :)
13:06 pkiraly
pdurbin, that's OK
13:07 donsizemore
@pdurbin morning. let me get Google happy with UNC Dataverse's sitemap and I'll be back on the ansible train
13:07 pkiraly
I gave you some more pull requests.
13:08 donsizemore
@pdurbin I can call it via HTTP and xmllint likes it, but Google doesn't
13:13 poikilotherm
OK guys, gotta run... Cu next week
13:28 pdurbin
pkiraly: thanks! Here's your column: https://github.com/orgs/IQSS/projects/3#column-6560524 . What should we put in it? :)
13:28 pdurbin
donsizemore: perfect, please ping me because I did a lot of testing late yesterday and it's probably hard to follow :)
13:29 donsizemore
i merged both of your PRs-on-my-PRs
13:29 pdurbin
ok, lemme go look, thanks
13:29 donsizemore
i'm re-running the 6142 branch first in vagrant, then in ec2
13:30 pdurbin
ok
13:32 pkiraly
pdurbin, you can give me a list of API calls to transform
13:33 pdurbin
pkiraly: sounds good. What about any of these? https://github.com/IQSS/dataverse/issues/created_by/pkiraly
13:39 pdurbin
donsizemore: can I merge https://github.com/IQSS/dataverse-jenkins/pull/16 ?
13:40 donsizemore
@pdurbin i can test that one more time before merging? unless you're happy with the secondary PR
13:40 pdurbin
I'm happy. Works on my machine.
13:41 donsizemore
then it works for me =)
13:42 pdurbin
perfect, merged, thanks!
13:46 pkiraly
@pdurbin, you can add them as well
13:47 pdurbin
pkiraly: ok, I'll put them in order of priority
13:56 donsizemore
@pdurbin does https://dataverse.unc.edu/sitemap/sitemap.xml look good to you?
13:59 pdurbin
donsizemore: yeah, looks fine, similar to https://demo.dataverse.org/sitemap.xml
14:02 donsizemore
Google sez "Sitemap could not be read" (general HTTP error)
14:02 donsizemore
with or without the parent directory
14:02 pdurbin
donsizemore: oh, yours has sitemap/sitemap.xml. Does that matter?
14:03 pdurbin
oh, that's the parent directory you're talking about :)
14:03 pdurbin
this works fine: curl https://dataverse.unc.edu/sitemap.xml | xmllint -format -
14:03 pdurbin
no validation errors
14:03 pdurbin
hmm
14:05 pdurbin
donsizemore: known bug according to https://stackoverflow.com/questions/53921636/google-search-console-fails-to-fetch-sitemaps-sitemap-could-not-be-read/55717596#55717596
14:05 pdurbin
but who knows
14:09 pdurbin
pkiraly: I just added an issue to your new to do board. :)
14:19 pdurbin
pkiraly: ok, now there are 13 (!) issues in your column. Please ping me when you're ready to talk about what you'd like to work on next. :)
14:19 pdurbin
I have an idea of which ones are small.
14:19 pdurbin
bricas_ created one of them so maybe you two can team up. :)
14:50 pkiraly
I have to go through all these tickets first. Next week I'll travel, so get back to you the week after.
14:52 pdurbin
pkiraly: enjoy the travel but please don't go through all 13 tickets. :) I'm happy to talk about the first 2 or 3. :)
14:52 pkiraly
OK
15:20 donsizemore
@pdurbin just tried 6142-solr-schema-changes in vagrant and ec2
15:42 pdurbin
donsizemore: great! And? :)
16:16 pkiraly joined #dataverse
17:19 donsizemore
@pdurbin develop branch deploys fine for me in vagrant and ec2
17:51 pdurbin
donsizemore: ok, let me try it again. I assume it doesn't matter how I specify the develop branch (flag or yaml). Thanks.
18:35 pdurbin
donsizemore: I'm so confused.
18:39 pdurbin
oh, I see my mistake
18:46 donsizemore
what happened?
18:47 pdurbin
Hmm, after I corrected my mistake, I still see the same "Unsupported parameters for (copy) module: with_fileglob" error I reported yesterday.
18:48 pdurbin
donsizemore: thie fileglob error: https://github.com/IQSS/dataverse-ansible/pull/105#issuecomment-533310423
18:48 pdurbin
this*
18:49 donsizemore
weird. on ec2? (works for me)
18:50 pdurbin
yeah, ec2
18:50 pdurbin
Are you specifying develop on the command line or in the yaml?
19:02 pdurbin
I've been putting "develop" in the yaml.
19:03 donsizemore
hmm, if i give it a branch without a repo... let me see what's going on
19:10 pdurbin
trying again with "develop" as a flag instead of having it in the yaml
19:14 pdurbin
donsizemore: same with_fileglob error when I put "develop" as a flag like this: ./ec2-create-instance.sh -g ec2config.yaml -b develop -r https://github.com/IQSS/dataverse.git -a dataverse-6142-solr-schema-changes
19:14 pdurbin
What am I doing wrong?
19:15 donsizemore
i'm re-running the role. i want to fix the branch/repo pair as well
19:17 pdurbin
Ok. I'm spinning up new each time.
20:40 pdurbin
donsizemore: for me this is all gonna wait for manana
20:40 pdurbin
I think you should go home.
20:40 pdurbin
You should all go home. I am. Have a good weekend!
20:40 pdurbin left #dataverse
21:12 donsizemore
@pdurbin i just got back from the gym
