IQSS logo

IRC log for #dataverse, 2019-09-20

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

All times shown according to UTC.

Time S Nick Message
06:46 poikilotherm joined #dataverse
07:43 pkiraly joined #dataverse
09:23 pdurbin pkiraly: you're becoming a regular. :) And I just looked at the "who's who" spreadsheet in the topic. Now that your installation of Dataverse is on the map, I'll add a link to it. :)
09:24 pdurbin poikilotherm: I'm looking forward to getting your installation on the map. :)
09:28 pkiraly @pdurbin, I just created the pull request. Now I have to leave for the Climate Strike demonstration (https://fridaysforfuture.de/allefuersklima/), but later I'll be back. I have a question to consider meantime: $SERVER or $SERVER_URL?
09:28 pkiraly I was not aware that we are already on the map ;-)
09:29 pdurbin pkiraly: yep. Announced in July: https://groups.google.com/d/msg/dataverse-community/dX2pZDNPzU0/KQnwqNMIBQAJ
09:34 pdurbin pkiraly: $SERVER_URL. I just left you a review on https://github.com/IQSS/dataverse/pull/6194 :)
09:35 pdurbin Have fun at the demonstration. :)
09:57 poikilotherm Good morning guys :-)
09:57 pdurbin Mornin'.
09:58 poikilotherm pdurbin I hope I wasn't too pushy on https://github.com/IQSS/dataverse/pull/6192
09:59 pdurbin poikilotherm: here you were a little pushy: https://github.com/IQSS/dataverse/pull/6146#issuecomment-530901855 :)
10:00 poikilotherm Ok, so my change request about not including code which is available from upstream was ok?
10:00 poikilotherm About #6146: I didn't mean to be pushy :-( Maybe its a language/ culture issue?
10:01 pdurbin Maybe. "Pushy" was the word used at standup back then. :)
10:01 pdurbin Not by me. :)
10:01 pdurbin But I nodded. :)
10:01 poikilotherm Guess you know me best :-D
10:02 pdurbin Maybe someday we'll have a code of conduct and we'll say, "Don't be pushy." :)
10:03 poikilotherm Yeah. I really try not to be pushy. I meant to ask gentle and serville, as I know you guys are all swamped
10:03 pdurbin But yeah, I'm glad you pointed out to alejandratenorio that upgrading ScribeJava will mean getting a Microsoft provider for free. I didn't know that. Maybe she didn't. Did you see she was in IRC yesterday? Hopefully we can talk more to her.
10:03 poikilotherm Yeah, I read the logs :-)
10:04 poikilotherm Hope she's coming back so we can talk
10:04 pdurbin Ah, so you saw the good news... What are you going to call your branch? :)
10:04 poikilotherm I saw it the moment you added me to dataverse-write :-D
10:04 poikilotherm (Does that mean I haven't been too pushy altogether?)
10:05 poikilotherm GitHub send me a notification
10:05 pdurbin Heh, you're fine. But it may be a cultural thing. I dunno. bencomp was pushy too. I doubt he'd mind me saying that. He still follows me on Twitter. :)
10:06 pdurbin Please push and pull us in healthy directions. We want to keep growing. :)
10:06 poikilotherm Yes Sir!
10:07 poikilotherm Challenge accepted :-D
10:07 pdurbin So what's the name of the branch gonna be?
10:07 poikilotherm Are you a Star Trek fan?
10:07 poikilotherm We could call it "next generation"
10:07 poikilotherm Or "deep space nine"
10:07 poikilotherm (Just kidding)
10:08 pdurbin I was more of a Next Generation guy. I only watched a bit of Deep Space Nine (and the original series).
10:08 pdurbin Danny was asking why to call it "next" instead of "payara". Any thoughts on this?
10:09 poikilotherm borg-cube? "We are Dataverse. Resistance is futile. Your life as it has been is over. From this time forward, we will service you in research data management"
10:10 poikilotherm Well I thought it might be usefull for other breaking stuff, too
10:10 poikilotherm You once told me that DV 3 was on GF 3, DV 4 on GF 4
10:10 pdurbin yeah
10:10 poikilotherm So I thought you might stick with that
10:10 poikilotherm And use DV 5 on Payara 5
10:11 poikilotherm This is a good idea anyway, is we will do breaking changes
10:11 poikilotherm Which always should have a major release version
10:11 pdurbin That's fine but eventually I want Dataverse to run on OpenLiberty and TomEE and down the road... Quarkus.
10:11 poikilotherm Yeah
10:11 poikilotherm So Payara might be to special
10:12 pdurbin I think there's already a milestone for Dataverse 5.
10:12 poikilotherm So I would use sth like "next" or "next-generation" or even shorter "ng"
10:12 poikilotherm That milestone has been there for quite a while
10:12 poikilotherm And people talk about DataTags and such, but there is not much movement
10:12 pdurbin It says "5.0 - DataTags and Support for Sensitive Data": https://github.com/IQSS/dataverse/milestone/56
10:13 pdurbin And under that milestone is "Glassfish 5 Upgrade" flagged with "Too Large to Estimate" :( T-T
10:15 pdurbin But you'll help us get off Glassfish 4, right? :)
10:16 poikilotherm Yes :-D
10:18 pdurbin "use Payara for any branch" sounds awesome: https://github.com/IQSS/dataverse-kubernetes/pull/88
10:19 poikilotherm It works quite well. Altough I will try to reduce memory usage of the dev container.
10:19 poikilotherm By default it will eat your RAM for breakfast
10:19 pdurbin yum
10:20 poikilotherm The container is in production mode, so it wil allocate 2G heap space on startup
10:20 pdurbin To spin it up on AWS I should follow https://github.com/IQSS/dataverse-kubernetes/blob/master/docs/aws-kops.md , right?
10:20 poikilotherm That's fine for real use cases, but a lot on a small dev platform :-D
10:20 poikilotherm More or less. You will need a K8s cluster on AWS first
10:21 poikilotherm After that you can use Skaffold, but we need to test how to push the images over there
10:21 poikilotherm I don't want them uploaded to Docker Hub, as it would clutter tags there
10:21 pdurbin Oh. How do I get a k8s cluster on AWS?
10:21 poikilotherm Different options. You can use kops, kubeadmin, ...
10:22 pdurbin Ok, so "Use a tool to deploy an EC2 based K8s cluster" will give me that.
10:22 poikilotherm Aye
10:22 pdurbin How many EC2 instances will it spin up?
10:22 poikilotherm You can configure that. IIRC default is 1
10:22 poikilotherm Per K8s service
10:23 pdurbin perfect
10:23 pdurbin oh, per service? how many services are there?
10:23 poikilotherm Gimme a sec, I go looking
10:24 poikilotherm https://kubernetes.io/docs/setup/production-environment/tools/kops/
10:25 pdurbin I'm a little lost in those docs. Is it 3 services? 30?
10:27 poikilotherm It looks like you will have 1 master node instance and 2 pod node instances by defaul
10:27 poikilotherm https://asciinema.org/a/97298
10:27 pdurbin cool, thanks
10:29 poikilotherm :-)
10:34 pdurbin Should we create a project board for the new branch?
10:35 poikilotherm Sure :-)
10:35 poikilotherm So do we have a name?
10:36 poikilotherm I would prefer have a more general name, so other could do stuff in there, too.
10:37 poikilotherm Naming it Payara sounds too limited to me, but I'll use whatever seems appropriate to you
10:37 pdurbin I'm fine with whatever. glassfish4begone
10:40 pdurbin thanks-for-the-fish
10:49 poikilotherm LOL
10:49 poikilotherm I like that one
10:49 poikilotherm I do like Douglas Adams
10:50 poikilotherm Let's call it 42. No one will ever know what this is about, but that's the answer :-D
10:50 pdurbin I'm fine with 42. For both the branch and the project?
10:51 poikilotherm Ok joking aside. Let's be realistic. A speaking name would be a good choice.
10:52 poikilotherm Would you like to stress about that this is primarily about the move away from GF?
10:52 poikilotherm Or would you like to see other contributions, too
10:53 pdurbin What I want is MicroProfile APIs, while keeping the Java EE APIs, of course. I also don't want to feel stuck on any particular dependency.
10:53 poikilotherm Ok.
10:54 poikilotherm Then just use sth. serious as "appsrv-move"
10:54 pdurbin I feel stuck on Glassfish 4.1.
10:55 poikilotherm That should be general enough without focusing too much on payara
10:55 poikilotherm And I like it short and quick
10:55 pdurbin We could call it "microprofile".
10:55 poikilotherm But that would just be a subset of things
10:55 poikilotherm There's a lot of work ahead
10:56 poikilotherm BTW I would really like to know about Gustavos findings. We should invite him to the board and collect ideas over there
10:57 pdurbin Absolutely. In Slack he's been talking about Payara and TomEE.
10:58 pdurbin I noticed he joined this channel: http://irclog.iq.harvard.edu/dataverse/2019-09-18#i_105834
10:58 poikilotherm Yeah, I saw that, too :-D
10:59 poikilotherm Dunno why he did not stay
11:02 poikilotherm Do you want to create the branch and project?
11:02 poikilotherm I'm working on tests for #5997
11:03 poikilotherm Great. Found the first NPE when signing in via Google
11:04 pdurbin Sure. Want to call the branch "microprofile" and the project "MicroProfile Support"? Or do you prefer "payara" and "Payara Support"?
11:04 poikilotherm Hehehe. Doing a 360 here :-D
11:05 poikilotherm [12:54] <poikilotherm> Then just use sth. serious as "appsrv-move"
11:05 poikilotherm But "payara" is fine, too
11:05 pdurbin appsrv-move doesn't bring me joy
11:06 pdurbin I'll create the project. You can create the branch when you're ready. Maybe fix those NPEs first. :)
11:06 poikilotherm Oh, you want joy... Then use "badass-fish".
11:07 poikilotherm https://blog.payara.fish/hubfs/Payara_Blog_Images/Be_badass_go_Micro.jpg
11:13 poikilotherm BTW pdurbin: did Danny mention when he wants to cut 4.17?
11:14 poikilotherm Oh and should I add a link to the mentioned ScribeJava Client to my comment? Commented on my mobile yesterday evening while glueing vapor retarder foil in my roof...
11:17 pdurbin poikilotherm: ok, I created a project called "Dataverse on Payara". Please look at the description at https://github.com/orgs/IQSS/projects and let me know what I'm forgetting, especially in terms of benefits.
11:23 poikilotherm I added a few points... :_D
11:24 pdurbin Good point about Hazelcast. I almost put that in there. But maybe we should use the name of the standard instead.
11:25 poikilotherm Done
11:26 pdurbin Better but let me go find the JSR.
11:27 pdurbin JSR 107: JCache is what I was thinking of: https://github.com/IQSS/dataverse/issues/2100
11:30 poikilotherm Aye. JCache. But that's not what has been on my mind
11:30 poikilotherm I thought about things like distributed timers, locks etc
11:31 poikilotherm Is this also part of JCache?
11:32 poikilotherm I doubt that after reading https://blog.payara.fish/glassfish-to-payara-server-migration-hazelcast-as-a-coherence-active-cache-alternative
11:33 poikilotherm https://docs.payara.fish/documentation/payara-server/hazelcast/
11:33 poikilotherm JCache is one part of the "Domain Data Grid"
11:35 pdurbin When you're done editing, please let me know and I'll add JCache to the list.
11:36 poikilotherm Done :-D
11:36 poikilotherm Go add JCache :-)
11:37 poikilotherm Reads like you already have a specific use case in your mind?
11:37 poikilotherm I'm curious to learn about that :-)
11:37 pdurbin Please see that issue. :)
11:38 * poikilotherm goes reading
11:40 pdurbin and please watch https://www.youtube.com/watch?v=FscLn-9crfE#t=12m21s
11:42 poikilotherm So you would like to cache... what? DB results? Index results?
11:43 pdurbin db results
11:43 pdurbin I'd like to play with it at least. :)
11:43 pdurbin I'm done adding JCache stuff.
11:43 pdurbin poikilotherm: do you want to email dataverse-dev about the new board?
11:48 pdurbin We could add to both of these threads:
11:48 poikilotherm Maybe. Definitly not know :-D
11:48 poikilotherm -k
11:48 pdurbin - founder of Payara wants to help Dataverse run on Payara: https://groups.google.com/d/msg/dataverse-dev/07dyNe5BvJA/GxsZLPEJBwAJ
11:48 pdurbin - Future use of Glassfish and/or Payara: https://groups.google.com/d/msg/dataverse-dev/L6rNXbbZG0o/UFVfI0_gAQAJ
11:50 poikilotherm pdurbin do you think I should refactor http://github.com/poikilotherm/dataverse/blob/5991-oidc-provider/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/AbstractOAuth2AuthenticationProvider.java#L134-L134
11:51 poikilotherm My auth with Google failed and the error message reads like sth failed with ORCID, which is obviously wrong
12:00 pdurbin poikilotherm: yes, definitely.
12:00 pdurbin good catch
12:00 pdurbin Also, I added those two threads from dataverse-dev to the "Dataverse on Payara" project description.
12:01 pdurbin poikilotherm: you could even report it as a bug if you can replicate it in the develop branch.
12:04 pkiraly @pdurbin, I am back. I accepted your suggestions, and added the note. What's the next step? I could go forward, and do the same at the next section. This would involve a new ticket, new branch, new pull request.
12:13 pkiraly I've created the ticket: https://github.com/IQSS/dataverse/issues/6195
12:19 poikilotherm pdurbin: looks like I added this myself. I should add a few message to the bundle then instead. Shall I attack those places you mentioned in the PR, too?
12:21 pdurbin pkiraly: I just clicked "Approve" to move your pull request to QA. Thank you!!
12:21 donsizemore joined #dataverse
12:23 pkiraly here is the next one: https://github.com/IQSS/dataverse/pull/6196
12:23 pdurbin pkiraly: I don't think /api/info/server does anything useful, unfortunately. Check out https://dataverse.harvard.edu/api/info/server
12:23 pdurbin poikilotherm: you can't replicate that bug in develop? Phew.
12:23 pkiraly it says: "This is useful when a Dataverse system is composed of multiple Java EE servers behind a load balancer"
12:24 poikilotherm Nope. I added check if the auth service returns the scope we requested
12:24 poikilotherm +a
12:24 poikilotherm And used that info message which in develop is only present in the ORCID provider
12:24 pkiraly @pdurbin, is it OK to continue with api/info/settings/:DatasetPublishPopupCustomText?
12:27 pdurbin pkiraly: yes, please!
12:27 pdurbin poikilotherm: so we need a switch statement?
12:28 poikilotherm Maybe. But it would be better to extend the providers and use the interface already present
12:28 poikilotherm -interface +abstract base class
12:31 poikilotherm Oh oh
12:31 poikilotherm [dataverse-5fcf65f655-kskps dataverse] Exception [EclipseLink-4002] (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): org.eclipse.persistence.ex​ceptions.DatabaseException
12:31 poikilotherm [dataverse-5fcf65f655-kskps dataverse] Internal Exception: org.postgresql.util.PSQLException: ERROR: value too long for type character varying(64)
12:31 poikilotherm [dataverse-5fcf65f655-kskps dataverse] Error Code: 0
12:31 poikilotherm [dataverse-5fcf65f655-kskps dataverse] Call: INSERT INTO OAUTH2TOKENDATA (ACCESSTOKEN, EXPIRYDATE, OAUTHPROVIDERID, RAWRESPONSE, REFRESHTOKEN, SCOPE, TOKENTYPE, USER_ID) VALUES (?, ?, ?, ?, ?, ?, ?, ?)
12:34 pdurbin yikes
12:34 poikilotherm Wow, Google auth doesn't seem to be in use very much, does it?
12:35 poikilotherm I bet my shoes on that this blew up due to the scope string of Google auth provider
12:35 pdurbin donsizemore has Google auth enabled at https://dataverse.unc.edu (good morning). And it's enabled at https://dataverse.harvard.edu too.
12:35 poikilotherm http://github.com/poikilotherm/dataverse/blob/5991-oidc-provider/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/impl/GoogleOAuth2AP.java#L25-L25
12:36 poikilotherm And scope is limited to 64 chars
12:36 poikilotherm http://github.com/poikilotherm/dataverse/blob/5991-oidc-provider/src/main/java/edu/harvard/iq/dataverse/authorization/providers/oauth2/OAuth2TokenData.java#L61-L62
12:36 poikilotherm That Google scopes have 96 char
12:36 pdurbin poikilotherm: I just created some columns on the new "Dataverse on Payara" board: https://github.com/orgs/IQSS/projects/16 . donsizemore, you're the only one with cards in "Done". Thank you! :)
12:37 poikilotherm I want one in Done too :-D Let me merge that Payara stuff :-D
12:37 pdurbin poikilotherm: please drag to "Doing" anything you're working on. :)
12:38 pdurbin donsizemore: annoyingly, to read the (long) description of the new "Dataverse on Payara" board, you have to go to https://github.com/orgs/IQSS/projects . poikilotherm and I would love feedback!
12:38 pdurbin (from everybody here)
12:57 pdurbin pkiraly: I have "to do" columns for donsizemore and poikilotherm at https://github.com/orgs/IQSS/projects/3 . Would you like me to make one for you? :)
13:06 pkiraly pdurbin, that's OK
13:07 donsizemore @pdurbin morning. let me get Google happy with UNC Dataverse's sitemap and I'll be back on the ansible train
13:07 pkiraly I gave you some more pull requests.
13:08 donsizemore @pdurbin I can call it via HTTP and xmllint likes it, but Google doesn't
13:13 poikilotherm OK guys, gotta run... Cu next week
13:28 pdurbin pkiraly: thanks! Here's your column: https://github.com/orgs/IQSS/projects/3#column-6560524 . What should we put in it? :)
13:28 pdurbin donsizemore: perfect, please ping me because I did a lot of testing late yesterday and it's probably hard to follow :)
13:29 donsizemore i merged both of your PRs-on-my-PRs
13:29 pdurbin ok, lemme go look, thanks
13:29 donsizemore i'm re-running the 6142 branch first in vagrant, then in ec2
13:30 pdurbin ok
13:32 pkiraly pdurbin, you can give me a list of API calls to transform
13:33 pdurbin pkiraly: sounds good. What about any of these? https://github.com/IQSS/dataverse/issues/created_by/pkiraly
13:39 pdurbin donsizemore: can I merge https://github.com/IQSS/dataverse-jenkins/pull/16 ?
13:40 donsizemore @pdurbin i can test that one more time before merging? unless you're happy with the secondary PR
13:40 pdurbin I'm happy. Works on my machine.
13:41 donsizemore then it works for me =)
13:42 pdurbin perfect, merged, thanks!
13:46 pkiraly @pdurbin, you can add them as well
13:47 pdurbin pkiraly: ok, I'll put them in order of priority
13:56 donsizemore @pdurbin does https://dataverse.unc.edu/sitemap/sitemap.xml look good to you?
13:59 pdurbin donsizemore: yeah, looks fine, similar to https://demo.dataverse.org/sitemap.xml
14:02 donsizemore Google sez "Sitemap could not be read" (general HTTP error)
14:02 donsizemore with or without the parent directory
14:02 pdurbin donsizemore: oh, yours has sitemap/sitemap.xml. Does that matter?
14:03 pdurbin oh, that's the parent directory you're talking about :)
14:03 pdurbin this works fine: curl https://dataverse.unc.edu/sitemap.xml | xmllint -format -
14:03 pdurbin no validation errors
14:03 pdurbin hmm
14:05 pdurbin donsizemore: known bug according to https://stackoverflow.com/questions/53921636/google-search-console-fails-to-fetch-sitemaps-sitemap-could-not-be-read/55717596#55717596
14:05 pdurbin but who knows
14:09 pdurbin pkiraly: I just added an issue to your new to do board. :)
14:19 pdurbin pkiraly: ok, now there are 13 (!) issues in your column. Please ping me when you're ready to talk about what you'd like to work on next. :)
14:19 pdurbin I have an idea of which ones are small.
14:19 pdurbin bricas_ created one of them so maybe you two can team up. :)
14:50 pkiraly I have to go through all these tickets first. Next week I'll travel, so get back to you the week after.
14:52 pdurbin pkiraly: enjoy the travel but please don't go through all 13 tickets. :) I'm happy to talk about the first 2 or 3. :)
14:52 pkiraly OK
15:20 donsizemore @pdurbin just tried 6142-solr-schema-changes in vagrant and ec2
15:42 pdurbin donsizemore: great! And? :)
16:16 pkiraly joined #dataverse
17:19 donsizemore @pdurbin develop branch deploys fine for me in vagrant and ec2
17:51 pdurbin donsizemore: ok, let me try it again. I assume it doesn't matter how I specify the develop branch (flag or yaml). Thanks.
18:35 pdurbin donsizemore: I'm so confused.
18:39 pdurbin oh, I see my mistake
18:46 donsizemore what happened?
18:47 pdurbin Hmm, after I corrected my mistake, I still see the same "Unsupported parameters for (copy) module: with_fileglob" error I reported yesterday.
18:48 pdurbin donsizemore: thie fileglob error: https://github.com/IQSS/dataverse-ansible/pull/105#issuecomment-533310423
18:48 pdurbin this*
18:49 donsizemore weird. on ec2? (works for me)
18:50 pdurbin yeah, ec2
18:50 pdurbin Are you specifying develop on the command line or in the yaml?
19:02 pdurbin I've been putting "develop" in the yaml.
19:03 donsizemore hmm, if i give it a branch without a repo... let me see what's going on
19:10 pdurbin trying again with "develop" as a flag instead of having it in the yaml
19:14 pdurbin donsizemore: same with_fileglob error when I put "develop" as a flag like this: ./ec2-create-instance.sh -g ec2config.yaml -b develop -r https://github.com/IQSS/dataverse.git -a dataverse-6142-solr-schema-changes
19:14 pdurbin What am I doing wrong?
19:15 donsizemore i'm re-running the role. i want to fix the branch/repo pair as well
19:17 pdurbin Ok. I'm spinning up new each time.
20:40 pdurbin donsizemore: for me this is all gonna wait for manana
20:40 pdurbin I think you should go home.
20:40 pdurbin You should all go home. I am. Have a good weekend!
20:40 pdurbin left #dataverse
21:12 donsizemore @pdurbin i just got back from the gym

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.