Time
S
Nick
Message
00:13 jri joined #dataverse
01:24 nanoz joined #dataverse
04:14 isullivan joined #dataverse
04:29 jri joined #dataverse
04:37 isullivan1 joined #dataverse
04:37 isullivan joined #dataverse
04:44 isullivan joined #dataverse
04:51 isullivan joined #dataverse
04:53 isullivan joined #dataverse
08:12 jri joined #dataverse
09:58 poikilotherm joined #dataverse
11:46 poikilotherm
Morning pdurbin :-)
11:56 pdurbin
poikilotherm: morning. I was thinking more about your database config idea. I wonder if it would help with https://github.com/IQSS/dataverse/issues/4765
12:15 poikilotherm
pdurbin if you see any chance, I would really like to go for https://docs.payara.fish/documentation/microprofile/config.html
12:15 poikilotherm
This can be used with both Payara Full and Micro
12:16 poikilotherm
And it will solve more than just the database config
12:16 poikilotherm
Yet it keeps the "old way" possible, so people preferring the install script and/or asadmin can just do so.
12:17 poikilotherm
For us having the plan to go for Docker (and others, too) that would make a HUGE step forward in terms of possibilities for deployment
12:19 poikilotherm
I'm currently writing an "epic" as an GH issue that explains my ideas and plans
12:19 poikilotherm
After that I will write up an issue about the config stuff
12:20 poikilotherm
And yes indeed - it might help with GH-4765 as a more dynamic configuration can be done at deploy time rather than build time
12:20 pdurbin
Sounds perfect. I'm looking at https://github.com/eclipse/microprofile-config#implementations and it looks like Glassfish isn't on the list. Payara is. Any idea if Eclipse Glassfish will support "config"?
12:20 poikilotherm
Dunno...
12:21 poikilotherm
But in theory when you stick to the Config API spec, you should even be able to switch to Thorntail/Wildfly and other app servers
12:21 pdurbin
I guess we could ask at https://gitter.im/eclipse/microprofile-config
12:22 poikilotherm
I get the impression that all this stuff with changing to Payara 5 and more will need a Dataverse 5.0.0 tag...
12:23 poikilotherm
I'm afraid what this means in terms of getting this released in the near future...
12:23 poikilotherm
+of
12:24 pdurbin
I just asked: https://gitter.im/eclipse/microprofile-config?at=5be42ade9600d10e6479a093
12:24 donsizemore joined #dataverse
12:25 poikilotherm
Cool :-)
12:25 pdurbin
The pattern has been that DVN 3 ran on Glassfish 3, Dataverse 4 runs on Glassfish 4. But that wasn't on purpose, I don't think.
12:26 poikilotherm
Well anyone using somethng like 4.10 / 4.11 / ... might not expect that you need to change your application server...
12:26 pdurbin
Yeah, I guess.
12:27 poikilotherm
Of course this is possible, but it would need VERY good and tremendous amounts of communication to the users
12:27 barryr joined #dataverse
12:27 pdurbin
I dunno, we sort of try to follow semver.
12:27 poikilotherm
Aye
12:27 poikilotherm
IMHO it would be better to make such a big change in a new major...
12:28 poikilotherm
But I dunno if this means "oh we have some other tons of stuff in the pipeline that needs to be done before"
12:28 pdurbin
We do have a ton of stuff to do. Have you seen our roadmap?
12:28 pdurbin
Or the Design Team column on our board?
12:29 poikilotherm
Sure! And that's why I am afraid this might stop us from getting this forward
12:29 pdurbin
Yeah.
12:30 pdurbin
Do you want to work on anything on our roadmap? :)
12:30 poikilotherm
...
12:30 poikilotherm
I could
12:30 poikilotherm
But we need to get our service in place :-D
12:31 pdurbin
Sure.
12:31 poikilotherm
And investing time to have it dockerized as is now could mean a waste of effort
12:31 pdurbin
We'd love to get you added to our map. We just added a new installation yesterday.
12:31 pdurbin
Dunno if you've seen the map.
12:31 poikilotherm
Not yet
12:31 poikilotherm
Errr... The addittion
12:32 poikilotherm
The map itself is already on my mind... ;-)
12:32 pdurbin
http://guides.dataverse.org/en/4.9.4/installation/config.html#putting-your-dataverse-installation-on-the-map-at-dataverse-org
12:33 poikilotherm
We don't have an installation yet... ;-)
12:34 poikilotherm
What are those grey dots? They don't look familiar
12:37 pdurbin
I forget what the grey dots are. They shouldn't be there. I think they're specific to Harvard Dataverse or something. Individual dataverses within that installation, maybe.
12:37 pdurbin
You can get the installations as JSON : https://services.dataverse.harvard.edu/miniverse/map/installations-json
12:38 poikilotherm
Thx
12:39 poikilotherm
GH-4248 (Glassfish 5 Upgrade) and GH-4260 (Java EE 8 Upgrade) are in the milestone for 5.0
12:39 pdurbin
poikilotherm: here you go. More on the grey dots: https://groups.google.com/d/msg/dataverse-community/eQlSLFgzQXI/t6FANVynBgAJ
12:39 poikilotherm
Maybe we can talk about reducing the 5.0 milestone to the upgrading, leaving the data tag stuff for a 5.1 and thus moving the stuff with 4.x on your roadmap to a 5.x?
12:40 poikilotherm
4.10 seems to be almost ready...
12:41 pdurbin
Almost, I guess, but we need to fix https://github.com/IQSS/dataverse/issues/5260
12:41 poikilotherm
Oh this is not in 4.10 milestone!
12:43 pdurbin
Yeah, I find the way we use milestones confusing. There are only 2 issues in the 4.10 milestone.
12:43 pdurbin
But we've already merged 58 pull requests for the next release (probably 4.10): https://github.com/IQSS/dataverse/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aclosed+is%3Amerged+no%3Amilestone
12:44 * poikilotherm
is puzzled
12:45 pdurbin
heh
12:45 pdurbin
I guess every project has its own culture.
12:46 pdurbin
Recently I added milestones to all the old pull requests we had already shipped in a release so I could do that query and get an accurate count.
12:46 pdurbin
Accurate count of what will be in the next release.
12:47 pdurbin
So that I can go back and add the milestone to all issues for that release.
12:47 pdurbin
So that we'll have more than two issues tagged with 4.10 or whatever. :)
12:53 poikilotherm
https://github.com/IQSS/dataverse/issues/5292
12:54 poikilotherm
This could be a good candidate for addition to a milestone... ;-)
12:57 pdurbin
Great write up. My first thought is that *everyone* still uses the "old" WAR approach right now. The 35 production installations, trial installations, devs, etc.
12:57 poikilotherm
Yes
12:57 poikilotherm
And in no way I would break that by bad design
12:58 poikilotherm
That's a no go
12:58 pdurbin
Good.
12:58 pdurbin
It reminds me a bit of https://github.com/IQSS/dataverse/issues/4040
12:58 poikilotherm
Sure!
12:59 poikilotherm
The OpenShift based things are using the install script, too!
12:59 poikilotherm
That's not microservices IMHO
12:59 poikilotherm
Good catch - I will add a reference in my epic
12:59 pdurbin
In that issue we explain that Red Hat may be interested in running Dataverse internally (or externally) but before they do they require that Dataverse run in the way they want, which is on OpenShift. For you, it's Docker/Kubernetes.
13:00 pdurbin
Which is fine but a number of people are already having success in production with the old ugly way. :)
13:00 poikilotherm
Of course it works
13:00 poikilotherm
But it "just works"
13:00 pdurbin
It barely works. :)
13:01 poikilotherm
As far as I can see, they still made an AIO approach
13:01 poikilotherm
At least from what I can see from the conf/docker/dataverse-glassfish tings
13:01 pdurbin
I think of it as project/product/team maturity. Folks like you and Red Hat are trying to help us mature. SBGrid also. It's very healthy for us.
13:02 poikilotherm
They already ripped out a lot of stuff, but still they start on a base image and install java, glassfish etc
13:02 pdurbin
Mature not just at a "you should add this feature" level but at an infrastructure level. Plumbing.
13:02 poikilotherm
I hope so :-D
13:02 poikilotherm
I am glad that you see value in that :-)
13:03 pdurbin
I was a sysadmin for 15 years.
13:03 poikilotherm
A lot of people out there are just on the "feature feature feature" side of life
13:03 poikilotherm
;-)
13:03 poikilotherm
That's why I like to call myself a DevOp
13:03 pdurbin
I try to stay far away from production systems. Don't give me the root password. But I symphathize with people who deal with production.
13:03 poikilotherm
Aye!
13:04 pdurbin
The scars will be with me forever. ;)
13:04 poikilotherm
;-)
13:06 pdurbin
This is the feature I'm working on right now: https://github.com/IQSS/dataverse/issues/2739
13:06 pdurbin
If anyone here has any opinions on how it should be implemented.
13:07 poikilotherm
Hell yeah! That's a great move :-)
13:08 poikilotherm
We are really looking forward to this
13:09 poikilotherm
Just my 2 cents I can spend: after a lot of talking to other RSEs, it seems valuable to store the code also inside dataverse and not just create a link to the git repo
13:10 poikilotherm
Many people out there would really like to have an archive that is foreseen to live at least 10 years, while the repository might be killed earlier
13:45 MrK joined #dataverse
13:46 pdurbin
poikilotherm: yes, the idea is to store the code itself (or whatever is in the git repo) in Dataverse as a zip (prototype: https://dev1.dataverse.org/file.xhtml?persistentId=doi:10.5072/FK2/FS7M3O/EBNKNB ) rather than only linking to the git repo. We're oriented toward preservation.
14:02 poikilotherm joined #dataverse
14:02 jri joined #dataverse
14:07 pdurbin
I just posted a screenshot at https://github.com/IQSS/dataverse/issues/5209#issuecomment-437003277
14:13 isullivan joined #dataverse
14:18 poikilotherm
pdurbin: my write up for the configuration stuff: https://github.com/IQSS/dataverse/issues/5293
14:20 poikilotherm
pdurbin: can you keep in mind that webhooks from gitlab should also work? they differ from the github ones... doesn't need to be implemented right away, but a concept of extendability from the beginning would be awesome
14:22 pdurbin
poikilotherm: ah, I didn't know if GitLab had webhooks or not.
14:23 pdurbin
poikilotherm: comments are welcome on https://github.com/IQSS/dataverse/issues/2739
14:24 poikilotherm
Will do so :-)
14:24 pdurbin
thanks
14:30 poikilotherm
Done. https://github.com/IQSS/dataverse/issues/2739#issuecomment-437012724
14:33 pdurbin
poikilotherm: thanks! I wrote a bit about CodeMeta at https://github.com/IQSS/dataverse/issues/3736
14:34 poikilotherm
Glad you already have that on the radar
14:34 pdurbin
me too
14:35 poikilotherm
ACtually CFF helps a lot with this... ;-) https://citation-file-format.github.io/1.0.0/specifications/
14:36 pdurbin
CFF is new to me. Thanks.
14:37 poikilotherm
Should I also leave a comment about CFF at GH-3736?
14:38 donsizemore joined #dataverse
14:43 pdurbin
It's up to you. I ended up closing that issue.
14:58 MrK
pdurbin: Hi, my friend responded to the yesterdays issue https://github.com/IQSS/dataverse/issues/5280 , and i just wanted to add that in the future, we will be able to merge those features into your instance if you would wish so.
15:03 poikilotherm
pdurbin: I got the response from Dataverse Support that I cannot get an account at your Slack. There are efforts to build up a Dataverse Consortium based Slack in progress.
15:04 poikilotherm
They suggested contacting them via email...
15:04 poikilotherm
I am very unsure if this is a good approach. Maybe for a first contact, but not for discussions.
15:05 poikilotherm
Any ideas? Stick with GH issues? Ask them to come to IRC ?
15:05 poikilotherm
Mos certainly this is important to MrK, too
15:23 pameyer joined #dataverse
15:23 pdurbin
poikilotherm: suggested contacting whom via email?
15:23 pdurbin
I'll go look at your ticket. One sec.
15:24 pdurbin
Oh, Gustavo and Matthew.
15:26 pdurbin
You could ask them to join us here. :)
15:28 poikilotherm
Maybe you could trick them to use Gitter... :-d
15:31 pameyer
"Gitter does not provide a regular password authentication. Instead, it asks for maximalist GitHub account permissions. " :(
15:31 pdurbin
Maybe. I think Slack is terrible for open source projects. Everything happening behind closed doors. No transparency.
15:32 poikilotherm
Yeah, as I told pdurbin yesterday: Gitter is not one of my favorites...
15:32 poikilotherm
But most certainly they like the Slack-XP and IRC is anything but a Slack-like experience
15:33 pdurbin
What I like about Gitter has nothing to do with technology. It has to do with norms and culture and what's healthy for communicating within open source projects. Transparency. Public by default. Sunlight.
15:34 poikilotherm
Ah sunlight...
15:34 poikilotherm
I got a shirt with something about sunlight on it...
15:34 poikilotherm
I think it reads like "Developer. Do not expose to direct sunlight."
15:34 pdurbin
I'm talking about stuff like https://en.wikipedia.org/wiki/Sunlight_Foundation
15:35 pdurbin
"The organization was founded in April 2006 with the goal of increasing transparency and accountability in the United States Congress, the executive branch, and in state and local governments."
15:35 poikilotherm
Oh I wasn't awre of that
15:35 poikilotherm
And it hasn't been closed by the recent president?
15:35 poikilotherm
Sorry, I'll quit the political stuff right away.
15:36 pameyer
public everything does have drawbacks. folks posting logs/configs w\ credentials world-readable, poor place to discuss security related dev/ops issues (w\out waving a red flag)
15:36 pdurbin
pameyer: yes, that's why I wrote, "Security deserves special mention in the context of transparency. Like many projects, Dataverse has a private email address for receiving reports of suspected security vulnerabilities. It would be irresponsible to put customers at risk with completely open discussion of security concerns."
15:36 pdurbin
at https://opensource.com/open-organization/17/11/transparency-dataverse-project
15:37 pameyer
pdurbin: yup - but that approach isn't great for community collaboration on security issues
15:38 poikilotherm
Sure - but my questions are not directly security related...
15:38 pameyer
right
15:38 poikilotherm
And most certainly those of MrK aren't, too.
15:38 pdurbin
pameyer: right but https://github.com/IQSS/dataverse/issues/3215 about ongoing security and communication didn't go anywhere
15:39 pameyer
and btw, I completely agree with @poikilotherm about configuration. current approach feels like it's got a lot of accidental complexity
15:40 pdurbin
pameyer: now you're channeling Rich Hickey and I love it. :)
15:40 pameyer
some portion of that likely due to folks like me trying to do something else, and doing the bare minimum to fit it into some kind of provisioning/deployment system
15:40 poikilotherm
To give this a user story flavor: "As a community member, I would be very glad to reach out to all devs not only via Github issues (which can be quite cluttered), but in a way that enables us to discuss in a direct channel (email sucks at that)."
15:41 pdurbin
poikilotherm: you don't like https://groups.google.com/forum/#!forum/dataverse-dev ? Why?
15:41 poikilotherm
I REALLY appreciate the near realtime discussion with you guys here
15:42 poikilotherm
That gives me the feeling my contributions are valuable and you want me to be part of this
15:42 poikilotherm
It is not as intense as a VC or even sitting next to each other
15:42 poikilotherm
But it is way better than a forum/email
15:43 pameyer
both true :) and lower lag is good
15:43 poikilotherm
Email can be read later and easily be forgotten.
15:44 poikilotherm
The liability of a direct communication is worth some effort
15:44 poikilotherm
And you don't get that from a google group
15:45 poikilotherm
I have been discussing this with other people over here, too. Many don't see the value of a chat system.
15:46 poikilotherm
For me, chat is a good addition in terms of ways to communicat on different levels of liability
15:46 poikilotherm
It is half way between email and a phone call. AFter that comes a VC and then sitting next to each other
15:47 pameyer
chat systems help alot - altough I sometimes get communication channel overload.
15:47 poikilotherm
That's definitly a great danger
15:47 poikilotherm
It helps if you don't need to look into too much systems
15:47 poikilotherm
Having IRC and Gitter and Slack would be overkill
15:48 poikilotherm
To many things requesting attention ;-)
15:55 poikilotherm
Alright guys, gotta go... Cu tomorrow :-)
16:32 pdurbin
Email isn't necessarily forgotten if it's publicly archived. You can link to particular messages or threads.
17:45 donsizemore joined #dataverse
17:48 asdfzxcvqwer joined #dataverse
18:14 pameyer
pdurbin: public archives of some email lists saved me (and probably a bunch of other folks) from having to re-ask the same question
18:15 pdurbin
absolutely
18:15 jri joined #dataverse
18:15 pdurbin
and I hear you on lower lag too
18:15 pdurbin
it's helpful to know if someone read the thing you wrote
18:32 pameyer
yup
18:57 pdurbin
I hear you on the overload too. :)
19:19 pdurbin
" In this tutorial, we'll create a repository webhook that will be responsible for listing out how popular our repository is, based on the number of Issues it receives per day." https://developer.github.com/webhooks/creating/
19:19 pdurbin
interesting way to measure popularity ^^
19:44 donsizemore joined #dataverse
19:45 donsizemore
@pdurbin the campus FauxBucks in Student Stores thinks the same way: "look at how long our line is! we must be the best!"
19:46 pdurbin
heh
20:07 pameyer
"nobody goes there anymore - it's too crowded"
20:10 pdurbin
heh. reminds me of "the food is awful... and such small portions!"
20:27 isullivan joined #dataverse
20:34 Jim__
pdurbin: I saw your "java.lang.IllegalStateException: This web container has not yet been started" comment - I see this during out auto-deploy and I have to restart the service to resolve - is this the same thing?
20:35 pdurbin
Jim__: it's killing me
20:35 Jim__
I thought it was just us/something in our setup...
20:36 pdurbin
It's a thousand times worse in my branch and I don't know why.
20:37 Jim__
I've just switched to restart everytime... a pain.
20:38 pdurbin
Jim__: if you want to go ahead and create an issue for it, please feel free. Steps to reproduce and all that.
20:39 pdurbin
Its so bad for me that I have to stop glassfish and rm -rf generated every time I run the one API test I'm working on.
20:39 pdurbin
It's
20:39 pdurbin
So I've got it worse, I'm sure.
20:39 Jim__
You win :-)
20:39 pdurbin
But it sounds like there's a general issue to fix. And Kevin has seen this too.
20:40 pameyer
might be a dump question - but is there new stuff in generated that wasn't previously there?
20:40 Jim__
(but I created this today: [+-]?[\\{\\[][^\\}\\]]*[\\}\\]]|[+-]?\\\"[^\\\"]*\\\"|([^\\s\"\\[\\{',]+([,]?([^\\s,\\[\\{'\":+-]|[:][\\{\\[][^\\}\\]]*[\\}\\]]|[:]\\\"[^\\\"]*\\\")+)+)+|[^\\s\"',]+ :-)
20:41 Jim__
Cool - if I notice any clues, I'll let you know. So far it isn't obvious to me when/why either.
20:42 pdurbin
looks better than my Perl
20:43 pdurbin
pameyer: I don't know. I was just telling Matthew that I rm -rf generated all the time now but I don't think I used to back in the day. I think it started after we added an EJB timer. Or maybe the app is just getting fatter? I don't know.
20:46 pameyer
I ran into some weirdness where tar files generated on osx would add a `._foo` file when extracted on linux. and if that was in a directory glassfish was trying to load all files as jars, it was unhappy and wouldn't start
20:46 pameyer
so I was thinking filesystem - but I'm not too clear on what's _supposed_ to be in generated, or why
20:49 Jim__
Ah Mac resource files! They're the initial reason the DVUploader got an exclude option...
20:52 pdurbin
I never look at what's in generated. I just blow it away.
20:52 pameyer
@Jim__ yup - oddly enough, they show up as real file on dropbox even on os x
21:11 cwillis joined #dataverse
21:13 cwillis62 joined #dataverse
21:15 craig-willis joined #dataverse
21:18 craig-willis
Hello pdurbin -- saw you on our Slack and thought I'd stop by with a question. We've been talking with DataONE about provenance models (i.e., ProvONE). I'm looking for alternative models and wondering where Dataverse is headed. Is https://github.com/IQSS/dataverse/issues/4346 the place to start?
21:23 pdurbin
craig-willis: hi! I was going to say you should start at http://guides.dataverse.org/en/4.9.4/user/dataset-management.html#data-provenance but I just realized it links to that issue. :)
21:24 craig-willis
Note to self -- search documentation before Github!
21:26 craig-willis
This is perfect. Thank you.
21:32 pdurbin
craig-willis: sure, please keep the question coming.
21:33 pdurbin
Tuesday or after. My kids don't have school tomorrow and I'm heading out. Four day weekend. :)
21:33 craig-willis
Enjoy the long weekend!
21:35 pameyer
pdurbin: have fun. and when you get back, I dn't see any IllegalStateExceptions for b9305c001f5565b2155e145e72ef8552fc3b6b0c from GithubApiIT
21:37 pdurbin
Good. Yeah, I'm convinced it's some bad mojo in my branch. My own fault.
21:38 pdurbin
Take care, all.
21:38 pdurbin left #dataverse
21:39 pameyer
pdurbin: I thought that was your branch
21:40 pameyer
... but that's what logs are for
21:46 craig-willis left #dataverse
22:17 donsizemore joined #dataverse
22:25 pdurbin joined #dataverse
22:26 pdurbin
pameyer: oh, duh. You mentioned GithubApiIT so you're on my branch. You would have to run GithubApiIT twice to reproduce the error.
22:36 pameyer
yup - no joy
22:36 pameyer
tried undeploy/redeploy, still nothing
22:36 pameyer
but don't worry about it til next week
23:03 pdurbin
the plot thickens
23:04 pdurbin
ok, next week. have a good weekend, all. see you tuesday
23:04 pdurbin left #dataverse
23:36 jri joined #dataverse
