IQSS logo

IRC log for #dataverse, 2018-11-08

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

All times shown according to UTC.

Time S Nick Message
00:13 jri joined #dataverse
01:24 nanoz joined #dataverse
04:14 isullivan joined #dataverse
04:29 jri joined #dataverse
04:37 isullivan1 joined #dataverse
04:37 isullivan joined #dataverse
04:44 isullivan joined #dataverse
04:51 isullivan joined #dataverse
04:53 isullivan joined #dataverse
08:12 jri joined #dataverse
09:58 poikilotherm joined #dataverse
11:46 poikilotherm Morning pdurbin :-)
11:56 pdurbin poikilotherm: morning. I was thinking more about your database config idea. I wonder if it would help with https://github.com/IQSS/dataverse/issues/4765
12:15 poikilotherm pdurbin if you see any chance, I would really like to go for https://docs.payara.fish/documentation/microprofile/config.html
12:15 poikilotherm This can be used with both Payara Full and Micro
12:16 poikilotherm And it will solve more than just the database config
12:16 poikilotherm Yet it keeps the "old way" possible, so people preferring the install script and/or asadmin can just do so.
12:17 poikilotherm For us having the plan to go for Docker (and others, too) that would make a HUGE step forward in terms of possibilities for deployment
12:19 poikilotherm I'm currently writing an "epic" as an GH issue that explains my ideas and plans
12:19 poikilotherm After that I will write up an issue about the config stuff
12:20 poikilotherm And yes indeed - it might help with GH-4765 as a more dynamic configuration can be done at deploy time rather than build time
12:20 pdurbin Sounds perfect. I'm looking at https://github.com/eclipse/microprofile-config#implementations and it looks like Glassfish isn't on the list. Payara is. Any idea if Eclipse Glassfish will support "config"?
12:20 poikilotherm Dunno...
12:21 poikilotherm But in theory when you stick to the Config API spec, you should even be able to switch to Thorntail/Wildfly and other app servers
12:21 pdurbin I guess we could ask at https://gitter.im/eclipse/microprofile-config
12:22 poikilotherm I get the impression that all this stuff with changing to Payara 5 and more will need a Dataverse 5.0.0 tag...
12:23 poikilotherm I'm afraid what this means in terms of getting this released in the near future...
12:23 poikilotherm +of
12:24 pdurbin I just asked: https://gitter.im/eclipse/microprofile-config?at=5be42ade9600d10e6479a093
12:24 donsizemore joined #dataverse
12:25 poikilotherm Cool :-)
12:25 pdurbin The pattern has been that DVN 3 ran on Glassfish 3, Dataverse 4 runs on Glassfish 4. But that wasn't on purpose, I don't think.
12:26 poikilotherm Well anyone using somethng like 4.10 / 4.11 / ... might not expect that you need to change your application server...
12:26 pdurbin Yeah, I guess.
12:27 poikilotherm Of course this is possible, but it would need VERY good and tremendous amounts of communication to the users
12:27 barryr joined #dataverse
12:27 pdurbin I dunno, we sort of try to follow semver.
12:27 poikilotherm Aye
12:27 poikilotherm IMHO it would be better to make such a big change in a new major...
12:28 poikilotherm But I dunno if this means "oh we have some other tons of stuff in the pipeline that needs to be done before"
12:28 pdurbin We do have a ton of stuff to do. Have you seen our roadmap?
12:28 pdurbin Or the Design Team column on our board?
12:29 poikilotherm Sure! And that's why I am afraid this might stop us from getting this forward
12:29 pdurbin Yeah.
12:30 pdurbin Do you want to work on anything on our roadmap? :)
12:30 poikilotherm ...
12:30 poikilotherm I could
12:30 poikilotherm But we need to get our service in place :-D
12:31 pdurbin Sure.
12:31 poikilotherm And investing time to have it dockerized as is now could mean a waste of effort
12:31 pdurbin We'd love to get you added to our map. We just added a new installation yesterday.
12:31 pdurbin Dunno if you've seen the map.
12:31 poikilotherm Not yet
12:31 poikilotherm Errr... The addittion
12:32 poikilotherm The map itself is already on my mind... ;-)
12:32 pdurbin http://guides.dataverse.org/en/4.9.4/installation/config.html#putting-your-dataverse-installation-on-the-map-at-dataverse-org
12:33 poikilotherm We don't have an installation yet... ;-)
12:34 poikilotherm What are those grey dots? They don't look familiar
12:37 pdurbin I forget what the grey dots are. They shouldn't be there. I think they're specific to Harvard Dataverse or something. Individual dataverses within that installation, maybe.
12:37 pdurbin You can get the installations as JSON: https://services.dataverse.harvard.edu/miniverse/map/installations-json
12:38 poikilotherm Thx
12:39 poikilotherm GH-4248 (Glassfish 5 Upgrade) and GH-4260 (Java EE 8 Upgrade) are in the milestone for 5.0
12:39 pdurbin poikilotherm: here you go. More on the grey dots: https://groups.google.com/d/msg/dataverse-community/eQlSLFgzQXI/t6FANVynBgAJ
12:39 poikilotherm Maybe we can talk about reducing the 5.0 milestone to the upgrading, leaving the data tag stuff for a 5.1 and thus moving the stuff with 4.x on your roadmap to a 5.x?
12:40 poikilotherm 4.10 seems to be almost ready...
12:41 pdurbin Almost, I guess, but we need to fix https://github.com/IQSS/dataverse/issues/5260
12:41 poikilotherm Oh this is not in 4.10 milestone!
12:43 pdurbin Yeah, I find the way we use milestones confusing. There are only 2 issues in the 4.10 milestone.
12:43 pdurbin But we've already merged 58 pull requests for the next release (probably 4.10): https://github.com/IQSS/dataverse/pulls?utf8=%E2%9C%93&q=is%3Apr+is%3Aclosed+is%3Amerged+no%3Amilestone
12:44 * poikilotherm is puzzled
12:45 pdurbin heh
12:45 pdurbin I guess every project has its own culture.
12:46 pdurbin Recently I added milestones to all the old pull requests we had already shipped in a release so I could do that query and get an accurate count.
12:46 pdurbin Accurate count of what will be in the next release.
12:47 pdurbin So that I can go back and add the milestone to all issues for that release.
12:47 pdurbin So that we'll have more than two issues tagged with 4.10 or whatever. :)
12:53 poikilotherm https://github.com/IQSS/dataverse/issues/5292
12:54 poikilotherm This could be a good candidate for addition to a milestone... ;-)
12:57 pdurbin Great write up. My first thought is that *everyone* still uses the "old" WAR approach right now. The 35 production installations, trial installations, devs, etc.
12:57 poikilotherm Yes
12:57 poikilotherm And in no way I would break that by bad design
12:58 poikilotherm That's a no go
12:58 pdurbin Good.
12:58 pdurbin It reminds me a bit of https://github.com/IQSS/dataverse/issues/4040
12:58 poikilotherm Sure!
12:59 poikilotherm The OpenShift based things are using the install script, too!
12:59 poikilotherm That's not microservices IMHO
12:59 poikilotherm Good catch - I will add a reference in my epic
12:59 pdurbin In that issue we explain that Red Hat may be interested in running Dataverse internally (or externally) but before they do they require that Dataverse run in the way they want, which is on OpenShift. For you, it's Docker/Kubernetes.
13:00 pdurbin Which is fine but a number of people are already having success in production with the old ugly way. :)
13:00 poikilotherm Of course it works
13:00 poikilotherm But it "just works"
13:00 pdurbin It barely works. :)
13:01 poikilotherm As far as I can see, they still made an AIO approach
13:01 poikilotherm At least from what I can see from the conf/docker/dataverse-glassfish tings
13:01 pdurbin I think of it as project/product/team maturity. Folks like you and Red Hat are trying to help us mature. SBGrid also. It's very healthy for us.
13:02 poikilotherm They already ripped out a lot of stuff, but still they start on a base image and install java, glassfish etc
13:02 pdurbin Mature not just at a "you should add this feature" level but at an infrastructure level. Plumbing.
13:02 poikilotherm I hope so :-D
13:02 poikilotherm I am glad that you see value in that :-)
13:03 pdurbin I was a sysadmin for 15 years.
13:03 poikilotherm A lot of people out there are just on the "feature feature feature" side of life
13:03 poikilotherm ;-)
13:03 poikilotherm That's why I like to call myself a DevOp
13:03 pdurbin I try to stay far away from production systems. Don't give me the root password. But I symphathize with people who deal with production.
13:03 poikilotherm Aye!
13:04 pdurbin The scars will be with me forever. ;)
13:04 poikilotherm ;-)
13:06 pdurbin This is the feature I'm working on right now: https://github.com/IQSS/dataverse/issues/2739
13:06 pdurbin If anyone here has any opinions on how it should be implemented.
13:07 poikilotherm Hell yeah! That's a great move :-)
13:08 poikilotherm We are really looking forward to this
13:09 poikilotherm Just my 2 cents I can spend: after a lot of talking to other RSEs, it seems valuable to store the code also inside dataverse and not just create a link to the git repo
13:10 poikilotherm Many people out there would really like to have an archive that is foreseen to live at least 10 years, while the repository might be killed earlier
13:45 MrK joined #dataverse
13:46 pdurbin poikilotherm: yes, the idea is to store the code itself (or whatever is in the git repo) in Dataverse as a zip (prototype: https://dev1.dataverse.org/file.xhtml?persistentId=doi:10.5072/FK2/FS7M3O/EBNKNB ) rather than only linking to the git repo. We're oriented toward preservation.
14:02 poikilotherm joined #dataverse
14:02 jri joined #dataverse
14:07 pdurbin I just posted a screenshot at https://github.com/IQSS/dataverse/issues/5209#issuecomment-437003277
14:13 isullivan joined #dataverse
14:18 poikilotherm pdurbin: my write up for the configuration stuff: https://github.com/IQSS/dataverse/issues/5293
14:20 poikilotherm pdurbin: can you keep in mind that webhooks from gitlab should also work? they differ from the github ones... doesn't need to be implemented right away, but a concept of extendability from the beginning would be awesome
14:22 pdurbin poikilotherm: ah, I didn't know if GitLab had webhooks or not.
14:23 pdurbin poikilotherm: comments are welcome on https://github.com/IQSS/dataverse/issues/2739
14:24 poikilotherm Will do so :-)
14:24 pdurbin thanks
14:30 poikilotherm Done. https://github.com/IQSS/dataverse/issues/2739#issuecomment-437012724
14:33 pdurbin poikilotherm: thanks! I wrote a bit about CodeMeta at https://github.com/IQSS/dataverse/issues/3736
14:34 poikilotherm Glad you already have that on the radar
14:34 pdurbin me too
14:35 poikilotherm ACtually CFF helps a lot with this... ;-) https://citation-file-format.github.io/1.0.0/specifications/
14:36 pdurbin CFF is new to me. Thanks.
14:37 poikilotherm Should I also leave a comment about CFF at GH-3736?
14:38 donsizemore joined #dataverse
14:43 pdurbin It's up to you. I ended up closing that issue.
14:58 MrK pdurbin: Hi, my friend responded to the yesterdays issue https://github.com/IQSS/dataverse/issues/5280 , and i just wanted to add that in the future, we will be able to merge those features into your instance if you would wish so.
15:03 poikilotherm pdurbin: I got the response from Dataverse Support that I cannot get an account at your Slack. There are efforts to build up a Dataverse Consortium based Slack in progress.
15:04 poikilotherm They suggested contacting them via email...
15:04 poikilotherm I am very unsure if this is a good approach. Maybe for a first contact, but not for discussions.
15:05 poikilotherm Any ideas? Stick with GH issues? Ask them to come to IRC?
15:05 poikilotherm Mos certainly this is important to MrK, too
15:23 pameyer joined #dataverse
15:23 pdurbin poikilotherm: suggested contacting whom via email?
15:23 pdurbin I'll go look at your ticket. One sec.
15:24 pdurbin Oh, Gustavo and Matthew.
15:26 pdurbin You could ask them to join us here. :)
15:28 poikilotherm Maybe you could trick them to use Gitter... :-d
15:31 pameyer "Gitter does not provide a regular password authentication. Instead, it asks for maximalist GitHub account permissions. " :(
15:31 pdurbin Maybe. I think Slack is terrible for open source projects. Everything happening behind closed doors. No transparency.
15:32 poikilotherm Yeah, as I told pdurbin yesterday: Gitter is not one of my favorites...
15:32 poikilotherm But most certainly they like the Slack-XP and IRC is anything but a Slack-like experience
15:33 pdurbin What I like about Gitter has nothing to do with technology. It has to do with norms and culture and what's healthy for communicating within open source projects. Transparency. Public by default. Sunlight.
15:34 poikilotherm Ah sunlight...
15:34 poikilotherm I got a shirt with something about sunlight on it...
15:34 poikilotherm I think it reads like "Developer. Do not expose to direct sunlight."
15:34 pdurbin I'm talking about stuff like https://en.wikipedia.org/wiki/Sunlight_Foundation
15:35 pdurbin "The organization was founded in April 2006 with the goal of increasing transparency and accountability in the United States Congress, the executive branch, and in state and local governments."
15:35 poikilotherm Oh I wasn't awre of that
15:35 poikilotherm And it hasn't been closed by the recent president?
15:35 poikilotherm Sorry, I'll quit the political stuff right away.
15:36 pameyer public everything does have drawbacks. folks posting logs/configs w\ credentials world-readable, poor place to discuss security related dev/ops issues (w\out waving a red flag)
15:36 pdurbin pameyer: yes, that's why I wrote, "Security deserves special mention in the context of transparency. Like many projects, Dataverse has a private email address for receiving reports of suspected security vulnerabilities. It would be irresponsible to put customers at risk with completely open discussion of security concerns."
15:36 pdurbin at https://opensource.com/open-organization/17/11/transparency-dataverse-project
15:37 pameyer pdurbin: yup - but that approach isn't great for community collaboration on security issues
15:38 poikilotherm Sure - but my questions are not directly security related...
15:38 pameyer right
15:38 poikilotherm And most certainly those of MrK aren't, too.
15:38 pdurbin pameyer: right but https://github.com/IQSS/dataverse/issues/3215 about ongoing security and communication didn't go anywhere
15:39 pameyer and btw, I completely agree with @poikilotherm about configuration.  current approach feels like it's got a lot of accidental complexity
15:40 pdurbin pameyer: now you're channeling Rich Hickey and I love it. :)
15:40 pameyer some portion of that likely due to folks like me trying to do something else, and doing the bare minimum to fit it into some kind of provisioning/deployment system
15:40 poikilotherm To give this a user story flavor: "As a community member, I would be very glad to reach out to all devs not only via Github issues (which can be quite cluttered), but in a way that enables us to discuss in a direct channel (email sucks at that)."
15:41 pdurbin poikilotherm: you don't like https://groups.google.com/forum/#!forum/dataverse-dev ? Why?
15:41 poikilotherm I REALLY appreciate the near realtime discussion with you guys here
15:42 poikilotherm That gives me the feeling my contributions are valuable and you want me to be part of this
15:42 poikilotherm It is not as intense as a VC or even sitting next to each other
15:42 poikilotherm But it is way better than a forum/email
15:43 pameyer both true :) and lower lag is good
15:43 poikilotherm Email can be read later and easily be forgotten.
15:44 poikilotherm The liability of a direct communication is worth some effort
15:44 poikilotherm And you don't get that from a google group
15:45 poikilotherm I have been discussing this with other people over here, too. Many don't see the value of a chat system.
15:46 poikilotherm For me, chat is a good addition in terms of ways to communicat on different levels of liability
15:46 poikilotherm It is half way between email and a phone call. AFter that comes a VC and then sitting next to each other
15:47 pameyer chat systems help alot - altough I sometimes get communication channel overload.
15:47 poikilotherm That's definitly a great danger
15:47 poikilotherm It helps if you don't need to look into too much systems
15:47 poikilotherm Having IRC and Gitter and Slack would be overkill
15:48 poikilotherm To many things requesting attention ;-)
15:55 poikilotherm Alright guys, gotta go... Cu tomorrow :-)
16:32 pdurbin Email isn't necessarily forgotten if it's publicly archived. You can link to particular messages or threads.
17:45 donsizemore joined #dataverse
17:48 asdfzxcvqwer joined #dataverse
18:14 pameyer pdurbin: public archives of some email lists saved me (and probably a bunch of other folks) from having to re-ask the same question
18:15 pdurbin absolutely
18:15 jri joined #dataverse
18:15 pdurbin and I hear you on lower lag too
18:15 pdurbin it's helpful to know if someone read the thing you wrote
18:32 pameyer yup
18:57 pdurbin I hear you on the overload too. :)
19:19 pdurbin " In this tutorial, we'll create a repository webhook that will be responsible for listing out how popular our repository is, based on the number of Issues it receives per day." https://developer.github.com/webhooks/creating/
19:19 pdurbin interesting way to measure popularity ^^
19:44 donsizemore joined #dataverse
19:45 donsizemore @pdurbin the campus FauxBucks in Student Stores thinks the same way: "look at how long our line is! we must be the best!"
19:46 pdurbin heh
20:07 pameyer "nobody goes there anymore - it's too crowded"
20:10 pdurbin heh. reminds me of "the food is awful... and such small portions!"
20:27 isullivan joined #dataverse
20:34 Jim__ pdurbin: I saw your "java.lang.IllegalStateException: This web container has not yet been started" comment - I see this during out auto-deploy and I have to restart the service to resolve - is this the same thing?
20:35 pdurbin Jim__: it's killing me
20:35 Jim__ I thought it was just us/something in our setup...
20:36 pdurbin It's a thousand times worse in my branch and I don't know why.
20:37 Jim__ I've just switched to restart everytime... a pain.
20:38 pdurbin Jim__: if you want to go ahead and create an issue for it, please feel free. Steps to reproduce and all that.
20:39 pdurbin Its so bad for me that I have to stop glassfish and rm -rf generated every time I run the one API test I'm working on.
20:39 pdurbin It's
20:39 pdurbin So I've got it worse, I'm sure.
20:39 Jim__ You win :-)
20:39 pdurbin But it sounds like there's a general issue to fix. And Kevin has seen this too.
20:40 pameyer might be a dump question - but is there new stuff in generated that wasn't previously there?
20:40 Jim__ (but I created this today: [+-]?[\\{\\[][^\\}\\]]*[\\}\\]]|[+-]?\\\"[​^\\\"]*\\\"|([^\\s\"\\[\\{',]+([,]?([^\\s,​\\[\\{'\":+-]|[:][\\{\\[][^\\}\\]]*[\\}\\]​]|[:]\\\"[^\\\"]*\\\")+)+)+|[^\\s\"',]+          :-)
20:41 Jim__ Cool - if I notice any clues, I'll let you know. So far it isn't obvious to me when/why either.
20:42 pdurbin looks better than my Perl
20:43 pdurbin pameyer: I don't know. I was just telling Matthew that I rm -rf generated all the time now but I don't think I used to back in the day. I think it started after we added an EJB timer. Or maybe the app is just getting fatter? I don't know.
20:46 pameyer I ran into some weirdness where tar files generated on osx would add a `._foo` file when extracted on linux.  and if that was in a directory glassfish was trying to load all files as jars, it was unhappy and wouldn't start
20:46 pameyer so I was thinking filesystem - but I'm not too clear on what's _supposed_ to be in generated, or why
20:49 Jim__ Ah Mac resource files! They're the initial reason the DVUploader got an exclude option...
20:52 pdurbin I never look at what's in generated. I just blow it away.
20:52 pameyer @Jim__ yup - oddly enough, they show up as real file on dropbox even on os x
21:11 cwillis joined #dataverse
21:13 cwillis62 joined #dataverse
21:15 craig-willis joined #dataverse
21:18 craig-willis Hello pdurbin -- saw you on our Slack and thought I'd stop by with a question.  We've been talking with DataONE about provenance models (i.e., ProvONE). I'm looking for alternative models and wondering where Dataverse is headed. Is https://github.com/IQSS/dataverse/issues/4346 the place to start?
21:23 pdurbin craig-willis: hi! I was going to say you should start at http://guides.dataverse.org/en/4.9.4/user/dataset-management.html#data-provenance but I just realized it links to that issue. :)
21:24 craig-willis Note to self -- search documentation before Github!
21:26 craig-willis This is perfect. Thank you.
21:32 pdurbin craig-willis: sure, please keep the question coming.
21:33 pdurbin Tuesday or after. My kids don't have school tomorrow and I'm heading out. Four day weekend. :)
21:33 craig-willis Enjoy the long weekend!
21:35 pameyer pdurbin: have fun.  and when you get back, I dn't see any IllegalStateExceptions for b9305c001f5565b2155e145e72ef8552fc3b6b0c from GithubApiIT
21:37 pdurbin Good. Yeah, I'm convinced it's some bad mojo in my branch. My own fault.
21:38 pdurbin Take care, all.
21:38 pdurbin left #dataverse
21:39 pameyer pdurbin: I thought that was your branch
21:40 pameyer ... but that's what logs are for
21:46 craig-willis left #dataverse
22:17 donsizemore joined #dataverse
22:25 pdurbin joined #dataverse
22:26 pdurbin pameyer: oh, duh. You mentioned GithubApiIT so you're on my branch. You would have to run GithubApiIT twice to reproduce the error.
22:36 pameyer yup - no joy
22:36 pameyer tried undeploy/redeploy, still nothing
22:36 pameyer but don't worry about it til next week
23:03 pdurbin the plot thickens
23:04 pdurbin ok, next week. have a good weekend, all. see you tuesday
23:04 pdurbin left #dataverse
23:36 jri joined #dataverse

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.