Time
S
Nick
Message
08:01 jri joined #dataverse
08:09 juancorr joined #dataverse
08:23 andrewSC joined #dataverse
08:46 MrK joined #dataverse
09:25 icarito[m] joined #dataverse
10:22 poikilotherm joined #dataverse
10:44 pdurbin
poikilotherm: morning. So are you blocked on deciding between Glassfish and Payara?
10:57 poikilotherm
Morning @pdurbin :-)
10:57 poikilotherm
A bit... ;-)
10:57 poikilotherm
As long as no one cries out loud, I would go for Payara for now, let EE4J settle down and switch to them later on.
10:58 poikilotherm
As I said yesterday: I assume that switch is much less of a pain than the hole step forward
11:01 pdurbin
You mean that moving from 4 to 5 will probably be more difficult than moving from Glassfish to Payara?
11:01 poikilotherm
Yepp
11:05 pdurbin
Ok. And it sounds like you'd rather move to Payara 5 than Payara 4 (and then Payara 5 later).
11:10 poikilotherm
Yes - that's because of EOL for Payara / Glassfish 4
11:11 poikilotherm
Doesn't make sense to bet on that horse... ;-)
11:14 pdurbin
Sure. I'm just thinking of smaller steps. And Oracle Glassfish 4 is already EOL, I'd say.
11:18 poikilotherm
Yes it is ;-)
11:18 poikilotherm
Oh by the way - there is no EE4J Glassfish 5 docker image available ATM
11:19 poikilotherm
So relying on Payara seems to be the only option at least if we don't want to wait another 3-4 months
11:19 pdurbin
That December date we saw.
11:28 poikilotherm
I will give this a quick first shot using the available Payara 5 docker container. This seems to be a more lightweight approach than what I saw in dataverse-docker. I will also try to use the fabric8io docker maven plugin to build the images and have the ability to use the watch function for code rebuilds, etc.
11:29 poikilotherm
Just getting my hands dirty to have a proper image plus surrounding services (database, etc) via a simple maven command
11:29 poikilotherm
IMHO this is a very clean approach for local development and future integration tests based on Docker
11:30 poikilotherm
If you guys dislike it, at least I learned something from it ;-)
11:31 pdurbin
Starting with a Docker image of the app server sounds way better (and smaller) than what we're doing now. Thanks!
11:32 poikilotherm
Most certainly this can be reused in dataverse-docker, as this image could be a "common ground" base image for application itself
11:33 pdurbin
Sure.
11:33 pdurbin
For local development, would I be able to push small changes into the running Docker image?
11:33 poikilotherm
https://dmp.fabric8.io/#docker:watch
11:33 poikilotherm
;-)
11:34 pdurbin
nice, sounds great
11:36 pdurbin
This reminds me of https://blog.openshift.com/fast-iterative-java-development-on-openshift-kubernetes-using-rsync/ which I link to from the containers page in our dev guide.
11:37 poikilotherm
Yepp. But this needs a full blown openshift / kubernetes locally. IMHO this is overkill. If this works by just installing docker, this much more lightweight.
11:37 poikilotherm
+is
11:38 pdurbin
I like that article because it was written by danmcp: http://irclog.iq.harvard.edu/search.pl?channel=dataverse&nick=danmcp
12:08 Jim__ joined #dataverse
12:16 Jim__
Good Morning... pdurbin re 5251 - do you want the perhaps-not-best-practice stuff from the google doc in the guides? I was assuming that things like this might still be best if 'unofficial'/caveat emptor...
12:43 pdurbin_m joined #dataverse
12:43 pdurbin_m
Jim__: hi! typing with thumbs
12:44 pdurbin_m
I don't want to just copy and paste all of it into the guides.
12:44 pdurbin_m
I want to userstand the needs.
12:44 pdurbin_m
the deficiencies
12:45 pdurbin_m
how we can make the metadata system better
12:45 pdurbin_m
I'm fine with adding some work arounds to the guides.
12:46 pdurbin_m
but I'd also like to see new GitHub issues having to do with how improvements can be made
12:46 pdurbin_m
I hope all that makes sense.
12:52 Jim__
~yeah. I'm not arguing for putting work-arounds in the guides, just wondering if the google doc as a whole should be deprecated, versus perhaps noting that much of the info has been moved to official guides...
13:13 donsizemore joined #dataverse
13:19 poikilotherm
pdurbin: is it correct that the setup-all.sh script is optional at least for a development instance?
13:34 poikilotherm
Ok sry for the noise - seems that setup-all.sh is used anyway.
13:41 pdurbin
poikilotherm: no problem. You're right, not optional.
13:42 poikilotherm
I'm not sure if I like that the container has the configuration baked in at built time.
13:43 pdurbin
Jim__: I'm not opposed to the community having a place to share workarounds and perhaps-not-best practices. I'm not sure that Google doc is the best place now that the Dataverse project is making an attempt to officially document our crazy metadata system.
13:43 pdurbin
poikilotherm: what configuration?
13:45 poikilotherm
Ah sry - that won#t work anyway - the setup-all.sh needs to run at boot time, because otherwise there is no database and index
13:46 poikilotherm
And all those pretty curl's don't work
13:47 poikilotherm
So actually it is alright to leave that out at build time, but there needs to be some script piping that stuff inside
13:47 poikilotherm
At container start I will need to have proper configuration of glassfish within the entrypoint script, so dataverse get's available in the first place
13:48 pdurbin
ok
13:48 pdurbin
poikilotherm: are you calling the "install" Perl script?
13:48 poikilotherm
Maybe I'll make a drawing of that :-D
13:48 poikilotherm
Not yet, but was not planning to do so...
13:49 pdurbin
Ok. donsizemore doesn't call "install" from https://github.com/IQSS/dataverse-ansible
13:49 pdurbin
But the Docker/Kubernetes/OpenShift work by Red Hat interns involved hacking on "install".
13:50 poikilotherm
*brrr* IMHO the install script is quite huge and difficult to maintain
13:50 pdurbin
Yeah.
13:50 poikilotherm
I thought about splitting it up in different smaller chunks at places...
13:50 pdurbin
Speaking of diagrams, one of the Red Hat interns (Charles) made this diagram: https://user-images.githubusercontent.com/25762021/41736416-652879ee-755a-11e8-8969-b88c4c165809.png
13:51 pdurbin
That's from https://github.com/IQSS/dataverse/issues/4764
13:51 pdurbin
He added an "init container". Do you know what that is? I didn't.
13:51 poikilotherm
Sure. https://kubernetes.io/docs/concepts/workloads/pods/init-containers/
13:52 pdurbin
yeah, that thing
13:59 poikilotherm
But that is not usable with a plain docker setup
13:59 poikilotherm
Or at least it makes stuff complicated ;-)
14:00 pdurbin
poikilotherm: you know about docker-aio, right? That's our plain Docker setup.
14:00 poikilotherm
Yepp :-)
14:00 pdurbin
cool
14:01 poikilotherm
I'm aware of it but want to make a more flexible approach. docker aio is not really reusable within a kubernetes / ... scenario
14:01 pdurbin
pameyer contributed docker-aio after he saw all the openshift activity. He was telling me that you don't need all of that, that you can spin up Dataverse in a single container with just Docker. Then he made a pull request. :)
14:01 poikilotherm
But of course I take inspiration from that...
14:02 pdurbin
Right, and Slava has taken inspiration from it for dataverse-docker.
14:02 pdurbin
See? https://github.com/IQSS/dataverse-docker/pull/9/files :)
14:03 poikilotherm
;-)
14:20 poikilotherm
I just had a look at the openshift stuff they came up with. Do you know why they used the AIO approach, too?
14:21 poikilotherm
(Just to be sure: they crafted the dataverse-glassfish config, right?)
14:21 pdurbin
They is me and I didn't. I'm confused. I was heavily inspired by https://github.com/nds-org/ndslabs-dataverse which is older than docker-aio.
14:22 pdurbin
poikilotherm: for the timeline of what happened when, please see https://github.com/IQSS/dataverse/issues/4665#issuecomment-391926503
14:24 poikilotherm
Thx! :-)
14:26 pdurbin
The timeline helps? :)
14:34 pameyer joined #dataverse
14:34 poikilotherm
Ok, read through it, thx.
14:37 pdurbin
Sure. I can explain what happened next if you're curious. :)
14:37 poikilotherm
So this actually IS the stuff that a lot of people already worked on :-)
14:37 pdurbin
Yes, Craig Willis from NDS was the first to dockerize Dataverse.
14:37 pameyer
as one of the people, it's felt more like fumbling around in the dark
14:38 pdurbin
yes, fumbling :)
14:38 pdurbin
lots of prayer, crying
14:38 pdurbin
maybe that's just me
14:38 pameyer
nowhere near ready for a PR yet, but I was able to get a straight docker setup where the installer/setup scripts don't need to run on container start
14:38 pdurbin
gnashing of teeth
14:38 donsizemore
@pdurbin yeah, the ansible role did a bunch of what install did, so it calls the helper scripts instead
14:39 poikilotherm
Basically all approaches made till today always used a CentOS 7 base image and build on that. What I would like to try is using the payara5 container to build a rather tiny layer of war and deps on it.
14:39 pameyer
one possible drawback of using a straight app server container is that dv needs some fiddling inside it prior to install
14:39 poikilotherm
And yes @pameyer I am very open for that approach not to run install scripts on startup (with an exception of the glassfish config)
14:39 pdurbin
pameyer: is that true? You may have been using CentOS 6 originally. But this is a distinction without a difference. :)
14:40 pameyer
yeah, switch from cent6 to cent7 was pretty minimal
14:40 poikilotherm
As far as I can see, the fiddling is only about the payara 4 stuff
14:40 poikilotherm
(glassfish 4)
14:40 pameyer
haven't been brave enough yet to try alpine based
14:40 pameyer
I vaguely remember donsizemore fiddling w\ payara a while back - but could be remembering wrong
14:40 poikilotherm
I would like to try to get to Payara 5 and a smaller docker approach in one wipe.
14:41 poikilotherm
You're right, pdurbin did so, too.
14:42 pameyer
I've personally had relatively bad luck trying to make big changes in one shot - but I get the sense you've got more java experience than me
14:43 donsizemore
i was trying a simple drop-in replacement: payara 5 for glassfish 4, and got decently far but there were dependency problems
14:44 * poikilotherm
ready to try my best to sort that out
14:44 donsizemore
@poikilotherm my contributions are well-intentioned but best-effort
14:45 poikilotherm
Mine too :-D
14:46 pameyer
that's a good phrase :)
14:58 pdurbin
heh
14:59 pdurbin
poikilotherm: yes! A thin layer of just the war and deps on top of a glassfish or payara layer please!
15:28 poikilotherm
I'm afraid by using the app server container image approach I can't use the hollow concept in an easy way. Thus the resulting war will always be around 200MB because of the HUGE dependency base.
15:28 poikilotherm
Thus the docker image will always be around that size :-/
15:29 poikilotherm
If we want that smaller, we need to go with payara micro and their approach of hollow jar + application war
15:32 poikilotherm
Of course that approach is more complex and would again spread the size of the step. So maybe better step back and just leave the WAR as is, as this can be used for the "classic" deployments, too.
15:32 poikilotherm
(And most certainly that would be something you guys still need and want, right?)
15:33 poikilotherm
It's not perfectly "thin", but seems to be a good start, right?
15:35 poikilotherm
pdurbin - we have two days of holidays here in Germany from as of tomorrow, so I will continue on this next week.
15:39 poikilotherm
Oh and guys - I could of course create a second maven profile generating a thin WAR and copy the deps via Dockerfile inside the container. That is kind of creating a "hollow" approach
15:39 poikilotherm
It is not perfectly portable, but might be a third way in between
15:48 poikilotherm
Any comments very much appreciated :-) Will read up the logs later, have to go now to pick up kids and use some pumpkins for scaring the crap out of 'em... ;-)
15:48 poikilotherm
Cu guys
16:33 donsizemore joined #dataverse
16:54 Jim__
pdurbin - so what's the action w.r.t. 5251/google doc? I'd like to keep the sense of some of the notes being plausibly deniable but that could mean removing the stuff you've put in the guides and focusing it on the unofficial
16:54 Jim__
or we could create another guide section, or ...
17:09 nanoz joined #dataverse
17:13 pdurbin_m2 joined #dataverse
17:14 jri joined #dataverse
17:14 pdurbin_m2
nanoz: welcome! You must have seen my reply at http://irclog.greptilian.com/friendlyjava/2018-10-31 :)
17:15 pdurbin_m2
Jim__: sorry, typing with thumbs again. Code no work. Need walk.
17:15 nanoz
yes
17:16 Jim__
:-) been there done that...
17:18 Jim__
and no rush - just wanted to make sure nothing gets dropped.
17:22 pdurbin_m2
nanoz: do you need an issue to work on?
17:29 nanoz
i need someone time to explore
17:30 pdurbin_m2
That's fine. We have a lot of issues. :)
17:31 pdurbin_m2
I have a "one line fix" issue in mind.
17:39 pdurbin
Jim__: I'm not sure I'm following what you're saying but we have a Halloween party in 20 minutes and then meetings the rest of the afternoon.
17:40 nanoz
what is the label?
17:41 nanoz
for `one line fix`
17:41 pdurbin
nanoz: no label but here's the issue: https://github.com/IQSS/dataverse/issues/5246
17:57 pdurbin
Jim__: and the walk helped a lot, by the way. :)
18:06 pdurbin
Jim__: since you were asking about release notes, please check this out: https://github.com/IQSS/dataverse/commit/5c2ed68
18:40 Jim__
OK - Happy Halloween! I added comments on the commit and we can chat about the other stuff some other time...
18:51 pdurbin
Jim__: thanks! I just commented on your comments. :)
22:01 pameyer
pdurbin: fun fact, from what I can see https://schema.org/Dataset doesn't have *any* required fields - which might be why it was impossible to get the validators to fail
22:02 pdurbin
I can get validators to fail. If you mispell "description" or whatever.
22:03 pameyer
that's better than I expected
22:07 pdurbin
pameyer: "you will find that these tools (and the underlying spec) can be extremely accommodating": https://github.com/IQSS/dataverse/blob/5c2ed685f66629989fc93c3bbf68dd22453fbaa1/src/main/java/edu/harvard/iq/dataverse/export/SchemaDotOrgExporter.java#L32
22:08 pameyer
that's a very diplomatic way of phrasing it
22:08 pdurbin
:)
22:08 pdurbin
"come crash on my couch! it's all good"
22:10 pdurbin
"Strings can become objects or arrays, for example, and Honey Badger don't care."
22:12 pameyer
now I'm picturing a startup's error page having "Honey Badger doesn't care" :)
22:12 pdurbin
heh, he don't!
