Time
S
Nick
Message
02:27 jri joined #dataverse
03:23 sivoais joined #dataverse
03:48 bhanu joined #dataverse
03:48 bhanu
Hello
04:44 pdurbin
bhanu: hi. Welcome.
05:27 jri joined #dataverse
08:25 jri joined #dataverse
08:55 MrK joined #dataverse
11:42 poikilotherm joined #dataverse
12:06 pdurbin
poikilotherm: mornin. When I said "docker-kubernetes" yesterday, I meant "dataverse-kubernetes".
12:06 poikilotherm
Morning :-)
12:07 poikilotherm
I got you :-)
12:07 pdurbin
good
12:09 poikilotherm
I just read up the logs...
12:10 poikilotherm
pameyer seemed to have missed http://irclog.iq.harvard.edu/dataverse/2018-11-28#i_80119
12:10 poikilotherm
;-)
12:12 pdurbin
the "-Pcontainer" part?
12:13 poikilotherm
Aye
12:13 pdurbin
poikilotherm: are you going to Open Science Days 2019? It's Feb 5-6 in Berlin: http://osd.mpdl.mpg.de
12:13 poikilotherm
And the package part... docker:build cannot get that triggered
12:13 poikilotherm
Oh sounds promising
12:15 poikilotherm
Are you considering to join?
12:16 pdurbin
I'd go if someone sends me. :)
12:16 pdurbin
I've never been to Germany.
12:17 pdurbin
There's also Open Repositories 2019 in June in Hamburg: https://or2019.blogs.uni-hamburg.de
12:17 poikilotherm
:-D
12:17 pdurbin
Just before the Dataverse Community Meeting.
12:19 poikilotherm
Thx for sharing
12:21 pdurbin
sure
12:39 pdurbin
poikilotherm: oh, I don't think we should delete the Dockerfile in the root of the repo. I put it there as a signal that there is Docker stuff somewhere in the repo. When I glance at a repo, I'm happy to see a README, LICENCE, CONTRIBUTING, Dockerfile, Vagrantfile, etc.
12:40 poikilotherm
Actually I wanted to replace it with a docker-compose file... ;-)
12:40 poikilotherm
For a demo purpose this is more usable than an empty Dockerfile
12:42 pdurbin
No objection to that. I just don't want to lose my signal.
12:43 poikilotherm
Is a docker-compose.yml a sufficient signal?
12:45 pdurbin
absolutely
12:45 poikilotherm
:-)
12:49 poikilotherm
The setup-all.sh is giving me a hard time and headaches
12:50 pdurbin
how so?
12:51 poikilotherm
Well you definitly don't want that script in a container...
12:51 poikilotherm
It does no checks and shoots at Solr
12:51 pdurbin
yeah
12:51 poikilotherm
Simply running it on startup of a container will lead to KABOOM
12:53 pdurbin
It makes a lot of assumptions about what you've set up already.
12:53 poikilotherm
Is anything in that JSON stuff triggering other stuff than just adding things to the database?
12:53 * pdurbin
thinks
12:53 poikilotherm
Because if not, one could use some clever database snapshot and load that at container start time
12:54 poikilotherm
errr - postgres container
12:54 poikilotherm
not dataverse container
12:55 poikilotherm
This could of course happen inside an init-container, but you will need Kubernetes/OpenShift for that.
12:56 pdurbin
I think it mostly just populates the database. We used to index users in to Solr but we moved away from that. The OpenShift stuff has an init-container. We also populate the database with scripts/database/reference_data.sql
12:57 poikilotherm
Yeah, but that reference data is only about some indexing and other minor stuff
12:57 poikilotherm
Without loading the api/data/*json stuff, you can't use Dataverse
12:57 pdurbin
right
12:57 poikilotherm
(Running into errors, that no root dataverse can be found)
12:58 pdurbin
I tried your small container branch and got this: [ERROR] The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
12:58 poikilotherm
Ui
12:58 poikilotherm
That sounds bad
12:59 poikilotherm
That looks like mvn error, right?
12:59 pdurbin
yeah
13:00 poikilotherm
Could you give me some context please?
13:01 pdurbin
more of the error?
13:01 poikilotherm
Yeah
13:01 poikilotherm
I am shortly AFK, but will be back in a few minutes
13:01 pdurbin
I could throw all the output into a comment on your latest commit.
13:06 pdurbin
poikilotherm: I added it to https://github.com/poikilotherm/dataverse/commit/3a9f15807d3ab426c6ba1672285515e4e9c28a52#commitcomment-31498949
13:07 pdurbin
Ubuntu 16.04.5 if it matters
13:13 poikilotherm
re
13:15 poikilotherm
Oh tests failing... That most certainly is not my fault... :-D
13:15 poikilotherm
Can you try again with -DskipTests
13:18 pdurbin
Sure, it's getting farther, downloading a Payara image.
13:19 poikilotherm
Great :-)
13:19 poikilotherm
I didn't change tests, so this should behave well
13:25 pdurbin
unending wall of output
13:25 poikilotherm
:-D With some "dv>" at the start?
13:25 poikilotherm
That's the docker container ;-)
13:26 pdurbin
It finally stopped.
13:26 pdurbin
[ERROR] DOCKER> Unable to build image [iqss/dataverse:4.9.4] : "Unknown flag: chown" ["Unknown flag: chown" ]
13:26 poikilotherm
Oh you got a pretty old docker version
13:26 poikilotherm
Pls update :-D
13:27 pdurbin
I'm using the package from Ubuntu. You want me to update to the latest Ubuntu?
13:27 poikilotherm
Nope. Could you first tell me what docker version you are on?
13:28 pdurbin
Docker version 17.03.2-ce, build f5ec1e2
13:29 poikilotherm
Ah that's it. --chown is supported since 17.09
13:29 poikilotherm
Which is more than a year old... :-D
13:29 poikilotherm
https://docs.docker.com/install/linux/docker-ce/ubuntu/
13:29 poikilotherm
You can just update Docker with official packages instead of Upgrading Ubuntu
13:30 pdurbin
Can you make it "fail fast" if the version of Docker is too old?
13:31 poikilotherm
Uhm... Unsure. Maybe the docker maven plugin offers a check
13:31 pdurbin
Thanks. That would make for a better user experience.
13:34 poikilotherm
I could make it using a chown command instead
13:34 poikilotherm
Version checks seem not possible in Dockerfiles and dmp hasn't a check either
13:35 pdurbin
Can you at least add in a doc the version of Docker that's required?
13:35 poikilotherm
Of course I will do that :-D
13:35 pdurbin
Thanks.
13:35 poikilotherm
Forgive me, this is a WIP... ;-)
13:36 pdurbin
And if you can support the version of Docker I have with a chown command it would be appreciated.
13:36 poikilotherm
:-D
13:37 poikilotherm
I am about to push some more commits that deal with the dep stuff
13:37 pdurbin
We still support installing Dataverse on CentOS 6 even though CentOS 7 has been out for quite a while.
13:37 poikilotherm
So you maybe should pull in changes in a few moments and make a fresh build
13:38 poikilotherm
Will give you a sign when the have been pushed
13:38 pdurbin
I won't be able to try again on this laptop at home until this evening. I'm heading to work where I can try it from my Mac. Is that what you're developing on? A Mac?
13:38 poikilotherm
Nope, Linux Fedora 29 here
13:38 poikilotherm
But this should run on Mac, too. You just need Docker installed
13:39 poikilotherm
https://docs.docker.com/docker-for-mac/install/
13:39 pdurbin
Sure, I have Docker on my Mac at work. I have no idea what version but I'll check.
13:40 poikilotherm
I will push in the change for the chown
13:42 pdurbin
thanks
14:04 * poikilotherm
runs up the flag for pdurbin
14:12 pdurbin
postgres> ERROR: relation "sequence" does not exist at character 15
14:13 poikilotherm
Thats OK
14:13 poikilotherm
This should be just the same on a normal deployment
14:13 poikilotherm
I am just pushing in latest develop
14:13 poikilotherm
Rebasing on it
14:14 poikilotherm
And testing if the build still works ok
14:14 poikilotherm
Need to do a force push then agian
14:15 pdurbin
dv> Loading application [dataverse-4.9.4] at [/]|#]
14:16 poikilotherm
Seems to work ok :-D
14:16 pdurbin
That sounds promising. What port is Dataverse on?
14:16 poikilotherm
Please pull in my latest changes :-)
14:16 poikilotherm
(Was a force push because of rebasing on develop)
14:17 poikilotherm
8080 standard. But this will fail.
14:17 pdurbin
Yeah, I'm getting this: javax.persistence.NoResultException: getSingleResult() did not retrieve any entities.
14:17 poikilotherm
The setup-all.sh stuff is missing, so Dataverse is not yet ready to serve
14:17 pdurbin
Can you please remind me how to shut it all down? Just Ctrl-C?
14:18 poikilotherm
Yeah :-)
14:18 poikilotherm
As easy as that :-D
14:18 poikilotherm
And it will clean up after itself
14:18 poikilotherm
No dangling containers or networks
14:18 pdurbin
thanks, shutdown worked fine
14:18 poikilotherm
Perfect for IT tests :-D
14:19 pdurbin
This is the commit I just tried: 008b6dc Support Dataverse system properties as ENV vars.
14:19 poikilotherm
Latest commit id is 20c25ee5
14:19 pdurbin
Should I try it or will I get NoResultException again?
14:20 poikilotherm
You will get that again
14:20 poikilotherm
Didn't start work on the scripting stuff yet
14:20 poikilotherm
Just commited and pushed my work from yesterday
14:20 pdurbin
Ok. I'll wait. Time to bike to work anyway. Good progress. Thanks for all this!
14:20 poikilotherm
At least I got it to this point
14:20 poikilotherm
Sure
14:20 poikilotherm
Cu later
15:22 pameyer joined #dataverse
15:24 poikilotherm
Hi pameyer :-)
15:25 poikilotherm
I read that you tried my feature branch yesterday
15:27 poikilotherm
You could try to do a `git reset --hard <remote name here>/5292-small-container` and then try `mvn -Pcontainer -DskipTests clean package docker:build docker:run`
15:28 poikilotherm
Be warned that this will not give you a working Dataverse installation - there is still a lot of stuff missing.
15:28 poikilotherm
Currently hanging on the setup-all.sh stuff
15:28 poikilotherm
Gives me some headaches...
15:37 pameyer
hi poikilotherm - I did see `-Pcontainer` in the logs :)
15:37 pameyer
that shifted the error to `[ERROR] DOCKER> I/O Error [Unable to pull 'iqss/dataverse:4.9.4`
15:38 pameyer
yeah, from reading the pom I hadn't expected it to be fully working. but I was curious to see where the most broken bits were
15:38 poikilotherm
Uhm sound like the image was not built - did you include docker:build?
15:40 poikilotherm
Still missing: no Solr image included, Jhove missing, some special settings in glassfish like comet and ajp support, mail ressource not yet provided, config of solr and admin email, doi config, running all of setup-all.sh or replace with sth else
15:40 pameyer
ah, no - I'd assumed dependency resolution
15:40 poikilotherm
Especially the last one and the solr image is the next step to tackle
15:41 pameyer
saw the missing solr, but I've got a reasonable idea of how that breaks things
15:41 poikilotherm
Oh you do? Could you share that with me? That would be quicker than try-and-error
15:42 pameyer
trying to get configuration working w\ containers (or ansible) made it somewhat obvious that they weren't designed for it
15:42 poikilotherm
:-D
15:42 poikilotherm
Oh yes...
15:42 pameyer
with no solr, most APIs usually work - but usually browsing doesn't
15:42 poikilotherm
Ok, so the setup-all.sh stuff should work without solr?
15:43 pameyer
I don't remember if the create dataset page (or just view dataset page) work without solr, although I'd expect some brokenness
15:43 drew-jhu joined #dataverse
15:43 poikilotherm
That would be very interesting for the IT testing stuff. That would mean solr does not need to be included for most tests there.
15:44 poikilotherm
Just for things where it is explicitly used and of course things like selenium end to end tests
15:44 pameyer
I've avoided touching setup-all.sh. my usual approach has been that I want my dev (or ansible) setup to be as close to prod as possible - and the prod install starts from dvinstall.zip/install
15:44 poikilotherm
Yeah...
15:44 pameyer
there are ITs for search, which will fail without solr
15:44 poikilotherm
But simply calling the setup-all.sh stuff on a container start will result in KABOOM
15:44 poikilotherm
It shoots at Solr with big delete guns
15:45 pameyer
that's an *interesting* race condition
15:45 pameyer
especially if you've got autoscaling on
15:45 poikilotherm
;-)
15:45 poikilotherm
That script is simply not made for those scenarios
15:46 poikilotherm
Which is ok, but that needs to be addressed then
15:46 poikilotherm
And I worry about the API stuff - just executing that on every start might change configurations someone added to a running dataverse installation
15:46 poikilotherm
Thus my "setup-all.sh gives me headaches"
15:46 poikilotherm
;-)
15:47 pameyer
why'd you drop the init container approach?
15:47 poikilotherm
Didn't drop it
15:47 poikilotherm
It is just not possible with Docker alone
15:47 poikilotherm
That is KUbernetes specific stuff
15:47 pameyer
:doh:
15:47 pameyer
meant multi-stage build
15:48 poikilotherm
How would you apply a database based config at build time?
15:49 poikilotherm
Sry - gotta run now, pickup kifs
15:49 poikilotherm
Will read up the logs later
15:49 pameyer
horrible ugliness. one stage for install postgres,solr,gf,etc - mess w\ /etc/hosts for the service names (db -> localhost) for install time; copy to separate stages
15:49 poikilotherm
Maybe say hello later
15:49 pameyer
poikilotherm - have a good night
15:52 pameyer
and thanks for the tips on your branch - I got it to startup, but not deployed. not at all urgent
15:57 pameyer
pdurbin: docker for mac switched things around to try to ask for a login before downloading
15:57 pameyer
if I'm remembering right, they're still available unauthenticated from somewhere - I *think* the release notes page
15:57 pdurbin
pameyer: huh. I thought Docker was open source. Weird.
15:58 pdurbin
on my Mac I have this version: Docker version 18.03.1-ce, build 9ee9f40
15:58 pameyer
yeah, but their docker desktop wrapper isn't :(
15:58 pameyer
hopebrew should work too, but I haven't tried that. it's on my list to switch to, but very low down
15:58 pdurbin
even newer than my Ubuntu laptop at home so I'm hoping it'll "just work"
15:59 pdurbin
Yeah, I didn't use homebrew to install Docker. I grabbed a dmg.
16:05 pdurbin
pameyer: I got a little lost in the logs above but setup-all.sh should work fine even if Solr isn't running
16:05 pdurbin
drew-jhu: hi! Welcome. I was just thinking of JHU. :)
17:15 pameyer
pdurbin: you're not worried about breaking other stuff w\ the jsessionid hammer?
17:19 nanoz joined #dataverse
17:21 pdurbin
pameyer: hi! Nope. Should I be worried?
17:22 pameyer
I don't know of a reason to be worried.
17:22 pdurbin
pameyer: what are you worried might break?
17:22 pameyer
it's more that I don't know that things won't break.
17:23 pameyer
but I have confidence in kcondon
17:24 pdurbin
Me too. And it's a small change. Easy to back out if things go sideways. :)
18:26 drew-jhu joined #dataverse
18:44 donsizemore joined #dataverse
19:48 poiki-at-home joined #dataverse
19:48 poiki-at-home
Pling plang plong
19:48 poiki-at-home
pameyer you actually gave me some inspiration
19:49 poiki-at-home
I am thinking about using a multistage build for a database image, that will build a SQL dump and one can use that to init a new DB
19:49 poiki-at-home
Most of the time, the postgres containers will be statefull
19:49 poiki-at-home
It's a database... ;-)
19:50 poiki-at-home
So just trigger a script that checks a default = false ENV var, and if true, the dump gets imported
19:51 poiki-at-home
That way for dev or demo purposes you get quick a quick database up and running, but can still use a relativly simple postgres image for production
19:51 poiki-at-home
That image and the script is the second step of the multistage build, copying the dump artifact from stage 1
19:54 pameyer
dump/restore is probably more robust than copying the pg data directory around
20:34 pameyer
pdurbin: I need to figure out how to configure my jenkins to send you a kudos whenever it makes my release testing easier ;)
20:42 pdurbin
Heh. Did you find a regression? I hope not.
20:42 pdurbin
poiki-at-home: you don't sound blocked. Good. :)
20:46 pameyer
pdurbin: mdunlap pointed it out to me. updates to pip caused some strangeness
20:47 pdurbin
yeah, we've been talking about it. sounds like you fixed it. thanks
20:49 pameyer
looks fixed to me now
20:52 pdurbin
good, he's trying it
20:53 * pameyer
crosses fingers
21:00 pdurbin
pameyer: what was the fix?
21:05 pameyer
version pinning for cent6
21:06 pdurbin
ah, versions of importlib and redis according to https://github.com/sbgrid/data-capture-module/commit/df160b0
21:10 pameyer
yup
21:10 pameyer
that cron to request queue switch is still haunting me
21:18 pdurbin
It's a blessing that I don't remember everything.
22:05 isullivan joined #dataverse
22:53 poiki-at-home72 joined #dataverse
22:54 poiki-at-home72
pdurbin: Sry, was busy with statics for our house...
22:54 poiki-at-home72
I'm not really blocked, just thinking :-d
22:55 poiki-at-home72
Will just carry on and try to come up with stuff ;-)
22:55 poiki-at-home72
And as you may have noted, I am into perfectionism
22:57 poiki-at-home72
I am off to bed now... Read you tomorrow guys
22:57 poiki-at-home72 left #dataverse
23:47 pdurbin
statics
