Time
S
Nick
Message
14:59 pdurbin
telnoratti bjonnh: https://irclog.perlgeek.de/crimsonfu/2016-11-06#i_13523349
16:15 bjonnh
f
16:15 bjonnh
sory
16:15 bjonnh
sorry
16:16 pdurbin
heh, no worries :)
16:17 bjonnh
pdurbin: nobody answered?
16:17 pdurbin
not yet
16:21 bjonnh
what channel is that?
16:21 bjonnh
did you try my option?
16:30 pdurbin
#crimsonfu on freenode - http://crimsonfu.github.io
16:31 pdurbin
bjonnh: if you the option (commit) at http://irclog.iq.harvard.edu/dataverse/2016-11-03#i_44444 I think it's a great idea but no, I haven't tried it yet
16:47 bjonnh
?
17:21 pdurbin
bjonnh: sorry, I must have confused you. Do you have a question?
17:22 bjonnh
I didn't understand your sentence
17:23 bjonnh
"if you the option (commit) at"
17:28 pdurbin
whoops! I meant to say, "if you *mean* the option..."
17:36 bjonnh
ok
17:36 bjonnh
:)
17:42 pdurbin
bjonnh: so you did mean that commit?
17:52 bjonnh
yes
18:20 pdurbin
bjonnh: so I end up with two Docker containers? And would I run them locally? Or deploy them to AWS or similar? Or both?
19:49 bjonnh
no
19:49 bjonnh
there is only one for now
19:49 bjonnh
but it is built in two parts
19:49 bjonnh
so it is quicker to rebuild
19:49 bjonnh
because all the packages are ready
19:50 bjonnh
I'm envisionning having multiple containers
19:50 bjonnh
one for solr, one for the db
19:50 bjonnh
etc
19:52 pdurbin
Ok, so I end up with a single Docker container. That sounds good! And I can deploy it to the cloud somewhere? Maybe AWS or some other place that hosts Docker containers?
20:03 pdurbin
It looks like Amazon calls it "EC2 Container Service" (ECS) – Docker Management – AWS - https://aws.amazon.com/ecs/
20:10 pdurbin
must be what telnoratti is using
20:21 bjonnh
yeah
20:21 bjonnh
that's also something I have to look
20:21 bjonnh
I remember that you can try aws for free
20:21 bjonnh
will see if it is worth it
20:26 pdurbin
bjonnh: so you're saying I can build the Docker container on my laptop and push that container directly to ECS or whatever?
20:40 telnoratti
pdurbin: actually ECS was too limiting, I'm using coreos
20:41 pdurbin
huh, ok
20:41 telnoratti
Just deployed a cluster in ec2 instances
20:42 pdurbin
telnoratti: can you please take a look at https://github.com/bjonnh/dataverse/commit/f7a82c9fc37aa1c901534d8a91f92ba18f3c3ce4 ?
20:48 telnoratti
bjonnh: take a look at our run_dataverse.sh in dataverse-aws. we found some settings needed to happen in each Glassfish container like the apiuser.
20:51 pdurbin
telnoratti: do you mean this? "${ASADMIN}" create-jvm-options "-Ddoi.username=${DOI_USERNAME}"
20:52 telnoratti
Yeah and ulimit, though that's less critical
20:53 pdurbin
ok, one of the many JVM options: http://guides.dataverse.org/en/4.5.1/installation/config.html#jvm-options
20:54 pdurbin
telnoratti: would it be helpful if we switch some of those JVM options over to database settings? http://guides.dataverse.org/en/4.5.1/installation/config.html#database-settings
20:55 telnoratti
Yeah I think so
20:55 telnoratti
It seems weird that they're different
20:59 pdurbin
yeah, we started with JVM options and then decided to switch to database settings
20:59 pdurbin
so the oldest features developed during the Dataverse 4 rewrite use JVM options
21:00 pdurbin
and the ones developed more recently use database settings
21:02 pdurbin
telnoratti: but getting back to that commit by bjonnh... could I use it to build a Docker container that will run on AWS?
21:05 pdurbin
or would some coreos need to be sprinkled in there first?
21:08 telnoratti
You should be able to run that on ECS I think
21:09 telnoratti
The block we hit was s3fs
21:09 pdurbin
oh! great!
21:23 pdurbin
telnoratti: I don't really understand what the block you hit was.
21:39 pdurbin
bjonnh: ok, I'm running `build_docker.sh` and it's doing stuff
21:50 pdurbin
uh oh, I'm getting this: Pulling repository docker.io /bjo/pre_dataverse Error: image bjo/pre_dataverse:latest not found
21:50 pdurbin
more output at https://github.com/bjonnh/dataverse/commit/f7a82c9fc37aa1c901534d8a91f92ba18f3c3ce4#commitcomment-19715521
21:50 pdurbin
bjonnh telnoratti: any ideas?
21:52 telnoratti
pdurbin: there wasn't a good way to get s3fs working, which we decided we would use for shared file space. I know a few places that have found running coreos clusters on aws better than ecs
21:55 telnoratti
pdurbin: looks like step 8 of pre_dataverse failed
21:59 pdurbin
ah, this: lstat downloads/weld-osgi-bundle-2.2.10.Final-glassfish4.jar: no such file or directory
22:01 telnoratti
Yeah, is there a script that pulls in those downloads?
22:01 pdurbin
yeah, at https://github.com/IQSS/dataverse/blob/v4.5.1/downloads/download.sh
22:02 telnoratti
Oh yeah I see it
22:02 pdurbin
I had downloaded glassfish and solr but that's it.
22:02 pdurbin
(In dev I don't bother to patch weld or grizzly.)
22:04 pdurbin
anyway, I downloaded that weld patch and `build_docker.sh` seems to be getting farther. Thanks!
22:04 telnoratti
bjonnh: I haven't seen this pattern before. Is the pre_image just a way of separating out the frequently changing parts and the stable parts of the build to make it faster?
22:05 pdurbin
telnoratti: that's what bjonnh was saying at http://irclog.iq.harvard.edu/dataverse/2016-11-03#i_44421
22:06 pdurbin
I think. :)
22:06 telnoratti
Ah yeah, I was just wondering about the rationale
22:06 telnoratti
I'm barely not a docker newbie myself
22:06 pdurbin
thank goodness we log this channel :)
22:15 pdurbin
takes a while to build the war file
22:21 pdurbin
"Successfully built 78c23dae7fa2"! I think that means it's done.
22:24 pdurbin
but where is the Docker container I just build?
22:24 pdurbin
built*
22:32 pdurbin
oh, `docker images` shows some stuff
22:33 pdurbin
now to get it off my laptop and onto AWS
22:39 pdurbin
somehow :)
23:14 bjonnh
telnoratti: yes
23:15 bjonnh
telnoratti: also it will allow to make different configurations later
23:15 bjonnh
telnoratti: separated shib or net
23:15 bjonnh
or not
23:15 bjonnh
etc
23:15 bjonnh
pdurbin: if you can create a test account for me, I can try to develop something to push to aws
23:16 bjonnh
pdurbin: or telnoratti can…
23:16 bjonnh
but there is still a lot of work to do on the docker itself
23:18 pdurbin
bjonnh: when I run `docker images` I can see "bjo/dataverse" but how can I export this to a file so I can upload it to AWS and run it there?
23:29 bjonnh
first
23:29 bjonnh
try it
23:30 bjonnh
docker run -p 8888:80 -p 9999:443 bjo/dataverse
23:31 bjonnh
docker inspect --format '{{ .NetworkSettings.IPAddress }}' theidofthecontainer
23:31 bjonnh
(you can find the id with docker ps)
23:31 bjonnh
then put your browser to http://172.x.y.z
23:31 bjonnh
etc
23:31 bjonnh
should see the dataverse running
23:32 pdurbin
bjonnh: it seems to work. I can see Dataverse running at http://localhost:8888/
23:32 bjonnh
oh means that I did do an export
23:32 bjonnh
I'll have to correct that
23:32 bjonnh
docker save -o filename.img bjo/dataverse
23:32 bjonnh
then on aws
23:33 bjonnh
docker load -i filename.img
23:33 bjonnh
( http://stackoverflow.com/questions/23935141/how-to-copy-docker-images-from-one-host-to-another-without-via-repository )
23:35 pdurbin
I can also see Dataverse at http://172.17.0.2/
23:35 pdurbin
I used `docker ps` to figure out theidofthecontainer
23:35 bjonnh
great
23:37 pdurbin
`docker save` is taking a while
23:39 pdurbin
filename.img is 2.2 GB
23:40 bjonnh
ouch :p
23:42 pdurbin
and my daughter is watching Harry Potter (first movie) so uploading is a bit slow :)
23:58 pdurbin
https://console.aws.amazon.com/ecs/home?region=us-east-1#/firstRun says, "Create and manage a new private image repository and use the Docker CLI to push and pull images. Access to the repository is managed through AWS Identity and Access Management."
23:58 pdurbin
sounds like what I want, I guess
