Time
S
Nick
Message
04:12 knikolla joined #dataverse
04:12 JonathanNeal joined #dataverse
04:12 jhand joined #dataverse
04:22 JonathanNeal joined #dataverse
04:27 karissa joined #dataverse
08:13 jri joined #dataverse
13:08 danmcp joined #dataverse
13:32 jri joined #dataverse
14:08 danmcp joined #dataverse
14:59 danmcp joined #dataverse
15:33 karissa joined #dataverse
15:51 TyrelSouza joined #dataverse
15:51 danmcp joined #dataverse
17:07 danmcp joined #dataverse
17:09 bret joined #dataverse
18:19 danmcp joined #dataverse
18:25 danmcp joined #dataverse
19:12 TyrelSouza
Hello channel, my first time here - pdurbin has suggested I look into helping with some dataverse stuff, but I'm trying to get tests running and one is still failing. pdurbin you mentioned asking here for help getting `TestConnection.test_get_dataverse` running properly?
19:21 pdurbin
TyrelSouza: hi! I just got back. Thanks for jumping in here! The first question is if there's a GitHub issue tracking the problem yet. Do you happen to know? I can go look in a minute.
19:21 TyrelSouza
yeah its like, 32 i think? lemme checj
19:22 TyrelSouza
Oh no thats test_create_dataverse that was in the one i thought, not test_get_dataverse
19:24 TyrelSouza
... of course when trying to duplicate it, i get 16 failing now! neat.
19:24 TyrelSouza
looks like a 503 on demo.dataverse, ahhh
19:27 pdurbin
woof, you're right about the 503
19:31 pdurbin
I'm making some noise in Slack about it.
19:32 pdurbin
TyrelSouza: meanwhile, can you please point the tests at http://phoenix.dataverse.org instead? That "phoenix" server is running the latest from our "develop" branch rather than our latest release, which is what demo runs.
19:33 TyrelSouza
http or https? Connection seems to default to https in the tests, will that affect anything?
19:35 pdurbin
I never did get around to putting a cert on that box.
19:35 TyrelSouza
ah yeah that would explain the "Failing max tries" thing
19:36 pameyer joined #dataverse
19:37 pameyer
shouldn't be a problem to run tests over http (aside from the obvious unencrypted bits)
19:38 pdurbin
https://dev1.dataverse.org has a valid cert but a random branch on it. I can put the latest release on there (or develop) on there if you want, TyrelSouza
19:38 pameyer
where is `TestConnection` coming from?
19:38 TyrelSouza
Oh sure, then i don't have to change Connection to default use_https to false for getting things testable
19:38 TyrelSouza
(or pass it in in the tests)
19:39 pdurbin
TyrelSouza: ah, ok. Then please use that dev1 server if it's easier.
19:40 pdurbin
oh hey, pameyer
19:40 pdurbin
TyrelSouza: pameyer is a Python person too
19:40 TyrelSouza
Nice!
19:40 pdurbin
pameyer: hey the research computing lunch was great. Ista and I went. I'll fill you in later.
19:41 pameyer
pdurbin: good to hear
19:41 pameyer
TyrelSouza: hi - don't think we've (virtually) met before
19:41 TyrelSouza
all tests pass too! with dev1
19:41 TyrelSouza
I don't believe so either, hello
19:41 pdurbin
pameyer: TyrelSouza and I hang out in #positivepython. I should drag you along to the next lunch.
19:43 pameyer
ah - so is this dataverse python binding tests then?
19:47 pdurbin
pameyer: yes! TyrelSouza is taking a look!
19:47 pdurbin
I'm glad to hear the tests are passing on dev1.
19:48 pameyer
good to hear!
19:48 pdurbin
TyrelSouza: so what's your take on https://github.com/IQSS/dataverse-client-python/issues/44 ? That issue says "test_create_dataverse failed".
19:51 TyrelSouza
I wonder if they were running into the same thing, with demo or phoenix having a 503 error on them at the time?
19:51 TyrelSouza
although wait no -- that would mean the rest of them failed, and it only says this one..
19:53 pdurbin
TyrelSouza: do you mind leaving a comment with your experience so far? It would be nice for that guy to hear that someone got the test suite to work.
19:53 TyrelSouza
yeah, maybe my comment will prompt them to reply
19:54 pdurbin
yep
19:54 pameyer
having the tests depend on demo / dev1 / apitest seems odd to me
19:55 TyrelSouza
It does seem kind of counter intuitive at first, but if the tool is used to connect directly to dataverse -- having to manage both a mock test and a live test might get out of sync with changes to the dataverse code
19:56 pdurbin
Well, we are trying to keep the Dataverse APIs stable.
19:57 pameyer
the dataverse integration tests (which also need a server) can be run with docker now; which wouldn't completely solve the sync problem but I think might mean you wouldn't need a mock
19:58 pdurbin
I guess I don't understand the sync problem but I trust you two. :)
20:04 pameyer
pdurbin: dataverse-python and dataverse API sync
20:04 pdurbin
But are we suggesting that all the python code should go in the main dataverse repo?
20:05 pameyer
nope
20:05 pdurbin
good :)
20:05 pameyer
but I was thinking that dataverse-python could test against main dataverse docker, rather than dev1 / demo / etc
20:06 pameyer
it would allow for a "known working" dataverse API version, and avoid the external dependencies; but it would also mean more developer setup
20:06 pameyer
so there's always tradeoffs
20:07 pdurbin
You're saying something (Travis or whatever) should kick off a build and deployment of a Dataverse server and then use that server when testing the Python client library? Yes. Sure. Sound fancy and nice.
20:08 pameyer
something like that
20:09 pdurbin
I think that's more or less what bjonnh and I were talking about a while back.
20:10 TyrelSouza
yeah that would make sense, run a "local live server test"
20:11 pdurbin
TyrelSouza: part of pameyer 's point (perhaps) is that if you'd like to run Dataverse in Docker on your laptop, we have a readme you can follow if you want. Or if you're more into Vagrant, we've got that too. :)
20:12 pdurbin
pameyer: I still reach for Vagrant because `vagrant up` is fewer commands. Just the one. :)
20:14 pameyer
pdurbin: I finally got the quote escaping sorted, so it's one fewer command now
20:15 pdurbin
pameyer: interesting. So you're working on a pull request? :)
20:16 pdurbin
pameyer: the tweaks I added to the readme are in https://github.com/IQSS/dataverse/pull/4434/commits if you'd like to add to them.
20:16 pdurbin
Rather than opening another pull request.
20:19 pameyer
still needs a little work - I'd like to get it to automaticly retry on setupIT failures (since I have no ideas why they intermittantly fail)
20:20 pameyer
I've been trying to keep with the "small chunks"...
20:21 pdurbin
pameyer: ok, no worries if that PR gets merged before you get to it. We can always open another one. They're free. :)
20:21 danmcp joined #dataverse
20:23 pameyer
I've got a reasonable number of PRs with single and double digit character count changes
20:25 pdurbin
pameyer: you're really racking up the merged PRs. I love it.
20:30 danmcp joined #dataverse
20:38 TyrelSouza
I'll try to get a vagrant running, the docs on the dev guide on dataverse.org?
20:38 pdurbin
TyrelSouza: yep. Here: http://guides.dataverse.org/en/4.8.5/developers/tools.html#vagrant
20:39 TyrelSouza
oh... thats stupid easy
20:39 pdurbin
:)
20:41 TyrelSouza
hmm. i guess not so stupid easy. `The box 'puppetlabs/centos-7.2-64-puppet' could not be found`
20:42 TyrelSouza
It's still on vagrantcloud site, hopefully the problem is on my side
20:42 pameyer
`geerlingguy/centos7` might work too
20:42 pdurbin
Huh. It's not downloading for you? It's cached for me.
20:43 TyrelSouza
Yeah says not found :( https://www.dropbox.com/s/57qnat0l5chhjoj/Screenshot%202018-03-02%2015.43.06.png?dl=0
20:43 pdurbin
404
20:43 pdurbin
bah
20:43 TyrelSouza
hmm it shoudl be there now
20:43 pameyer
sometimes `vagrant box add $foo` prior to `vagrant up` helps
20:43 pameyer
sometimes not
20:45 pdurbin
Should we stop using `config.vm.box = "puppetlabs/centos-7.2-64-puppet"`? Is it broken? 404?
20:46 TyrelSouza
it might be me, i cant get `geerlingguy/centos7` either
20:46 pameyer
:(
20:46 TyrelSouza
Both seem to be in enterprise systems I don't have access to on terraform?
20:48 pameyer
terraform?
20:48 TyrelSouza
i think terraform is where hashicorp hosts a bunch of their vagrant boxes? not sure. it said:
20:49 TyrelSouza
if this is private, use `vagrant login`, then i did that and it said If you do not have an Atlas account, sign up at
20:49 TyrelSouza
https://atlas.hashicorp.com .
20:49 TyrelSouza
which redirects to terraform now
20:49 pameyer
gotcha - that didn't exist when I last messed with my vagrant configs
20:50 TyrelSouza
but yeah, its probably my configs, they both can't be down...
20:51 TyrelSouza
>The syntax for the vagrant box add subcommand is changed with the version 1.5, due to the Vagrant Cloud introduction.
20:51 TyrelSouza
hmm. i guess thats the problem
20:52 pameyer
which version of vagrant are you on?
20:54 pameyer
I've been on 1.7.4 for a while
20:54 TyrelSouza
1.9.1 i think?
20:54 TyrelSouza
and oh https://app.terraform.io/geerlingguy/centos7 yeah its a 404 for me
20:58 pdurbin
TyrelSouza: you can keep using that dev1 server. I'm happy to deploy whatever branch there for you. Sorry for all the Vagrant trouble. :(
21:01 TyrelSouza
sounds good :)
21:01 pdurbin
You could also try our "all in one" Docker thing: http://guides.dataverse.org/en/4.8.5/developers/dev-environment.html#all-in-one-docker-images-for-testing
21:02 pdurbin
TyrelSouza: do you want a specific branch on dev1? develop maybe?
21:02 pameyer
that *should* work; but if dev1 is working now it might make sense to leave docker-aoi to a not-friday-afternoon time
21:02 pdurbin
I'm not sure what your goal is at the moment. Anything you do helps. :)
21:03 TyrelSouza
Right now i just wanna get everything running so i can play around and see what everything is, thats my first goal.
21:03 TyrelSouza
and oof its already 4olock?
21:04 pdurbin
TyrelSouza: you should go enjoy your weekend. Stay dry!
21:04 pdurbin
Thanks again from running the Python test suite.
21:04 TyrelSouza
We're going Gokating and dave and buster's tomorrow -- it'll definitely be a nice break from the "I'm ramping up the job hunt" mentality.
21:05 pdurbin
nice
21:39 TyrelSouza
I did the dead simple fix - updated vagrant to 2.0.2, it's working.
21:41 pdurbin
\o/
21:41 pdurbin
good job TyrelSouza
21:42 TyrelSouza
I figured nuke it.. it had to be me.
22:01 danmcp joined #dataverse
22:11 danmcp joined #dataverse
22:51 aculich joined #dataverse
22:54 aculich
anyone around who can answer a few installation questions and/or how to use the docker containers that are mentioned (but undocumented) in the developer docs? http://guides.dataverse.org/en/latest/developers/dev-environment.html?highlight=docker
22:56 pameyer
hi aculich
22:57 pameyer
I might be able to answer questions about installation / docker
22:58 pameyer
what questions do you have?
23:01 aculich
I am at UC Berkeley working with our Librarians and group of undergrads working on a Dataverse project.... we are not familiar with setting up Dataverse and we're running into installation problems on a CentOS7 system
23:01 aculich
I saw that there were some docker containers which I would prefer to use anyway (and also Kubernetes), but I don't see much documentation to help us get going using those containers
23:02 pameyer
ok - it sounds like your targeting a demo or staging system, rather than an individual developer setup; right?
23:02 aculich
yeah, a demo or staging system would be helpful (though the ind. dev. system wouldn't hurt, too)
23:03 aculich
whichever we can get running quicker would be useful
23:03 aculich
in the longer run having both would also be nice
23:04 aculich
btw: are you on Eastern Time? Just realize late on a Friday evening might not be a good time to be diving into this with ya'll, but wanted to reach out to start the conversation anyway
23:04 pameyer
the docker/kubernetes/minishift containers that are described on that page were are an initial prototype (if I'm remembering correctly, pdurbin had them tagged as "kick the tires" at one point)
23:04 pameyer
I'm EST; but I tend towards the later shift
23:04 pameyer
so I'll probably be around for another 30m-1hr or so
23:04 aculich
we're fine with "kick the tires" stuff... and we hope we might be able to contribute useful stuff back to your project
23:05 aculich
I just don't know anything about installing the Dataverse software itself (but plenty familiar with Docker/Kubernetes/cloud providers)
23:05 pameyer
for a multi-user system, it might be easier to get a standard cent7 install running
23:05 pameyer
ah - gotcha
23:05 aculich
yeah, that's what we have been trying using an XSEDE Jetstream VM
23:07 aculich
we're using one of these VMs: https://use.jetstream-cloud.org/application/images/241
23:07 pameyer
for some background, there's some ongoing work with those docker containers focused on getting them running on openshift. I think pdurbin may have left for the day, but he's got more details than I do. there's also a more minimal (all-in-one) docker image available, but it's not designed for persistence or multi-user
23:07 aculich
btw: we're happy to keep chatting here.... and/or I can send you a Zoom link if you want to chat with us via voice
23:08 pameyer
let's stay here for now; maybe zoom next week (when more EST folks are only) if it would be helpful
23:09 pameyer
one thing I don't see in that VM link is how much RAM the VM will have
23:09 pameyer
but I haven't used jetstream
23:10 aculich
sounds good chatting here for now
23:10 aculich
with the students there is a time next friday earlier in the day, so if we're still stuck it might be helpful to chat with your team via zoom
23:10 pameyer
in my experience, dataverse usually needs at least 2 GB for a minimal system; but for multi-user you'd probably want more
23:11 aculich
as for Jetstream, it's an OpenStack system with a simplified UI called Atmosphere (developed by the CyVerse folks)
23:11 aculich
currently we're running with 4GB on this VM
23:12 pameyer
so RAM should be good.
23:12 aculich
we are just running this for development/integration purposes.... not a production system (yet)
23:12 aculich
we are working on an integration with Dataverse+Binderhub http://binderhub.readthedocs.io/en/latest/
23:12 pameyer
are you thinking to use swift storage on openstack?
23:12 aculich
which is a project under the Jupyter umbrella (along with JupyterHub)
23:12 aculich
which uses docker containers + Kubernetes
23:13 aculich
as for storage, there are many different storage backends.... we could use swift storage on openstack with Jetstream, but we'll also plan to deploy to Google Cloud, AWS, and Azure
23:14 pameyer
do you have a timeframe for your project?
23:14 aculich
with the undergrads working with us we're thinking on semester-long timelines
23:15 pameyer
makes sense
23:15 aculich
and they're working on it for a couple of hours on one day a week.... so not a lot of time, but just enough to get something done in a semester around this integration project
23:15 aculich
once we have a working set of docker containers with Dataverse running inside it we can more easily compose the containers with the Binderhub system
23:16 pameyer
from a quick read through of the binderhub page, I'm assuming that the compute pipelines are abstracted as docker images with a jupyter notebook UI
23:17 pameyer
I'm guessing the plan is to either run those pipelines on datasets in dataverse, or deposite the pipelines alongside datasets
23:17 aculich
but right now we don't have a lot of on-site hands-on experience installing Dataverse so we thought we'd ask here first before reinventing the flat tire on our own
23:17 pameyer
:)
23:17 aculich
yeah, that's the right idea
23:17 aculich
we can easily move the compute environments around to wherever the data lives
23:18 aculich
or we can move the data via Globus and high-speed networks to where the compute lives
23:19 pameyer
sounds similar to what I try to do :)
23:19 aculich
in a sec one of the students will log in here to share the error he encountered when installed on the CentOS7 VM
23:19 pameyer
sounds good
23:19 aculich
oh? what is it you're working on and trying to do?
23:20 sdooher joined #dataverse
23:21 pameyer
moving https://data.sbgrid.org to dataverse ; aka - big data upload/replication to HPC sites/download, direct compute access
23:21 pameyer
part of a collaboration with IQSS
23:21 pameyer
hi sdooher
23:22 aculich
thanks for the pointers to SBGrid....
23:22 sdooher
Hi, I'm one of the students working with aculich on the UC Berkeley project
23:22 aculich
...i'll look around and I'm sure i'll have more questions in the future about it and the underlying infrastructure
23:22 pameyer
aculich mentioned you had some installation questions
23:22 pameyer
aculich: sure
23:22 aculich
is it possible to get demo access to the SBGrid to kick the tires and try it out?
23:22 sdooher
The issue we have ran into so far is that while running the dvinstall script, it is failing on the glassfish deployment step with the following error:
23:22 sdooher
Copying additional configuration files... done! Attempting to deploy the application. Command line: /usr/local/glassfish4/bin/asadmin deploy dataverse.war remote failure: Error occurred during deployment: Exception while loading the app : EJB Timer Service is not available. Please see server.log for more details. Command deploy failed. Failed to deploy the application! WAR file: dataverse.war. (exit code: 256)
23:23 sdooher
If we run the glassfish asadmin deploy it deploys, but then the page only loads with a blank screen
23:23 pameyer
sdooher: this is with an interactive installation, correct?
23:24 pameyer
aculich: unfortunately we don't have a demo system available :(
23:24 sdooher64 joined #dataverse
23:25 aculich
no worries
23:25 sdooher64
Sorry about that, I timed out
23:25 sdooher64
Yes I believe so. I am using the dvinstall script from one of the releases on github
23:26 pameyer
do you know which version?
23:26 pameyer
also - is glassfish running as root, or a non-root user?
23:26 sdooher64
It's a nonroot service account I setup for glassfish. v4.8.1
23:28 pameyer
ok - let me check something, but I think I may have an idea what's going on
23:28 pameyer
could you let me know which postgresql version you're running?
23:29 aculich
actually.... looks like sdooher64 got it running:
23:29 aculich
http://149.165.170.58:8080/
23:29 pameyer
:)
23:30 sdooher64
If found this on github that fixed the issue: https://github.com/IQSS/dataverse/releases
23:30 pameyer
congrats!
23:30 sdooher64
Oh wrong link: https://github.com/IQSS/dataverse/issues/3669
23:30 aculich
do you have ansible deployment scripts to deploy non-interactively?
23:32 pameyer
https://github.com/IQSS/dataverse/blob/develop/conf/docker-aio/install.bash
23:32 pameyer
much less ugly than my ansible scripts; but that's an example of using the non-interactive installation
23:33 aculich
ah, okay.... so that can be run after manually installing the pre-reqs?
23:33 pameyer
I haven't used https://github.com/IQSS/dataverse-ansible , but that might be worth a look
23:34 aculich
good sign that it looks like files have been updated within the last week, so it's under active development?
23:34 pameyer
correct - the `docker-aio` image is intended for integration tests, but I started off doing a standard (non-interactive) install because I was more farmiliar with that
23:35 pameyer
dataverse-ansible isn't an IQSS project, but donsizemore does a pretty good job keeping it up to date
23:35 pameyer
and he's usually pretty open to pull requests
23:35 aculich
great, this has been super helpful! thank you for spending time with us this afternoon.... we're wrapping up shortly, and will be working on this again next Friday
23:36 pameyer
glad to help - I'll spread the word so there are iqss folks online
23:36 aculich
awesome! much appreciated
23:36 aculich
have a great weekend!
23:38 pameyer
you too!
