Time
S
Nick
Message
07:04 icarito[m] joined #dataverse
07:22 poikilotherm joined #dataverse
07:30 jri joined #dataverse
07:47 rigelk joined #dataverse
08:59 jri joined #dataverse
09:02 jri_ joined #dataverse
09:03 poikilotherm
pdurbin: See Jenkins and Mr Robot in action...
09:04 poikilotherm
https://i.imgur.com/Zyr8950.png
09:04 poikilotherm
https://github.com/IQSS/dataverse-kubernetes/pull/74
10:04 juancorr joined #dataverse
10:14 pdurbin
poikilotherm: awesome
10:29 poikilotherm
I hope for donsizemore to materialize here...
10:34 pdurbin
poikilotherm: me too. Let's add your config to dataverse-jenkins. I just created this issue: https://github.com/IQSS/dataverse-jenkins/issues/11
10:35 poikilotherm
Yeah, I wanted to play around with it first :-D
10:35 pdurbin
sure
10:36 pdurbin
And I don't know how we should name these files. config.xml is too broad for what it is. IQSS-dataverse-api-test-suite-docker-aio-develop.xml may be a little long.
10:36 poikilotherm
maybe the xml should be captured in a subfolder k8s
10:36 poikilotherm
I plan to add more jobs
10:37 pdurbin
subfolders are fine
10:37 pdurbin
We're just getting started.
10:37 poikilotherm
Is this repo for documentation / examples or do you plan to do gitops on it?
10:38 pdurbin
It's not for examples. It's supposed to represent the actual jobs in use. So that anyone can spin up their own Jenkins server. If you run a fork or whatever, for example.
10:39 pdurbin
Please see "How to install Jenkins to build and test Dataverse" at https://github.com/IQSS/dataverse-jenkins/issues/1
10:40 pdurbin
I've learned a lot. A month or so ago I had never installed Jenkins. :)
10:45 poikilotherm
pdurbin would you be ok with me adding a docker hub user similar to the dataverse-bot?
10:45 poikilotherm
To push images automatically, I will need an account over there...
10:55 pdurbin
Do you want to use your Docker Hub account for now?
10:56 poikilotherm
Meh :-/
10:56 poikilotherm
I could, but that means adding my personal credentials to Jenkins
10:56 pdurbin
yuck
10:56 poikilotherm
I am not very fond of that...
10:57 pdurbin
What name would you give it?
10:57 poikilotherm
dataverse-bot?
10:57 poikilotherm
Same name as github
11:10 pdurbin
Sure, sounds fine.
11:11 pdurbin
Sorry, distracted. Making breakfast. And my nine year old and I just showed her mom this crazy YouTube video that we keep singing to.
11:12 xarthisius joined #dataverse
11:12 xarthisius joined #dataverse
11:13 pdurbin
I'm glad you're getting back to the automated testing stuff. It's so important. The phoenix server caught a regression this week.
12:45 poikilotherm
Meh Docker Hub does not allow dashes in usernames :-/
12:52 poikilotherm
pdurbin could you grant permission for dataversebot to the repos?
12:52 poikilotherm
(On docker hub)
12:59 donsizemore joined #dataverse
12:59 poikilotherm
Morning donsizemore :-)
13:01 donsizemore
@poikilotherm morning =) i was going to offer to tick off dataverse-kubernetes #11 if i won't step on your toes
13:01 donsizemore
also, i suppose it's no longer morning there, but thought that counts!
13:02 donsizemore
^^ the dataverse-java-client job fails because i didn't want to pop any credentials into jenkins IIRC
13:02 poikilotherm
By ticking it off you mean you would work on the refactoring in upstream?
13:03 poikilotherm
There is an upstream ticket for this, too: https://github.com/IQSS/dataverse/issues/5733
13:04 donsizemore
i was thinking i'd just grab the XML and drop it into place? i'll hold off pending refactoring
13:06 poikilotherm
I'm buzzled... Grab the XML and drop it? You mean add a workaround?
13:06 donsizemore
i think phil just wanted the job config documented
13:07 poikilotherm
Are talking about the same things here?
13:07 poikilotherm
+we
13:07 donsizemore
p.s. i just noted some of those temperatures in your "heat" wave... it gets hotter than that here =)
13:07 donsizemore
we may not be
13:07 poikilotherm
I thought you are talking about https://github.com/IQSS/dataverse-kubernetes/issues/11
13:07 donsizemore
i mean https://github.com/IQSS/dataverse-jenkins/issues/11
13:08 donsizemore
sorry, only on my 4th cup of coffee here
13:08 poikilotherm
Ah ah ah! Now that makes sense :-D
13:09 pdurbin_m joined #dataverse
13:09 poikilotherm
Feel free :-) That job should be good as is. It will need some additions to the global config and list of plugins
13:09 poikilotherm
, too
13:09 poikilotherm
Otherwise it would not be redeployable instantly
13:09 pdurbin_m
donsizemore: you know what I want. :)
13:10 poikilotherm
I'm going to add an enchantment to the card deck "Know what Phil wants"
13:10 pdurbin_m
heh
13:11 pdurbin_m
I want reproducible builds.
13:11 poikilotherm
I'll take that as a cite on the card :-D
13:12 poikilotherm
Ok, jokes aside. donsizemore feel free to fire at will
13:12 poikilotherm
Be warned though, that I already am working on more jos
13:12 poikilotherm
jobs
13:12 pdurbin_m
poikilotherm: I'm assuming you have an account on the Jenkins server. I don't. On purpose. :)
13:12 poikilotherm
Maybe it might be less work to work on the hole bunch in one go
13:13 poikilotherm
pdurbin_m: Oh! Is that so? Why not?
13:13 donsizemore
i can update them any time
13:13 poikilotherm
As you prefer
13:13 donsizemore
hey @pdurbin any way i can help debug https://dataverse.harvard.edu/dataverse/ajps
13:14 donsizemore
@pdurbin_m i made @poikilotherm an admin of the jenkins instance hoping i could learn something via osmosis
13:15 poikilotherm
donsizemore I was going to ask you about how much steroids the VM has where poor Mr. Robot is going to break stones
13:15 pdurbin_m
donsizemore: yes! Please help!!
13:15 poikilotherm
But that can wait after debugging the 500
13:15 donsizemore
@poikilotherm it currently has 4 cores and 8GB of RAM to match phil's setup at harvard, but i can give it more oomph if need be
13:16 donsizemore
@pdurbin_m any chance i can get in your splunk or unprivileged AWS access or...?
13:16 poikilotherm
It felt kinda clumsy building the image. But that might be related to stdout capturing
13:16 poikilotherm
I don't want to overload ;-)
13:17 donsizemore
if it fills up we give it more
13:17 poikilotherm
Alright
13:17 poikilotherm
What about disk space?
13:17 poikilotherm
Images tend to get leave a footprint...
13:18 poikilotherm
-get
13:18 pdurbin_m
donsizemore: any access you want
13:18 donsizemore
i gave it 30G for starters but that was before docker/kubernetes came on the picture
13:19 poikilotherm
Yeah, that might be a bit... tight.
13:19 donsizemore
whatever can let me read server.log? i can use mandy's api key if we want to try to retrieve info that way
13:19 donsizemore
@poikilotherm how much would you suggest?
13:19 poikilotherm
Lemme check my docker daemon
13:20 pdurbin_m
donsizemore: could you please start by emailing Danny?
13:21 poikilotherm
Ok local images ~10GB, but lots of reclaimable. On my minikube ephemeral vm ~10G, but only 50% reclaimable
13:22 poikilotherm
So maybe try with 100GB and put monitoring in place?
13:23 poikilotherm
Reclaimable is a dangerous thing... Caches speeding up the build are also falling under "reclaimable"...
13:23 pdurbin_m
I'm walking kids to camp.
13:24 donsizemore
@pdurbin_m done
13:25 donsizemore
@poikilotherm i'll give it 100GB once we get AJPS sorted
13:25 poikilotherm
Sure, no problem. Its not urgent
13:26 poikilotherm
pdurbin: I am outta here in about 30 minutes. Could be so kind to add DockerHub user dataversebot to the IQSS repos for dataverse-k8s and solr-k8s later? Then I have a chance to test things this weekend / Monday morning
13:28 pdurbin
donsizemore: thanks! People on Twitter are noticing the instability: https://twitter.com/ronpatz/status/1144494633998639109 :(
13:29 pdurbin
poikilotherm: can you please rename the GitHub account to dataversebot (no dash) to match?
13:30 poikilotherm
Is that possible?
13:31 pdurbin
or make a new account
13:31 poikilotherm
Done :-D
13:31 poikilotherm
https://github.com/dataversebot
13:36 pdurbin
poikilotherm: thanks! And I made a note about your request: https://trello.com/c/e6SHgTnc/3858-could-be-so-kind-to-add-dockerhub-user-dataversebot-to-the-iqss-repos-for-dataverse-k8s-and-solr-k8s-later
13:36 poikilotherm
Thx :-)
13:37 xarthisius
pdurbin: r2d's Zenodo integration? Yeah I am. What about it?
13:44 pdurbin_m joined #dataverse
13:44 pdurbin_m
xarthisius: how do you feel about adding a dataverse.py? :)
13:45 xarthisius
I thought I did :) In a slightly different project though ;)
13:47 pdurbin_m
xarthisius: right! That's the point! You already wrote the code. I've been thinking it might be somewhat straightforward for you to copy and paste it into dataverse.py.
13:50 pdurbin_m
xarthisius: I would be happy to create the issue in repo2docker. :)
13:50 xarthisius
some of that code for sure, but there are subtle differences
13:51 xarthisius
e.g. we maintain state, so that we can grab location of dataverses once. In r2d that would need happen in `detect`
13:52 xarthisius
which means you'd get a request to dataverse installations-json every time someone build a binder...
13:52 xarthisius
if you'd port our code directly that is
14:05 pdurbin_m
xarthisius: if I find someone who's willing to Port the code are you able to help answer questions?
14:07 xarthisius
yes, of course
14:08 pdurbin_m
xarthisius: thank you!
14:09 pdurbin_m
donsizemore: do you think Andre would be interested?
14:13 pdurbin_m
Andrey*
14:20 pdurbin_m
xarthisius: oh, can you please link me to the code to port over?
14:32 xarthisius
pdurbin_m: https://github.com/whole-tale/girder_wholetale/blob/master/server/lib/dataverse/provider.py
14:33 xarthisius
these are Dataverse specific bits, but some of the functionality is general and defined in a class that this specific provider is derived from
14:35 pdurbin_m
xarthisius: thanks!
14:41 poikilotherm joined #dataverse
14:41 xarthisius
pdurbin_m: is there a way of knowing that a url points to a Dataverse installation (except for checking for a match in installations-json which we do now), or is there a certain range of DOIs that are issued solely for Dataverses?
14:47 donsizemore
@pdurbin_m today is andrey's last day
15:06 pdurbin
donsizemore: bah. Please tell him I really enjoyed his CoReRe presentation!
15:07 pdurbin
xarthisius: I would suggest checking URLs like this: https://demo.dataverse.org/api/info/version
15:09 pdurbin
It should return something like this: {"status":"OK","data":{"version":"4.14","build":"1326-57a392d"}}
15:20 donsizemore
@pdurbin you may now CoReRe all you like https://core2test.irss.unc.edu/
15:21 donsizemore
(but you have to roll your neck when you say it)
15:40 pdurbin
Heh. "Co Ray Ray!" But I'm getting curl: (7) Failed to connect to core2test.irss.unc.edu port 443: Connection refused
15:45 pdurbin
Out of curiousity, what do you have http://guides.dataverse.org/en/4.15/installation/config.html#zipdownloadlimit set to?
15:45 pdurbin
andrewSC bricas_ donsizemore juancorr ^^
15:46 donsizemore
oh, i need to open the firewall for you all.
15:46 donsizemore
"Setting :ZipDownloadLimit not found"
15:46 pdurbin
curiosity*
15:47 pdurbin
ok, so the default, which is 100 MB if we can trust the docs
15:48 pdurbin
donsizemore: do people ever complain that the limit is too low?
16:51 andrewSC joined #dataverse
16:55 bricas_
pdurbin: i see the new note on the dataverse release page on github about pg 9.6 -- does this mean that requirement will not change, or is it still under consideration?
16:58 donsizemore
@pdurbin i don't know that we've run into it. let me ask mandy and cheryl
17:01 pdurbin
bricas_: well, it's kind of too late for 4.15. PostgreSQL 9.6 is required because of the synax we're using. It would be nice to be more flexible. Not long ago Dataverse ran fine on PostgreSQL 8.4. Maybe 4-6 months ago.
17:01 pdurbin
donsizemore: ok, just curious. Thanks.
17:02 bricas_
pdurbin: i understand 4.15 has been minted, but can 4.15.1 use <9.6 syntax and i can upgrade straight to that?
17:03 pdurbin
bricas_: well, I don't want to say no. It's more like I don't know. All these SQL upgrade scripts we're writing are now being tracked in the "flyway_schema_history" table of your database: http://phoenix.dataverse.org/schemaspy/latest/tables/flyway_schema_history.html
17:04 pdurbin
... and I don't really know what it means to try to edit a script that's already in that table from the past.
17:05 pdurbin
We're still getting used to Flyway and it's a little painful to be honest. We haven't studied it very closely. The goal was to make upgrades for sysadmins easier. No more having to run SQL scripts on the side. They get run automatically on deployment.
17:07 bricas_
understood. just wondering what to tell others, like, we're stuck on 5.14 until we upgrade pg, which means a non-base apt repo... or, maybe it'll work in a future release if we just wait?
17:15 pdurbin
Honestly, I would suggest upgrading Postgres, even if it means a non-base apt repo. That's what we all do on CentOS. You could try Posgres 10, maybe with donsizemore's blessing. :)
17:15 pdurbin
donsizemore ran Shib 3 before anyone else. He's our pioneer.
17:15 donsizemore
@bricas_ i'd want to run whatever harvard's running...
17:17 pdurbin
Harvard Dataverse uses Amazon's "postgres as a service" so our upgrades are tied to that. I think we had to get off postgres 9.3 recently because it was going away.
17:18 donsizemore
@pdurbin it didn't go away, it just stopped receiving security updates. we're migrating to our new Dataverse digs (and Postgres-9.6) Real Soon Now
17:18 pdurbin
"Amazon RDS will be discontinuing support for version 9.3 of the PostgreSQL database in RDS, starting on September 5th, 2018." https://github.com/IQSS/dataverse/issues/4709
17:18 pdurbin
donsizemore: are you on RDS too?
17:19 donsizemore
no, we're running 9.3 from the postgres repo
17:19 pdurbin
ok
17:19 donsizemore
(it was the newest stable version when we migrated)
17:19 pdurbin
bricas_: I'm not sure if this is helping you. :)
17:19 bricas_
heh
17:19 donsizemore
@bricas_ how big is your DB ?
17:19 bricas_
very small
17:19 pdurbin
I mean, eventually we'll all be off Postgres 9, right? 5 or 10 years from now, if not sooner. :)
17:20 donsizemore
ours is 144MB in postgres' custom dump format. takes about 3 minutes to import to a test box
17:21 bricas_
222K Jun 28 12:00 dvndb.gz
17:21 bricas_
:)
17:22 bricas_
i will say that it is unlikely pg 9.5 will ever be supported so we should plan on 9.6
17:22 donsizemore
should be pretty easy to dump and reload as a test?
17:22 bricas_
typically custom repos aren't discouraged, but we have been bit before when an os upgrade didn't quite go as planned
17:23 donsizemore
i'd never want to do an in-place upgrade on a production box
17:23 donsizemore
i've pushed our production database around to test machines like a bad fruitcake, and thankfully never encountered import errors
17:24 bricas_
i've been doing super fun in place upgrades of our ubuntu 14.04 boxes since january (14.04->16.04->18.04). got them all done by end of april luckily.
17:25 donsizemore
maybe i got burned enough during the 8.04/10.04/12.04 days that i haven't tried an in-place since then
17:25 pdurbin
I always just spin up a new Digital Ocean droplet for a new Ubuntu LTS for my home stuff. I guess I'm just paranoid from many years of using CentOS.
17:27 bricas_
it kind of a shit-show. the upgrade from 14.04->16.04 will die because they somehow messed up the mysql 5.6 -> 5.7 upgrade which means you have to restore all dbs from disk
17:28 bricas_
anyway, another 4 years until i worry about it again ;)
17:28 pdurbin
:)
17:31 donsizemore
@pdurbin dumb question?
17:32 pdurbin
hit me
17:32 donsizemore
mandy's having trouble publishing up yonder, so i tried via the API per http://guides.dataverse.org/en/latest/api/sword.html#publish-a-dataset
17:33 donsizemore
i cobble together the command, but dataverse.harvard challenges: "Enter host password for user '<api-token>':
17:34 donsizemore
if this is expected, i'll have her come in and enter her password, but...
17:34 pdurbin
no, a password shouldn't be required
17:34 pdurbin
an api token should work
17:34 pdurbin
you could also try the native api: http://guides.dataverse.org/en/4.15/api/native-api.html#publish-a-dataset
17:35 donsizemore
oh. pffft. sword came back from a search
17:35 pdurbin
but that sword version should work too
17:35 pdurbin
clicking publish doesn't work?
17:36 donsizemore
my suspicion is it's datacite but i'll try via native. one sec
17:38 pdurbin
anything different about the dataset?
17:39 donsizemore
native API threw a 404 using the DOI as the ID. i'll ask her to sneaky-sneaky me the database ID
17:40 donsizemore
she said what had happened was "it says that it is updating or registering file identifiers and then a little spinny wheel might pop up, but then it just kicks me back to the already published v1.0 If i go to edit metadata, my draft is still there.... it just won't update to v1.1 with the updated metadata"
17:42 pdurbin
donsizemore: can you please ask her to open a ticket by emailing support
17:43 donsizemore
yis. tried native API with database ID and got a 503, will keep trying
17:44 donsizemore
mine was more of a usage question or to see if you'd say "yeah, dat broke"
17:45 donsizemore
native API throws a 404 with DB ID. support it is
17:50 pdurbin
The APIs should work fine. Clicking "Publish" should work fine too. I don't know what's going on.
17:52 donsizemore
i blame AWS. i got the API to respond "WORK_IN_PROGRESS" but mandy says it's still at 1.0
17:57 pdurbin
I got nuthin. Sorry.
18:04 donsizemore
i think it's trying to tell us to leave early for andrey's happy hour
18:05 pdurbin
probably, and thanks for opening https://help.hmdc.harvard.edu/Ticket/Display.html?id=278072
18:06 donsizemore
Dwayne!
18:09 donsizemore
ok, i'm out. have a great weekend!
19:35 andrewSC joined #dataverse
19:39 pdurbin
I'm out too. Have a good weekend, everyone.
19:39 pdurbin left #dataverse
