Time
S
Nick
Message
05:49 juancorr joined #dataverse
06:43 poikilotherm joined #dataverse
07:17 pdurbin joined #dataverse
07:24 poikilotherm joined #dataverse
07:33 poikilotherm joined #dataverse
08:16 stefankasberger joined #dataverse
08:59 jri joined #dataverse
10:52 pdurbin
poikilotherm: hi. This afternoon we will be discussing https://github.com/IQSS/dataverse/issues/5478 so if you have any opinions on if https://github.com/IQSS/dataverse/pull/5680 should be merged or not, please leave a comment on the issue.
10:58 poikilotherm
Mornin' pdurbin
10:58 poikilotherm
Done
10:58 poikilotherm
https://github.com/IQSS/dataverse/issues/5478#issuecomment-481204230
10:59 pdurbin
gave it a thumbs up, thanks
11:01 pdurbin
Does https://github.com/IQSS/dataverse-kubernetes have anything that could be built or tested from https://jenkins.dataverse.org ?
11:01 poikilotherm
Sure. The images could be build by Jenkins instead of Docker Hub
11:01 poikilotherm
See e.g. https://github.com/IQSS/dataverse-kubernetes/issues/22
11:02 pdurbin
Docker Hub builds images? I've always built them on my laptop.
11:02 poikilotherm
Yarp. Currently using them for k8s ;-)
11:02 poikilotherm
Remember connecting to Github?
11:03 poikilotherm
That was because of the necessary link between them for autobuilds
11:04 pdurbin
Thanks, I left a comment.
11:04 pdurbin
Are you running the API test suite at all?
11:05 poikilotherm
Nope.
11:05 pdurbin
Too busy or not interested?
11:05 poikilotherm
This is business of upstream :-D
11:05 poikilotherm
At least for dataverse-kubernetes
11:05 pdurbin
Hmm. Ok.
11:05 poikilotherm
It might be a good idea to use things like serverspec to test my own code
11:06 pdurbin
never heard of it but sure
11:06 poikilotherm
As the images currently are only usable for released versions, you can't use them for commit based API tests
11:07 poikilotherm
This might be a good idea to attack later on ;-)
11:07 poikilotherm
You remember my wishes for integration testing, right?
11:07 poikilotherm
Having k8s around makes this entirely possible
11:08 poikilotherm
(Of course this is possible with smaller guns than K8s, but on CI this is easy to use)
11:16 pdurbin
Sure. I mean, the reason that new Jenkins server exists is that on Friday Don an I were chatting and he was saying that we could run the "ec2 spin up" script from Jenkins. Which I've thought about for a long time but I have very little control over the installation of Jenkins at IQSS/HMDC.
11:17 pdurbin
I guess I have very little control over https://jenkins.dataverse.org at the moment but I'm confident that the configs will be opened so I can rebuild it from scratch some day if I feel the need to. At least I have control over DNS and certs. :)
11:17 poikilotherm
Yeah
11:18 pdurbin
Anyway, yes the ec2 spin up scripts allow us to spin up arbitrary branches. Thank you, Don!
11:18 pdurbin
I'm still eager to figure out how we get it all working from Jenkins, running the API test suite, I mean.
11:19 poikilotherm
EC2 sounds promising but expensive
11:19 pdurbin
I basically want this: https://build.hmdc.harvard.edu:8443/job/phoenix.dataverse.org-apitest-develop/ ... with a Test Result Trend and all.
11:19 pdurbin
Well, I don't think we should build every commit.
11:20 poikilotherm
But...?
11:21 pdurbin
What if you make a pull request for the main code base and could ask us to run the API test suite before asking for code review?
11:21 poikilotherm
Err... You should not need to ask.
11:21 poikilotherm
PRs get a build and API test by default
11:21 poikilotherm
Rerun on new commits added to them
11:22 pdurbin
Hmm, I'd have to change the timing of my pull requests.
11:22 poikilotherm
Why?
11:22 poikilotherm
Failing builds from PRs don't hurt. Actually that is what CI is all about, right?
11:23 poikilotherm
Get feedback (ideally fast) and early in your development
11:23 pdurbin
I like to make pull requests early. An example is https://github.com/IQSS/dataverse/pull/5708 . I'm making assertions around code I'm about to change. I'm still hacking on it. I wouldn't want to spend the money to test that branch yet.
11:24 poikilotherm
Maybe it would be better then not to use EC2 for this... ;-) Actually the API tests or other simple maven based integration tests could be done in other ways.
11:24 pdurbin
other ways?
11:24 poikilotherm
But as you said: one step after the other
11:25 poikilotherm
Yeah. Docker or Kubernetes. For simple tests, you actually only need 3 containers: psql, solr and dataverse
11:26 poikilotherm
Everything else needing external deps can be either stubbed in low level tests and run on EC2 or K8s with full stack later on
11:27 pdurbin
sure, docker-aio has only those three containers
11:27 poikilotherm
Like testing the S3 storage layer with a full stack is a good idea, but can be mocked for business logic first
11:27 poikilotherm
That doesn't even need a running Glassfish or similar
11:28 poikilotherm
But all of this full stack stuff is way ahead of everything one should start with
11:28 pdurbin
So should we run docker-aio from Jenkins?
11:29 poikilotherm
Personally I don't think docker-aio is a good approach for the increased complexity Dataverse has nowadays. It had been designed for a narrowed scope, which is perfectly fine. But IMHO this should be changed.
11:30 donsizemore joined #dataverse
11:32 jri joined #dataverse
11:32 poikilotherm
donsizemore: morning! I had a look at your logs and I have absolutely no clue, what happened. There is no error message I don't have when having a successfull deployment
11:33 poikilotherm
It just all of a sudden seems to recieve a SIGTERM
11:33 poikilotherm
Just before the EclipseLink persistance layer kicks in
11:33 donsizemore
@poikilotherm it's glassfish =)
11:34 donsizemore
@pdurbin i've been meaning to offer i'm happy to make you an admin of our jenkins server
11:34 poikilotherm
I have repeated the deployment of dataverse-k8s on a few different boxes and kubes with no problem at all
11:35 poikilotherm
I just can ask you to try again with a clean Minikube, maybe on another machine. Maybe also try to use a different Hypervisor.
11:35 pdurbin
donsizemore: meh, I don't think I need to be an admin right now. Mostly I'm interested in helping figure out a sane automated testing strategy. It sounds like poikilotherm isn't a fan of docker-aio but I think we should maybe try calling it from Jenkins.
11:39 pdurbin
Basically, I want to have https://github.com/IQSS/dataverse-jenkins start to become a store of knowledge about how to configure automated testing for Dataverse. In executable form, I mean. We have a human readable write up already at http://guides.dataverse.org/en/4.12/developers/testing.html
11:39 pdurbin
Does that make sense?
11:40 poikilotherm
donsizemore IIRC you said once that Jenkins is also able to read sth similar to .gitlab-ci.yaml/.travis.yaml?
11:41 poikilotherm
Maybe that would be a good thing to start adding to the repo and use it?
11:45 pdurbin
Instead of being an admin I want anyone to be able to spin up Dataverse automated testing infrastructure to be their own admin on their own hardware. For this we need docs. How to install Jenkins. How to create jobs from XML or whatever.
11:47 pdurbin
If someone creates dataverse-client-javascript we would add an XML file to dataverse-jenkins that defines how to create a job to test it.
11:48 donsizemore
@pdurbin writing docs is on my list but i'm happy for someone smarter to do it
11:48 donsizemore
also, regarding dataverse-ansible and certs, you're cool with self-signed?
11:49 pdurbin
I can't decide on the certs. I have a private key for dev2 on my laptop. I think I'd like to be able to spin up an ec2 instance and get the private key onto that day's dev2. Somehow.
11:50 pdurbin
donsizemore: I'm happy to write docs for dataverse-jenkins but can you add an XML definition of a job?
11:51 donsizemore
I was going to point potential CI-ers at GeerlingGuy's excellent ansible role, and add the XML from our job
11:56 pdurbin
sounds great, thanks!
12:01 pdurbin
donsizemore: you're ok with me offering that Amber and crew can use the new Jenkins for language packs, right? She replied here: https://groups.google.com/d/msg/dataverse-internationalization-wg/2zSl9xDpH7g/dtV9DY6lBAAJ
12:02 donsizemore
absolutely. just tell me what all you want done
12:03 pdurbin
Sure. Once we have the first XML file for creating a job I'll ask them to create one like it.
12:10 pdurbin
Are you planning on creating a Jenkins job for Akio's TRSA web app?
12:19 jri_ joined #dataverse
12:26 donsizemore
we have one for it, and for his trsa-api Dataverse branch
12:27 donsizemore
I can mirror it over to jenkins.dataverse if you like
12:29 pdurbin
No rush. Whenever. We could add dataverse-metrics if you want. There are a couple tests that I currently run in Travis.
13:07 donsizemore
@pdurbin you're going to kill me.
13:08 donsizemore
I forgot we were running maven with -DskipTests, saw it in the generated config.xml
14:00 juancorr joined #dataverse
14:04 pdurbin
how dare you skip the tests
14:07 donsizemore
tests will run after the next commit
14:08 pdurbin
heh, ok, thanks
14:16 pdurbin
ah and thanks for renaming the test
14:16 pdurbin
er, job
16:00 pdurbin
andrewSC bjonnh bricas_ candy` dzho jri_ juancorr pmauduit xarthisius yoh: the community call is starting in a couple minutes: https://dataverse.org/community-calls
16:45 jri joined #dataverse
17:04 pdurbin
exciting hotfix proposal: https://github.com/IQSS/dataverse/issues/5724#issuecomment-481337554 :)
17:21 donsizemore joined #dataverse
17:22 donsizemore
@pdurbin well, what i'd like to do to the role would allow provided certs such as yours, then default to self-signed (in case of vagrant or some other throwaway VM ). it was the "valid cert" part of the issue that was the sticking point
17:31 pdurbin
I forget what I even wrote. :)
17:32 pdurbin
ah, I didn't write much
17:32 pdurbin
at https://github.com/IQSS/dataverse-ansible/issues/53 for anyone else who's interested
17:32 pdurbin
Shoud I explain my use case?
17:33 pdurbin
For years I've had a server at https://dev1.dataverse.org that is patched by the ops group here. So I leave it up all the time.
17:34 pdurbin
At the moment https://dev2.dataverse.org is on EC2 and not patched by our ops group. It's on me to patch. Plus, it's adding to our Amazon bill. I don't need it up the time. So I'll probably kill it before I go on vacation next week.
17:34 pdurbin
But when I get back, I may spin up dev2 again. And I'll want to put my cert on it, I think.
17:35 pdurbin
And I'll have to repoint DNS to the new hostname from EC2.
17:35 pdurbin
Is this making sense?
17:35 donsizemore
yes
17:47 poikilotherm joined #dataverse
17:47 pdurbin
thanks :)
18:42 pdurbin
Is there anything I can do to help?
19:00 donsizemore
@pdurbin you can guard my office door =) i started the previewer run to check on doubles before i had to run out
19:06 pdurbin
:)
19:13 donsizemore
@pdurbin on my test run there are 13 curl calls in the script and 13 QDR viewers added to the externaltool table
19:35 pdurbin
donsizemore: I run this... curl http://localhost:8080/api/admin/externalTools | jq '.data[].displayName' | tr -d "\n"
19:35 pdurbin
I get this: "Read Text""View Html""Play Audio""Play Audio""View Image""View Image""View Image""View Image""Read Document""Play Video""Play Video""Play Video""View Annotations""Whole Tale"
19:35 pdurbin
so two "Play Video" tools, for example
19:35 donsizemore
@pdurbin there are different contentTypes: audio/ogg vs audio/mp3
19:35 pdurbin
oh!
19:36 donsizemore
it was your saying there were multiple text entries that concerend me
19:41 donsizemore
i think we're good in vagrant and ec2
20:05 pdurbin
great!
21:20 jri joined #dataverse
