Time
S
Nick
Message
07:10 jri joined #dataverse
07:45 poikilotherm joined #dataverse
07:53 poikilotherm
donsizemore: I'm not only scanning daily, but also do use webhooks.
07:53 poikilotherm
They get setup by the plugin (at least have been), as I use the dataversebot user, who has proper rights in dataverse-kubernetes project
07:54 poikilotherm
pdurbin: that reminds me - we've been talking about handing that user over....
07:54 poikilotherm
I'm fine with running and administrating it if you want me to do so
07:54 poikilotherm
Whatever works for you guy
07:54 poikilotherm
+s
08:20 jri joined #dataverse
10:10 poikilotherm
Mornin' pdurbin - r u around already=
10:10 poikilotherm
?
10:19 pdurbin
poikilotherm: haven't had coffee yet
10:19 poikilotherm
Ok, need your help later. But: coffee first!
10:20 pdurbin
yes, always coffee first
10:33 pdurbin
poikilotherm: you said you need some help?
10:34 poikilotherm
Right.
10:34 poikilotherm
I looked everywhere, but I can't remember the AWS console credentials
10:34 poikilotherm
IIRC you guys send me a prebuilt account
10:34 poikilotherm
But I can't find it :-(
10:35 poikilotherm
Uuuups
10:35 poikilotherm
False alert.
10:35 poikilotherm
I put it in my password sage
10:35 poikilotherm
safe
10:35 poikilotherm
Sorry for all that noise early in the morning :-(
10:36 pdurbin
you woke me up for this? kidding, kidding, I turn off all notifications :)
10:36 poikilotherm
:-D
10:37 poikilotherm
I told myself not to ask before 12:00, as I know you catch the early birds
10:39 pdurbin
It's all good. And yes, you should probably send credentials for dataversebot some day. Not a hand off but we should both be able to log in. Because bus factor.
10:39 poikilotherm
Exactly
10:39 poikilotherm
Are you ok with opening a KeePass file?
10:40 poikilotherm
I could send you both bot accounts on Dockerhub and Github
10:41 pdurbin
Is KeePass open source?
10:41 poikilotherm
Aye
10:41 poikilotherm
https://keepassxc.org/
10:41 poikilotherm
I really enjoy that cross platform port
10:42 poikilotherm
The original version from Mr. Schneier is OSS IIRC, too
10:42 poikilotherm
Yes: "KeePass is a free open source password manager", see keepass.info
10:43 pdurbin
This looks good but let me get to work before I try it.
10:44 poikilotherm
Sure
10:44 poikilotherm
Dunno what you guys use for password management
10:44 poikilotherm
Happy with whatever you are
10:45 pdurbin
We don't have a good solution. Could I put a private key for an HTTPS cert in thins thing? For https://demo.dataverse.org for example?
10:45 pdurbin
this*
10:45 poikilotherm
Sure.
10:45 poikilotherm
Please be aware that this is a file based thing
10:46 pdurbin
Great, because it was down the other day and I spun up a new instance on EC2 but I didn't have the private key. I still don't, actually.
10:46 poikilotherm
It suffers from the usual file based database hazards
10:46 poikilotherm
You are aware of AWS KMS, aren't you?
10:46 pdurbin
nope
10:46 poikilotherm
https://aws.amazon.com/kms/
10:47 pdurbin
I use https://github.com/jamessan/vim-gnupg for some stuff but I wouldn't force that on anyone.
10:48 poikilotherm
I've been using https://teampass.net/ for a while in a larger team
10:48 poikilotherm
We had lots of passwords, but did not want to go with lastpass et al
10:49 pdurbin
I'm fine with whatever. Again, we don't have a good solution right now. Harvard does have a license for lastpass.
10:49 pdurbin
lastpass premium it looks like: https://security.harvard.edu/lastpass
10:50 poikilotherm
IIRC they do have an API , too
10:50 pdurbin
LastPass Personal Premium
10:56 stefankasberger joined #dataverse
11:40 pdurbin
stefankasberger: hi, are you around?
11:47 stefankasberger
yes, but really busy. whats up?
11:50 pdurbin
stefankasberger: no rush but I wanted to let you know that I like your metadata block repo idea and wrote about it. Please take a look at https://groups.google.com/d/msg/dataverse-community/nDMbMv4fKf4/w4Dzqe7lCQAJ
11:56 stefankasberger
+1
12:35 donsizemore joined #dataverse
12:59 Sharif_DE joined #dataverse
13:00 Sharif_DE
Hi All! Greetings from Germany. :)
13:00 Sharif_DE
hello poikilotherm
13:00 Sharif_DE
Hi Phil
13:01 pdurbin
Sharif_DE: hi! Reminder to please create an issue in dataverse-docker. Please see https://github.com/QualitativeDataRepository/dataverse-previewers/issues/22#issuecomment-543917877
13:01 Sharif_DE
pdurbin I will do it right now
13:01 pdurbin
thanks!
13:02 Sharif_DE
u r wc
13:06 Sharif_DE
done!
13:08 Sharif_DE
poikilotherm my SPSS data ingestion is running without any end!!!
13:08 Sharif_DE
What should I do?
13:12 Sharif_DE
poikilotherm pls tell me also how do I use the curl command to use File Previewers?
13:28 jri
Hello, I was reading this beautiful work: https://github.com/IQSS/dataverse-ansible :-)
13:28 jri
Do you use it for production deployments ?
13:29 Sharif_DE
jri we are planning to use it for production
13:29 jri
Oh cool Sharif_DE !
13:30 jri
Can I ask you how do you do to upgrade Dataverse if you've deployed it with ansible ?
13:30 jri
Do you do updates manually ?
13:30 jri
Because the procedure often changes a little
13:31 jri
so I don't get how to automate this
13:31 Sharif_DE
sorry, misunderstood. I am trying right now docker and Kubernetes
13:31 jri
Oh ok :)
13:42 dataverse-user joined #dataverse
13:54 donsizemore
@jri that's why I haven't tackled upgrades via ansible. the process is different with each release.
13:55 jri
Yes :/
13:57 jri
What do think about making a folder "updgrades" and a .yaml for each upgrade ? Like upgrade_2.15.1_to_2.16.yaml , upgrade_2.16_to_2.17.yaml, etc. Is it ugly ?
13:58 jri
Because I thinking about using ansible to install and maintain my production
13:59 donsizemore
@jri I wish there were versioned artifacts with each release (schema.xml in particular)
13:59 donsizemore
@jri the ansible role has been most useful for testing so far
14:00 poikilotherm
donsizemore: I'd be happy to share my scripts and create a common codebase
14:00 poikilotherm
If that helps
14:00 Sharif_DE
poikilotherm my SPSS data ingestion is running without any end!!!
14:00 donsizemore
@poikilotherm that would be wonderful
14:01 poikilotherm
donsizemore: if you would like to take a look, feel free.
14:01 poikilotherm
https://github.com/IQSS/dataverse-kubernetes/tree/master/docker/dataverse-k8s/bin
14:02 poikilotherm
https://github.com/IQSS/dataverse-kubernetes/tree/master/k8s/dataverse/jobs
14:02 poikilotherm
Most jobs simply call a script or directly use the cmdline
14:02 poikilotherm
Creating a common set of scripts/cmdlines whatever could be baked into sth for me to use and you to use in ansible
14:03 poikilotherm
I could even just re-use your ansible
14:03 donsizemore
for vanilla installations that could be very useful
14:03 poikilotherm
Because we could wrap it in a container ;-)
14:04 poikilotherm
Actually going with ansible for this is a clever thing, as it might lead to creating an Operator
14:04 poikilotherm
https://coreos.com/operators/
14:09 Sharif_DE
poikilotherm pls help
14:11 poikilotherm
Hey Sharif_DE, I'm pretty busy today. All of this sounds like a dataverse issue, unlikely related to dataverse-kubernetes. So maybe pdurbin can give you a hand, too
14:11 donsizemore
@poikilotherm so on jenkins - i can port the job into a jenkinsfile (fine) but a multi-branch jenkins job tries to index every(!) branch (and fails)
14:11 donsizemore
@Sharif_DE R ingest processes each file line by line, and can take quite a while
14:12 poikilotherm
Oh yeah, that might be a problem! I totally forgot about the sheer number of branches...
14:12 Sharif_DE
@poikilotherm Ok, thanks. i will ask him
14:12 poikilotherm
Maybe pdurbin can tell us why these branches never get deleted once they are merged
14:12 donsizemore
@poikilotherm i've got everything working except the webhook trigger
14:13 Sharif_DE
donsizemore, it is running now for 7 hours
14:13 donsizemore
which seems to be a very common problem in the GHPR plugin
14:13 poikilotherm
Hmm is dataversebot an admin for the repo?
14:13 donsizemore
@Sharif_DE how big is the file? how many rows?
14:13 poikilotherm
I had no problems at all with setup :-/
14:13 Sharif_DE
45 MB
14:13 donsizemore
@Sharif_DE yeah... that'll take a while.
14:13 poikilotherm
Did you create the webhook yourself?
14:13 donsizemore
@poikilotherm I created the webhook myself, that part is fine
14:14 donsizemore
I'm getting the dread "but there aren't any new comments nor commits; that may mean that commit status was updated." bug
14:14 donsizemore
jenkins sees the pull requests (and updates to said) arrive; it just doesn't launch the build and it seems to be an unresolved bug in the plugin =(
14:15 donsizemore
what they really seem to want is a graphical, manual way to spin up EC2 when there's a PR; @pdurbin mentioned writing a front-end for that
14:15 poikilotherm
let me take a quick look
14:15 poikilotherm
I had problems when I did not specify a strategy
14:16 poikilotherm
Err... what job?
14:16 poikilotherm
-PullRequest?
14:17 pdurbin
How can we make Dataverse upgrades easier?
14:17 donsizemore
@pdurbin standardization
14:17 donsizemore
@poikilotherm correct
14:17 poikilotherm
pdurbin: get rid of eclipse link creating a new set of tables :-D
14:18 pdurbin
Ok, did adding Flyway help? Or is it worse now?
14:18 poikilotherm
Flyway is great, it's just not used 100% the way it should be :-D
14:18 pdurbin
oh, whoops
14:18 poikilotherm
There is that stale issue from a while ago ;-)
14:18 pdurbin
Is there a GitHub issue about how we are using it wrong?
14:19 poikilotherm
https://github.com/IQSS/dataverse/issues/5871
14:20 pdurbin
poikilotherm: thanks, can I put that issue on your to do list on my board?
14:20 poikilotherm
No :-D At least not for now ;-)
14:20 poikilotherm
I'm already swamped
14:20 Sharif_DE
pdurbin is it normal that a 45 MB SPSS file ingestion taking more than 7 hours...still not done!?
14:20 pdurbin
poikilotherm: not even at the bottom? :)
14:21 poikilotherm
pdurbin: https://i.imgur.com/Afg14CF.png
14:22 pdurbin
poikilotherm: what am I looking at? New Sphinx-based guides for dataverse-kubernetes?
14:22 poikilotherm
Yes Sir
14:22 pdurbin
Sharif_DE: is the possible to make the file public? Or does it have private data in it?
14:23 Sharif_DE
pdurbin private
14:23 Sharif_DE
:(
14:23 poikilotherm
donsizemore have you seen the hint that it might be a good idea to switch from github pull request plugin in to https://wiki.jenkins.io/display/JENKINS/GitHub+Branch+Source+Plugin
14:23 poikilotherm
Seems like it is developed much more actively
14:23 donsizemore
@poikilotherm yes, have you looked at the docs for it?
14:24 donsizemore
@poikilotherm it's been abandoned and the docs are a 404
14:25 pdurbin
Sharif_DE: :(
14:25 poikilotherm
donsizemore: https://docs.cloudbees.com/docs/admin-resources/latest/plugins/github-branch-source
14:25 pdurbin
Sharif_DE: can you please try to reproduce the problem with a public file?
14:26 poikilotherm
donsizemore: actually I think that I use the new plugin under the hood
14:26 Sharif_DE
I used another spss public file...the ingestion worked smoothly
14:27 poikilotherm
I don't configure it directly, as I use the multibranch plugin
14:28 poikilotherm
donsizemore: by the way: you can configure multibranch projects with strategies what to discover
14:29 poikilotherm
so you should be able to restrict it to PRs and maybe named branches like master/develop
14:29 poikilotherm
let me create a test element
14:32 poikilotherm
donsizemore: this doesn't look to bad... https://jenkins.dataverse.org/job/IQSS-PR-test/
14:32 poikilotherm
Seems to need some tweaks, but might be ok
14:33 poikilotherm
dataversebot definitly needs more permissions :-D
14:39 Sharif_DE left #dataverse
14:40 donsizemore joined #dataverse
14:40 poikilotherm
Hey donsizemore, did you get my last messages
14:41 donsizemore
@poikilotherm hello. i was doing what you did in PR TEST...
14:41 donsizemore
@poikilotherm I'll port my stuff into a Jenkinsfile and see if they behave the same way as a job
14:42 poikilotherm
:-)
14:42 poikilotherm
Just ask if you need help
14:42 donsizemore
@poikilotherm but i'm trying to do four things at once today, so this may have to wait
14:42 poikilotherm
pdurbin might take a look at giving dataversebot the appropriate rights
14:43 poikilotherm
Aye
14:43 poikilotherm
I feel you :-D
14:43 poikilotherm
Maybe https://github.com/jenkinsci/pipeline-examples can provide some usefull examples
14:44 poikilotherm
Gotta run now, grab some kids ;-)
14:44 poikilotherm
Cu guys around
16:56 donsizemore joined #dataverse
17:42 donsizemore joined #dataverse
19:16 poikilotherm joined #dataverse
19:18 poikilotherm
Hey guys, how was community call?
19:23 pdurbin
poikilotherm: great! I sung your praises!
19:24 poikilotherm
My praises?
19:24 poikilotherm
What praises?
19:24 poikilotherm
So many of 'em :D
19:24 pdurbin
I talked about hooks and https://github.com/IQSS/dataverse-kubernetes/issues/68
19:25 poikilotherm
Hrhr
19:25 poikilotherm
Verrrrrry good
19:25 poikilotherm
Thank you for singing my song :-)
19:26 poikilotherm
I just tweeted you in a PM about that Jenkins job...
19:29 donsizemore
@poikilotherm you can take a look at https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/configure if you'd like
19:29 * poikilotherm
goes looking
19:29 poikilotherm
(but pdurbin's job is about dataverse-installations)
19:30 poikilotherm
donsizemore: that looks nice, but do you really want to discover branches?
19:31 poikilotherm
I know, you configured to discover only branches that are part of a pr
19:31 poikilotherm
But shouldn't "discover pr from origin" be sufficient?
19:32 pdurbin
poikilotherm: I don't understand your DM but that's probably because my head is in Jim's Make Data Count pull request, which I just hacked on.
19:32 poikilotherm
It might be nit picking, but....
19:33 poikilotherm
pdurbin: I try to collect enough merged PRs to gain that shirt from Hacktoberfest ;-)
19:33 poikilotherm
But obviously I don't want to waste time or do things that spark no joy
19:34 pdurbin
you want a t-shirt?
19:34 poikilotherm
Sure :-)
19:34 pdurbin
ok
19:34 pdurbin
I'm looking for someone to edit a couple google spreadsheets and run a python script.
19:35 pdurbin
and then make a pull request
19:35 pdurbin
poikilotherm: will that help you get a t-shirt?
19:36 poikilotherm
Sure. But that's a really easy task, that might attrack someone who's is not yet part of the community to engage
19:37 poikilotherm
attract
19:37 pdurbin
poikilotherm: how about the README? Do you want to work on https://github.com/IQSS/dataverse-installations/issues/1 ?
19:38 poikilotherm
I don't want to take that away from someone else, looking for easier tasks to achieve that shirt
19:38 poikilotherm
Isn't that part of the other issue?
19:38 pdurbin
The two issues are related.
19:39 pdurbin
And no one has volunteered for either yet.
19:39 poikilotherm
Ok I'll keep it in mind. When no one works on it till 10/29 I'm going to do it :-D
19:40 pdurbin
ok, thanks
19:40 poikilotherm
That way others do have a chance
19:40 pdurbin
maybe pmauduit will do it :)
19:40 pdurbin
or bjonnh
19:40 poikilotherm
Maybe he wants that shirt
19:40 poikilotherm
He
19:41 poikilotherm
Maybe some of the other new aspirants might be interested, too?
19:41 poikilotherm
Thinking about Sharif_DE, thanhthanh etc
19:41 pdurbin
Hard to say. I think you should go ahead. :)
19:42 poikilotherm
BTW: are you aware of sphinx-autobuild?
19:45 poikilotherm
donsizemore: you might test using "trust = contributors" with IQSS/dataverse. It *might* be the case that dataversebot doesn't need much more rights that way
19:45 donsizemore
I thought I had that set
19:46 poikilotherm
donsizemore: the Github pull request builder plugin behaviour of commenting and rules triggering seems to be replaceable with the pipeline-github plugin, to be used in pipelines
19:47 poikilotherm
Since you send me that job link it had been set to trust=admin
19:47 pdurbin
donsizemore: if you need anything, please let me know. You said you had four things going at once so I'm trying not to bother you. :)
19:47 donsizemore
fixed. i had that set in a previous iteration
19:48 donsizemore
well, i think i have everything in the jenkinsfile and it says it's queuing the build but then...
19:49 poikilotherm
donsizemore: I just looked at the log output for the job - it said sth about no permission. you use dataversebot to access odum/dataverse. is that by design?
19:49 poikilotherm
You could add dataversebot to your orgs to enable it
19:49 poikilotherm
;-)
19:49 poikilotherm
dataversebot is a very friendly guy
19:49 donsizemore
oh, i tried that yesterday, github couldn't/wouldn't find dataversebot
19:50 poikilotherm
Well I see https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/view/change-requests/job/PR-18/lastBuild/console failing...
19:50 poikilotherm
And currently the job has been configured to use dataversebot
19:50 poikilotherm
I saw in the list that you added your own credentials too
19:53 poikilotherm
dataversebot told me to tell you he is very excited to join you guys at odum institute
19:54 poikilotherm
I dunno if he really needs admin access
19:54 donsizemore
I can always take it away
19:54 donsizemore
IQSS is the real target, I'm just aiming at OdumInstitute first
19:54 poikilotherm
:-)
20:03 donsizemore
i think it's happening https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/view/change-requests/job/PR-18/4/console
20:10 poikilotherm
Tschakka!
20:11 poikilotherm
+ ./ec2-create-instance.sh -b develop -r https://github.com/IQSS/dataverse.git -t jenkins_delete_me -l target -g groupvars.yml
20:11 poikilotherm
/var/lib/jenkins/workspace/IQSS-Dataverse-Develop-PR_PR-18
20:11 poikilotherm
Oh oh
20:11 donsizemore
re-running now
20:12 donsizemore
and i'm going to need to stash jenkins' ansible group_vars somewhere. as a traditional job it just sat in the workspace
20:12 donsizemore
but https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/view/change-requests/job/PR-18/5/console
20:12 poikilotherm
Nice
20:13 poikilotherm
I'm curious what will take longer - running the EC2 script or use Docker/K8s
20:14 poikilotherm
Sth. for later: I think it would be cool to have different steps for building and the unit tests
20:15 poikilotherm
you might even take a look at doing unit and integration tests in a parallel step ;-)
20:17 poikilotherm
Oh and you might be interested in https://jenkins.io/doc/book/pipeline/syntax/#post
20:20 donsizemore
i had a "post { always" in there but jenkins got mad
20:34 bjonnh
pdurbin: I'm deep into spring boot currently
20:34 pdurbin
bjonnh: do you like it?
20:35 bjonnh
I'm using it with Kotlin, it is pretty agreable
20:35 pdurbin
cool
20:35 bjonnh
no xml involved
20:35 bjonnh
I'm missing some of the easy stuff that django did bring
20:35 bjonnh
but on the other hand, it is way more robust
20:36 pdurbin
bjonnh: if you like Django, someday we should get you hacking on https://github.com/IQSS/geoconnect :)
20:37 bjonnh
I just glanced over it
20:37 bjonnh
I still have a project to port to the last django
20:37 bjonnh
it is getting there
20:37 bjonnh
that wasn't fun
20:38 pdurbin
I can imagine. :)
21:49 poikilotherm joined #dataverse
21:49 poikilotherm
pdurbin: very early stage: https://iqss.github.io/dataverse-kubernetes/
21:49 poikilotherm
I have more files waiting on my laptop, but I will need to refactor first
21:49 poikilotherm
The image on the page is missing by purpose - has not yet been merged into master... ;-)
21:52 pdurbin
poikilotherm: looks great!
21:52 pdurbin
Still no takers on updating the map. I'm heading home.
21:53 poikilotherm
I'm heading for my bed :-D
21:53 poikilotherm
Sleep well
22:17 rigelk joined #dataverse
22:35 rigelk joined #dataverse
22:56 pdurbin joined #dataverse
23:31 icarito[m] joined #dataverse
