Time
S
Nick
Message
00:30 nils`` joined #dataverse
08:07 jri joined #dataverse
08:29 poikilotherm joined #dataverse
10:52 pdurbin
Anybody blocked? Anybody need anything?
10:55 poikilotherm
Goog morning :-)
11:02 pdurbin
mornin'
11:09 pdurbin
poikilotherm: I wasn't quite following you comment yesterday. One of the new tests you added is failing? Can you just take it out? :)
11:13 poikilotherm
Err... It's a unit test. It tests functionality of the state parser...
11:13 poikilotherm
It should not be left out :-D
11:13 poikilotherm
Test business logic, right?
11:17 pdurbin
Sure, I was just thinking that there will naturally be a follow up pull request to add the OIDC feature (and docs) so we could defer adding it until then. :)
11:21 poikilotherm
That doesn't feel right for me... I hate PRs that are not complete
11:21 poikilotherm
I know I know, seeking perfection...
11:22 pdurbin
Don't let the perfect be the enemy of the good.
11:24 poikilotherm
Right. But failing tests are bad. And using System.currentTimeMillis() is bad practice since Java 8, too
11:24 poikilotherm
And who remembers to fix that unit test once we are done?
11:25 poikilotherm
I tend to forget such things ;-)
11:25 pdurbin
You could add // TODO or // FIXME
11:25 poikilotherm
Yeah.
11:25 poikilotherm
I could
11:25 poikilotherm
:-D
11:26 pdurbin
I'm just thinking there isn't much in QA right now.
11:26 poikilotherm
Have you seen all these TODOs and FIXMEs all over the place?
11:26 pdurbin
Yeah, I added most of them, probably. :)
11:26 poikilotherm
They tend to be left alone for a few years ;-)
11:26 pdurbin
yeah
11:27 poikilotherm
Maybe we should add another metric in jenkins: count "todos"
11:27 poikilotherm
And use that as a badge
11:27 pdurbin
heh, sure
11:28 poikilotherm
IntelliJ found 945 TODO items in the sources...
11:29 pdurbin
I think that's fine if it means stuff gets to QA faster.
11:30 poikilotherm
Sure
11:30 poikilotherm
Keep it small, scoped and simple
11:30 poikilotherm
But at some point in time someone should take a look at the todos ;-)
11:31 poikilotherm
Of course most of the time you are not going to earn fame for those...
11:31 poikilotherm
It's always a tradeoff or balancing
11:55 pdurbin
Yeah. That reminds me, have I talked about https://github.com/publiclab/community-toolbox ?
11:56 jri joined #dataverse
11:56 poikilotherm
Nope, not yet
11:57 poikilotherm
looks really nice :-)
11:57 pdurbin
A friend of mine made it and runs an instance at https://code.publiclab.org . What do you think?
11:58 poikilotherm
+1 :-)
11:59 pdurbin
Note how you can select between different repositories like this: https://code.publiclab.org/#r=plots2
11:59 pdurbin
poikilotherm: if I install this, would you want dataverse-kubernetes to be included?
12:00 poikilotherm
Oh there are a lots of projects to be included... No idea how to do that in a way so its easy for first timers
12:01 pdurbin
Well, a typo in some code would be a good First-Timers Only (FTO) issue. :)
12:02 pdurbin
Here's an excellent writeup of how Public Lab (my friend's non-profit) asks new contributors to create FTO issues for the next new contibutors: https://publiclab.org/notes/sashadev-sky/05-16-2019/public-lab-s-contribution-framework-deep-dive-2019
12:20 tai joined #dataverse
12:26 pdurbin
poikilotherm: oh, I meant to ask you. Do you feel like creating an issue about having multiple maps in https://github.com/IQSS/dataverse-installations ? An a proposal for a namespace like we talked about? Or should I create it? :)
12:27 poikilotherm
Go ahead... I have my head in tests right now :-D
12:29 pdurbin
Ok, I'll add it to my list. Thanks.
12:31 poikilotherm
:-)
12:36 pdurbin
You still have plenty of time before any QA happens today. :)
13:13 jri_ joined #dataverse
13:39 poikilotherm
pdurbin: crossing fingers that flakiness vanished...
13:39 poikilotherm
Using fixed clocks and a De Lorean for time traveling now
13:40 poikilotherm
And tests on Travis are so slow because no caches in use.... :-/
13:40 poikilotherm
Maven downloading the internet...
13:50 jri joined #dataverse
13:53 donsizemore joined #dataverse
13:58 pdurbin
any faster on Jenkins?
13:58 poikilotherm
Sure. Uses local .m2 by default.
13:58 pdurbin
I see it chugging along at https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/job/PR-6365/
13:59 pdurbin
Test Result (no failures)
13:59 poikilotherm
I gave you a shout out at https://github.com/IQSS/dataverse/pull/6365#issuecomment-557538037
13:59 donsizemore
that box has 2 cores and 8GB of RAM , let me know if it needs more
14:01 poikilotherm
Morning donsizemore :-)
14:01 donsizemore
good morning!
14:02 donsizemore
(or afternoon as the case may be)
14:02 poikilotherm
;-)
14:02 poikilotherm
I am impressed of the integration testing
14:02 poikilotherm
Seems to work pretty smooth
14:03 poikilotherm
Although it's taking ages
14:03 poikilotherm
20 minutes
14:03 pdurbin
poikilotherm: what if we spun up dataverse-kubernetes from Jenkins?
14:03 poikilotherm
You can have a few timesavers with that
14:04 poikilotherm
Depends on how much AWS bill you are ok with :-D
14:04 pdurbin
We're ok adding to the AWS bill. :)
14:04 pdurbin
Plus, donsizemore added logic to spin down EC2 instances after a certain amount of time.
14:04 pdurbin
"terminate" them
14:04 poikilotherm
Then you could let run a K8s cluster. Building and throwing away these takes time
14:05 poikilotherm
But it will then constantly need running machinas
14:05 poikilotherm
I have no experience in autoscaling this stuff
14:05 poikilotherm
Like only let a small/cheap/free node run the master
14:05 poikilotherm
And spin up workers as needed
14:06 donsizemore
have you used EKS?
14:06 poikilotherm
I tried it once
14:06 poikilotherm
And it still costs you money
14:06 donsizemore
yes but less than VMs
14:06 pdurbin
It's ok if EKS costs us money.
14:06 pdurbin
Honestly, we're saving money in developer time, QA time, etc.
14:06 poikilotherm
EKS will cost you VMs in addition
14:07 poikilotherm
EKS is just about not running the K8s stack yourself
14:07 poikilotherm
It will still consume VMs for the nodes
14:08 pdurbin
poikilotherm: to back up a bit. What if https://demo.dataverse.org was powered by dataverse-kubernetes?
14:10 poikilotherm
You could run that service on a K8s cluster and use it for ephemeral things, too
14:10 poikilotherm
namespaces for the win
14:10 poikilotherm
tests would run in ephemeral namespaces
14:11 poikilotherm
That way you could have a minimum configuration running always, maybe add autoscaling for nodes (that get teared down as necessary)
14:12 poikilotherm
You can take away lots of stuff from K8s, but you need some basic services running.
14:12 poikilotherm
Another wild guess: one could look into running the basic stuff with t2.micro instances
14:13 poikilotherm
And add bigger nodes for tests when they are needed
14:13 poikilotherm
t2.micro are free, up to 750 of em
14:17 pdurbin
Heh. Sure but we have to walk before we can run. :)
14:18 poikilotherm
Just found https://zero-to-jupyterhub.readthedocs.io/en/latest/amazon/step-zero-aws.html
14:18 poikilotherm
They use t2.micro for master
14:18 pdurbin
cool
14:19 poikilotherm
As long as we don't try to run real test scenarios on those, this should be worth a try
14:19 donsizemore
except ^^ doesn't use java ;)
14:19 pdurbin
poikilotherm: I may have asked you this before, but do you think it would be possible to get the spinning up of dataverse-kubernetes on AWS down to a single script like we have with dataverse-ansible?
14:19 poikilotherm
I could create a shell script. It will contain 2 lines
14:19 poikilotherm
1 for firing up the K8s cluster
14:19 poikilotherm
The other to deploy
14:20 pdurbin
That sounds fantastic.
14:20 pdurbin
Would you like me to create an issue?
14:20 poikilotherm
LOL
14:20 poikilotherm
Sure, go ahead. Or add to the existing about AWS usage
14:25 pdurbin
poikilotherm: here you go: https://github.com/IQSS/dataverse-kubernetes/issues/127
14:26 pdurbin
donsizemore: on a related note, I brainstormed a bit with Erik (the OCR guy and lots of other stuff guy) about how to add a GUI to ec2-create-instance.sh. I think we settled on fronting that shell script with a REST API . And then writing a little web front end to call into it. What do you think?
14:31 donsizemore
@pdurbin sounds good to me
14:32 pdurbin
What's a good language to use to wrap a shell script with a REST API ?
14:34 tai joined #dataverse
14:36 * poikilotherm
shudders
14:37 pdurbin
heh
14:37 pdurbin
poikilotherm: what do you suggest? :)
14:38 poikilotherm
Depends on what you are going to do
14:38 pdurbin
The GUI for that script will only be run on laptops of people who don't want to think about scripts and command line arguments.
14:39 poikilotherm
How about reimplementing the logic in python instead of shell?
14:39 poikilotherm
You could then go ahead and use things like pyqt
14:39 poikilotherm
No need for a REST API
14:40 pdurbin
that sounds like more work
14:40 poikilotherm
Actually IMHO fronting ansible with a shell script fronting with a REST API (it wouldn't be restful BTW) fronting with a guys sounds like an over-engineered thing :-D
14:40 pdurbin
and less cross platform
14:41 poikilotherm
https://docs.python.org/3/faq/gui.html
14:42 pdurbin
But I don't know Qt or any of the rest of these. (Ok, I did hack a little Gtk+ back in the day.) I know how to do a POST with jQuery. :)
14:43 pdurbin
And I can teach Danny and Tania how to run python3 -m http.server
14:45 poikilotherm
https://github.com/ansible/ansible-runner-service
14:45 poikilotherm
Sounds like it does a lot of stuff you already want
14:46 poikilotherm
That shell script logic can be done in javascript
14:46 poikilotherm
And you could use that with npm on CLI
14:46 poikilotherm
Err... s/npm/node/
14:46 pdurbin
poikilotherm: I'm just glad you're not telling use to re-write Dataverse itself from Java to whatever language. :)
14:46 poikilotherm
LOL
14:46 pdurbin
Because we had a guy like that.
14:47 poikilotherm
We are talking about the automation part. Not the application ;-)
14:47 pdurbin
yeah
14:48 poikilotherm
There's also an AWS SDK for NodeJS
14:48 poikilotherm
So you might even make it easier to use by removing the necessity to shell out aws commands
14:48 pdurbin
Would Flask be ok? I'm looking at https://stackoverflow.com/questions/53380988/how-to-execute-shell-script-from-flask-app
14:49 pdurbin
poikilotherm: I'd say Quarkus but wouldn't it be a pain in the next to shell out from Java?
14:49 pdurbin
neck*
14:49 poikilotherm
Hehe
14:49 poikilotherm
Lots of options to choose from
14:50 pdurbin
Actually, Flask would require a pip install so I probably wouldn't want that. Maybe Go.
14:50 poikilotherm
My 2 cents: refactor and move away from the shell script.
14:51 poikilotherm
F**K gotta run pickup kids
14:51 pdurbin
Actually, they already have to do a pip install to get the aws client.
14:51 poikilotherm
Read you all next week
14:51 pdurbin
so what's one more pip install
14:51 pdurbin
donsizemore: all this is low priority, of course. Don't mind me. :)
15:18 donsizemore
@pdurbin don't mind what? just got back from ordering matthew's new MBP!
15:22 pdurbin
nice
15:27 donsizemore
(and whatever you all want to do is fine with me, i just implemented stuff as requested)
15:30 pdurbin
you sure do
15:38 donsizemore
p.s. related to wholetale: have you seen Dendro?
15:38 pdurbin
nope and I miss searchbot
15:42 pdurbin
I think I found it: https://github.com/feup-infolab/dendro
16:49 donsizemore joined #dataverse
18:13 jri joined #dataverse
18:47 pdurbin
Heh. 'Docker Image NEEDED "like bread to the mouth"' https://github.com/feup-infolab/dendro/issues/368
18:57 jri joined #dataverse
19:04 nightowl313 joined #dataverse
19:15 nightowl313
can anyone help me with creating the metadata file from our DV installation to send to my provider (my organization is a provider) for shibboleth? I have sent them my metadata file from the instructions on the harvard site, but they need it to include our certs. I have added the provider, and it shows on the login screen, but not sure how to configure that metadata file with our certs for them
19:20 pdurbin
nightowl313: it should be at a URL like https://demo.dataverse.org/Shibboleth.sso/Metadata
19:42 nightowl313
that is the file that I saved and gave them, but it didn't include my certs
19:43 nightowl313
maybe I generated it wrong, but I did follow the instructions here: http://guides.dataverse.org/en/latest/installation/shibboleth.html #
19:43 nightowl313
to the best of my ability
19:44 pdurbin
nightowl313: do you see how https://demo.dataverse.org/Shibboleth.sso/Metadata has "<ds:X509Certificate>MIIDJ..."? Does yours have something similar?
19:46 nightowl313
no, and I just ran the curl command again to generate it
19:46 pdurbin
weird
19:47 pdurbin
nightowl313: can you please email support
19:48 nightowl313
oh sure...will do that...thank you!
19:48 pdurbin
thanks, and please let me know the ticket number
19:58 nightowl313
ticket #283463 ... thanks!
20:03 pdurbin
nightowl313: I see it. Thanks!
21:31 pdurbin
Ok, folks. I'm out of here. Have a lovely weekend. See you Monday.
