Time
S
Nick
Message
08:06 stefankasberger joined #dataverse
10:56 pdurbin joined #dataverse
11:56 donsizemore joined #dataverse
11:59 donsizemore
@pdurbin Dear Leader!
12:19 pdurbin_m joined #dataverse
12:19 pdurbin_m
donsizemore: talk to me
12:20 donsizemore
@pdurbin welcoming you back
13:03 pdurbin
donsizemore: thanks! I highly recommend Banff, Jasper, etc. Great hiking. Cool in the summer.
13:11 donsizemore
@pdurbin at this point i'd settle for a day by a pool
13:12 pdurbin
oh, we did some hot springs too, great after a big hike
13:19 donsizemore
i'm insanely jealous!
13:30 pdurbin
donsizemore: should I take a look at your todo column? I've lost track or where we are with everything. :)
13:31 donsizemore
@pdurbin i think heppler should be going, he was going to try his sampledatabranch
13:31 pdurbin
ah, right, I saw you added the ability to specify the branch, thanks!
13:31 donsizemore
@pdurbin also, we added a jenkins job: https://jenkins.dataverse.org/job/IQSS-Dataverse-PullRequest/ (which I think I've corrected for round two which will happen on the next PR submission)
13:32 pdurbin
Sounds useful. How much testing does that job do?
13:32 donsizemore
right now it's at parity with https://jenkins.dataverse.org/job/IQSS-dataverse-develop/
13:32 donsizemore
Gustavo wanted tests against PRs as they came in
13:33 pdurbin
Sure but what kinds of tests does he want?
13:36 donsizemore
I think optimally the API test suite as well but on that box...
13:37 pdurbin
Right, we all want the API test suite. Are we blocked on that?
13:39 donsizemore
eh, i just haven't fired it off on that box in a while
13:41 pdurbin
Ok. What would help us move forward on getting the API test suite running? Just start firing it off again and remind ourselves what doesn't work?
13:47 donsizemore
something knocks glassfish over and it starts returning 500s. despite the same tests on the same code succeeding on your box, which has identical virtual hardware specs
13:48 donsizemore
of course, it's been a month, so let me kick it off again
13:52 pdurbin
The same thing happens on Glassfish running directly on my Mac. I can't run the full test suite because Glassfish dies in the middle. Strangely, Glassfish doesn't fall over when I run the test suite against Glassfish running within Docker on my Mac.
14:04 donsizemore
well, this might do it: Internal Exception: org.postgresql.util.PSQLException: ERROR: deadlock detected
14:05 pdurbin
yuck
14:05 pdurbin
donsizemore: heads up that the API test suite recently started failing on phoenix. I just opened this issue a couple hours ago: https://github.com/IQSS/dataverse/issues/6073
14:06 donsizemore
yes, i saw that
14:07 pdurbin
I could ssh into the phoenix server and see if deadlocks are server.log.
14:09 donsizemore
[ERROR] Tests run: 97, Failures: 11, Errors: 0, Skipped: 4. I hit FilesIT.testForceReplaceAndUpdate:631 XML path data.files[0].label doesn't match.
14:09 donsizemore
after that call everything comes back 500
14:10 pdurbin
interesting, on a related note I'm seeing this in the server.log for phoenix: Error processing /api/v1/files/84/replace: javax.ejb.EJBException
14:11 donsizemore
@pdurbin i ticked off a number of issues and submitted a couple small PRs i'll let you review at your leisure; in general i still owe you a few closed tickets
14:12 pdurbin
Ok, I have 53 unread emails having to do with GitHub issue updates so it may take me a while to find them. :)
14:13 pdurbin
Let me go through your column quick: https://github.com/orgs/IQSS/projects/3#column-5977566
14:14 pdurbin
Any update on Munin?
14:16 donsizemore
for me the postgres deadlock happens right after edu.harvard.iq.dataverse.api.UtilIT createRandomUser
14:16 donsizemore
Munin: https://github.com/IQSS/dataverse-ansible/commit/6902d576ee318a6fba09173a7f25c7b610dc28cf
14:17 donsizemore
Prometheus: https://github.com/IQSS/dataverse-ansible/commit/5346f05dadf007ea79c1af1766bfa888b16af116
14:17 pdurbin
Prometheus! Very cool.
14:19 pdurbin
Should I spin up an ec2 instance with prometheus and you can show me around? Is it anything like Munin? I love how Munin "just works" and starts creating a dashboard of graphs within minutes.
14:20 donsizemore
you're welcome to spin up an instance with both. i got the basic framework in place and am happy to work on customization
14:20 pdurbin
Cool. Is there a dashboard?
14:20 pdurbin
A dashboard of graphs like this I mean: http://munin.greptilian.com/greptilian.com/server3.greptilian.com/index.html
14:23 pdurbin
Or do we have to add something to make the graphs?
14:23 donsizemore
that was last week; gimme a minute and i'll spin up a VM to poke at
14:23 pdurbin
cool, cool
15:02 donsizemore
@pdurbin lemme know when you want to look at what prometheus does and doesn't do in its current state and i can send you URL
15:04 pdurbin
donsizemore: ready!
15:05 donsizemore
starting possibly with https://github.com/prometheus/client_java
15:05 donsizemore
now that i look again, it's just a vanilla install here http://irss-dls-buildbox.irss.unc.edu:9090
15:06 donsizemore
so i'll point client_java at glassfish and solr for starters
15:10 pdurbin
donsizemore: cool but where are the graphs?
15:11 donsizemore
there aren't any. i started with the basic framework and got sidetracked. will point client_java at glassfish and solr next?
15:14 pdurbin
Hmm, maybe. But maybe it would be nice to have a few graphs enabled from a dashboard or something? This one seems to work, related to memory: http://irss-dls-buildbox.irss.unc.edu:9090/graph?g0.range_input=1h&g0.expr=process_resident_memory_bytes&g0.tab=0
15:14 pdurbin
Does that make sense?
15:14 pdurbin
I mean, maybe we could start with three graphs or so.
15:15 pdurbin
- memory
15:15 pdurbin
- disk space
15:15 donsizemore
so i'll start with pecking out some built-ins
15:15 pdurbin
- cpu
15:15 pdurbin
or whatever
15:15 donsizemore
the memory seems to be the sticking point ;)
15:15 pdurbin
ah :)
15:15 pdurbin
well, it's always useful to know when your disk is full :)
15:15 pdurbin
standup time
15:19 donsizemore
will do
15:49 pdurbin
I feel like the Binder folks have nice graphs of their infra. I don't know what they use.
15:49 pdurbin
here it is: https://prometheus.mybinder.org/graph
16:28 donsizemore
i see nothink
17:06 pdurbin
donsizemore: here we go. Please go to https://grafana.mybinder.org and click "Node Activity"
17:17 donsizemore
those look really nice
17:21 pdurbin
Yeah. But I'm not trying to pile on the work. Do we need Grafana?
17:24 donsizemore
mybinder inserts a number of calls in their python to collect data for prometheus https://github.com/search?p=4&q=org%3Ajupyterhub+prometheus&type=Code
17:25 pdurbin
Yeah, and I'm seeing some stuff Adam Bien put together to get Glassfish metrics into Prometheus.
17:26 donsizemore
https://github.com/AdamBien/firehose
17:27 pdurbin
yeah, that's the one I found
17:35 pdurbin
Anyway, I fear that if there aren't pretty graphs out of the box people won't see the value in it. I get immediate value from Munin for the operating system level stuff. :)
17:45 donsizemore
i'm working on the out-of-the-box part for system-wide numbers but for real information from dataverse you'll want to pepper the codebase for prometheus
17:52 pdurbin
Oh yes, absolutely.
17:53 pdurbin
Is there a way to create a dashboard of graphs without Grafana?
17:59 donsizemore
i mean, i'm just tinkering with prometheus. if you like munin better that's enabled with group_vars as well. but yes it looks like the prettiest graphs would come from grafana
18:02 pdurbin
I see.
18:02 pdurbin
I don't know what I want. Except that I don't want to waste your time. :)
18:07 donsizemore
this is interesting and even fun, but to spin something up at random for basic testing, one may select say go_memstats_heap_inuse_bytes and generate a chart in real time from that VM
18:08 pdurbin
Sorry, when you say VM do you mean an instance of the JVM ?
18:12 donsizemore
let me tinker with jmx_exporter
18:12 pdurbin
ok :)
18:13 pdurbin
Why do so many metrics start with "go_"?
18:15 donsizemore
prometheus is written in go
18:15 pdurbin
I was wondering if that's why. :)
18:15 pdurbin
So it has nothing to do with go?
18:18 pdurbin
I just found this: https://prometheus.io/docs/practices/naming/
19:15 pdurbin
donsizemore: anyway, I'm glad you're having fun. :)
19:15 donsizemore
i retested the munin branch, BTW - works as advertised
19:16 pdurbin
Cool. You mean that the OS level Munin stuff works, right? We'll have to hack on the jstat stuff.
20:55 poiki-mobile joined #dataverse
20:55 poiki-mobile
Hey guys, what's up?
21:02 pdurbin
poiki-mobile: just got home after picking the kids up from camp. I thought you were on vacation. :)
21:02 poiki-mobile joined #dataverse
21:02 poiki-mobile
Yes I am... :-)
21:03 pdurbin
tsk, tsk, you should be relaxing :)
21:03 poiki-mobile
I am. Did construction work since 8am till 10pm
21:04 poiki-mobile
On my way home I listened to the latest airhacks episode...
21:04 poiki-mobile
And was dreaming about a Dataverse on steroids ;-)
21:05 pdurbin
Hmm, I just finished the second to last.
21:05 pdurbin
Which was also good.
21:05 poiki-mobile
Yeah
21:06 poiki-mobile
Sebastian mentioned Quarkus in that episode
21:06 pdurbin
yeah, and fast deployments
21:06 poiki-mobile
I was very curious for the current episode all about Quarkus
21:07 poiki-mobile
This is so amazing stuff on so many different levels
21:08 pdurbin
Yeah, he's covered Quarkus in previous episodes but I'm looking forward to a deep dive.
21:11 pdurbin
poiki-mobile: thanks for reaching out to BalusC
21:12 poiki-mobile
:-)
21:12 pdurbin
I hope you don't mind but I forwarded it to our internal list with my own summary. A summary of my understanding, I mean. I can forward it to you if you like.
21:13 poiki-mobile
So many people out there, so I thought I should just ask someone who wrote a book about this stuff... ;-)
21:13 poiki-mobile
That's totally fine and that's why I put you and Danny in CC
21:14 pdurbin
Cool. I'm just trying to get the word out. It's a good idea.
21:14 poiki-mobile
I you don't mind, I would be glad if you could forward it :-)
21:15 pdurbin
sure, sent
21:15 poiki-mobile
s/I/If/
21:15 poiki-mobile
Thx
21:15 pdurbin
Please correct any misunderstanding I have. I'm trying to go quickly through a lot of unread email. :)
21:17 poiki-mobile
The summary sounds good to me. I wonder about the differences Danny may have...
21:18 pdurbin
you mean Gustavo
21:18 poiki-mobile
Whoops
21:18 poiki-mobile
Yeah
21:19 pdurbin
It's hard when you haven't met them. :)
21:19 pdurbin
At #dataverse2019 I kept saying that we should have figured out a way to fund your travel. It's a bummer you weren't there.
21:20 poiki-mobile
Yeah... better luck next time. ;-)
21:20 pdurbin
I already put a bug in my wife's ear about hacktoberfest. :)
21:20 poiki-mobile
I hope I did not step on someone's foot in #6070?
21:20 pdurbin
Oh and did I tell you I'll probably hopefully be in Portugal in January?
21:20 poiki-mobile
Shht, this is not official yet...
21:21 poiki-mobile
Oh cool! Where in Portugal? I've been to Feragudo last year.
21:21 pdurbin
Lisbon for https://pidapalooza.org
21:23 pdurbin
I'm still 4 unread emails away from that issue.
21:23 poiki-mobile
Ah nice. I will ask my boss if I should go there instead of going to ETH Zürich...
21:24 pdurbin
Would you also go to FOSDEM? It's the weekend after.
21:24 poiki-mobile
Oh that's in reach... Brussels is not far away, I can go there by train in about 2 hours
21:25 poiki-mobile
maybe 3
21:26 pdurbin
I don't think I'm funded to go to FOSDEM though. I'll probably ask. My wife approves it, which is the more important thing. :)
21:26 poiki-mobile
Ah, that sounds familiar... :-D
21:27 pdurbin
She might even come with me to Portugal. Especially if my parents come to Boston to take care of the kids. I haven't asked yet. :)
21:28 pdurbin
It would probably mean less hacking for us though. :)
21:29 poiki-mobile
LOL
21:31 pdurbin
She's more of a biotech hacker.
21:31 pdurbin
She plays God: https://twitter.com/philipdurbin/status/664645552353755136
21:58 pdurbin
poiki-mobile: ok, I finally got around to leaving you a comment: https://github.com/IQSS/dataverse/issues/6070#issuecomment-518859597
22:00 poiki-mobile
pdurbin I think you misunderstood my explanation ...
22:00 pdurbin
whoops
22:00 poiki-mobile
Lyncode is relying on v3, Not v2
22:01 poiki-mobile
Nothing else than Dataverse Code itself is relying on v2.
22:01 poiki-mobile
At least according to mvn dep tree
22:01 poiki-mobile
So the WAR will get smaller, as v3 ist already included, but v2 can be left out.
22:02 pdurbin
Ok. So you want to remove v2. Would you also delete all the jars in "local_lib/com/lyncode"?
22:02 poiki-mobile
Err why should I do so?
22:03 poiki-mobile
Isn't this beyond scope for #6070?
22:03 pdurbin
I think we're trying to figure out the scope. :)
22:03 poiki-mobile
Small chunks... ;-) Dunno where I heard that...
22:03 pdurbin
I'm interested in:
22:03 pdurbin
- making the war file smaller
22:04 poiki-mobile
I totally agree in ditching and dumping that devil local_lib...
22:04 pdurbin
- removing local_lib libraries that we have forked
22:04 pdurbin
- keeping harvesting (and all features) working :)
22:04 poiki-mobile
Nothing that should be kept in a git repo...
22:06 pdurbin
Death to local_lib.
22:07 poiki-mobile
I think removing local_lib is beyond scope for #6070, but aligned with the goal of making the WAR smaller
22:08 poiki-mobile
As this part of local_lib is already relying on v3 and it is "just" Dataverse code that needs changes, it should be pretty straight forward as long as there are no things in use that had breaking changes.
22:08 pdurbin
I guess I'm confused. You want to use v3. It's already there. So go ahead and use it. Right?
22:08 poiki-mobile
Nonono...
22:08 poiki-mobile
Current codebase uses v2.
22:09 poiki-mobile
This is a rotten fish.
22:09 pdurbin
heh
22:09 pdurbin
But I thought you said v3 gets pulled in.
22:09 poiki-mobile
v3 ist included as a transient dep of the lyncode
22:09 pdurbin
ok
22:10 poiki-mobile
So you could use it in the codebase, but you would be breaking the rules of dep managment
22:10 pdurbin
heh, what rules? :)
22:10 poiki-mobile
actually the Golden rule...
22:10 poiki-mobile
Declare want you use.
22:10 poiki-mobile
-want +what
22:11 pdurbin
So you could declare v3 explicitly. And add a comment saying that we should stop using v2.
22:11 pdurbin
Then you won't be breaking the rules. :)
22:11 poiki-mobile
Yeah.
22:11 poiki-mobile
I could.
22:11 poiki-mobile
IMHO this does not make sense.
22:12 poiki-mobile
Commons Lang v3 contained some breaking changes.
22:12 pdurbin
What would you rather do?
22:12 poiki-mobile
I don't think Dataverse is affected by those.
22:13 poiki-mobile
So this merily bouls down to replacing some imports and changing the version in POM
22:13 poiki-mobile
boils
22:13 pdurbin
But doesn't harvesting require a patched jar? Would your change break harvesting?
22:14 poiki-mobile
Again: lyncode does not use v2, but relies on v3.
22:14 poiki-mobile
It is just our code using the v2 lib as far as I can tell from mvn dep tree plugin
22:15 poiki-mobile
I would just swap out v2 and use v3 everywhere.
22:15 pdurbin
Ok, so in your pull request you would not remove our patched version of lyncode. How much Dataverse code uses v2? How much testing would QA have to do?
22:16 poiki-mobile
Let me switch to my laptop to get some numbers...
22:16 pdurbin
ok
22:17 poikilotherm joined #dataverse
22:17 pdurbin
and sorry, I'm a little distracted trying to catch up on email
22:18 pdurbin
18 unread github issue threads
22:18 pdurbin
poikilotherm: what's the benefit or removing v2? A smaller war file? How much smaller?
22:19 poikilotherm
It's always benefical ripping out a library not updated since 2011
22:20 poikilotherm
Let me check on the size
22:20 pdurbin
If we're going to work on ripping out old things, let's put our energy into ripping out old app servers. :)
22:21 poikilotherm
This is a low hanging fruit...
22:21 poikilotherm
commons.lang is used in 87 class files
22:22 poikilotherm
find src -name '*.java' -exec grep -Hi "commons.lang" {} \; | cut -f1 -d":" | sort | uniq -c | wc -l
22:22 poikilotherm
Oh and lang3 is already used in 3 files
22:22 poikilotherm
Err... 5
22:22 pdurbin
Sound like a lot. Would you write up a "how to test" doc for QA?
22:23 poikilotherm
There should be no exhaustive testing necessary. Breaking changes will lead to compilation failures AFAIK
22:23 poikilotherm
But of course I can provide a list of features affected
22:24 pdurbin
That would be helpful. But should I go look at the list I made for you? :)
22:24 poikilotherm
Hmm maybe I can even shorten that list to things not covered by tests
22:24 poikilotherm
Your board?
22:25 poikilotherm
(column)
22:25 pdurbin
Yeah, I made a column for you. You told me I could. :)
22:25 pdurbin
I forget what's on there.
22:25 pdurbin
Something about spinning up arbitrary brankes on k8s.
22:25 pdurbin
branches*
22:25 poikilotherm
Yeah
22:25 poikilotherm
I am currently testing this locally
22:25 poikilotherm
Redeployment times ARE DRIVING ME NUTS
22:25 pdurbin
Yeah, that's the only thing in your column: https://github.com/orgs/IQSS/projects/3#column-5999026
22:26 poikilotherm
I have to wait ~8 min
22:26 poikilotherm
for
22:26 poikilotherm
every
22:26 poikilotherm
fricking
22:26 poikilotherm
single
22:26 poikilotherm
small
22:26 poikilotherm
change
22:26 pdurbin
heh
22:26 poikilotherm
Build is fast
22:26 poikilotherm
But fear deployment times
22:27 pdurbin
Should we take something tiny like https://github.com/pdurbin/javaee-docker and put all the goodies and good practices into it?
22:27 poikilotherm
That's why I am so exited about Quarkus BTW ;-)
22:27 pdurbin
...goodies including quarkus if you want
22:28 poikilotherm
Adam mentioned some showcase project for Quarkus in the last episode
22:28 pdurbin
ah, ok
22:28 poikilotherm
http://www.adam-bien.com/roller/abien/entry/quarkee_the_java_ee_stic
22:28 pdurbin
mvn archetype:generate -DarchetypeGroupId=com.airhacks -DarchetypeArtifactId=quarkee-archetype
22:28 pdurbin
ok
22:29 pdurbin
Sure, we could use that as a starting point.
22:30 pdurbin
We could call it dataverse-next or something. :)
22:30 poikilotherm
LOL
22:31 poikilotherm
https://developers.redhat.com/blog/2019/04/12/migrating-java-applications-to-quarkus-lessons-learned/
22:32 poikilotherm
Maybe one should think about getting of Glassfish/Payara and move to Wildfly...
22:32 pdurbin
A long time ago I created https://github.com/IQSS/iqss-javaee-template
22:32 poikilotherm
Getting there is most likely making it a lot easier to have sth. like Quarkus get used, as it uses Wildfly under the hood
22:33 poikilotherm
WOW! That's cool :-)
22:34 pdurbin
:)
22:34 pdurbin
It's funny to see icefaces in there.
22:34 poikilotherm
Ancient archetypes from the past are amazing :-)
22:35 poikilotherm
Man this is from 2012 and look where we are today
22:35 pdurbin
At the time there were two projects at IQSS that used very similar stacks. Dataverse is the open source one.
22:36 poikilotherm
I just browsed through the logs for today...
22:37 poikilotherm
Saw you and donsizemore talking about Prometheus...
22:37 poikilotherm
Just recently I began to play with Prometheus and Grafana for my laptop ;-)
22:37 poikilotherm
Planning to add a small howto to dataverse-kubernetes later on, as this is state of the art
22:38 poikilotherm
Especially for K8s deployments
22:38 poikilotherm
There are more options obviously ;-)
22:39 poikilotherm
I already have some experience with the other classic collectd+Influx+Grafana
22:39 pdurbin
oh, great!
22:39 pdurbin
speaking of ancient, I still use Munin on my home server (and I love it)
22:40 poikilotherm
We could easily create a dashboard in Grafana and share it on the Grafana website
22:40 poikilotherm
And we could easily share a prometheus config
22:41 poikilotherm
You could already have a look at the data with prometheus itself, but Grafana has a lot of more options for graphing
22:41 poikilotherm
And there is a good chance these days there is already an installation of Grafana present
22:43 pdurbin
Did you see Don's pull request to add Prometheus to dataverse-ansible?
22:45 poikilotherm
Nope. I don't follow dataverse-ansible very closely...
22:45 pdurbin
poikilotherm: please feel free to leave a review: https://github.com/IQSS/dataverse-ansible/pull/96
22:45 poikilotherm
pdurbin should we write an update to #6070?
22:46 pdurbin
Maybe, I'm sorry I got it wrong.
22:46 pdurbin
I'd like to find a better use of your time, honestly. Monitoring would be great.
22:47 pdurbin
A reproducible test case for a question for BalusC on Stack Overflow would be great.
22:47 poikilotherm
I need to work on #6000 first... ;-)
22:47 pdurbin
I have some "help wanted: python" stuff on my board if you'd like to take a look at any of that.
22:47 poikilotherm
I have sth. working, but wanted to have tests, too
22:48 pdurbin
Some help with Jenkins would be great.
22:48 poikilotherm
Hihihi
22:49 poikilotherm
Go get some funding, I try to get a month or two of and work for IQSS on whatever you want :-D
22:49 pdurbin
:)
22:49 poikilotherm
(Although I doubt my boss would do that, at least not until we are in production)
22:50 pdurbin
Let's have you work on something your boss cares about and that I care about. :)
22:50 pdurbin
I'm sure there's plenty of common ground. :)
22:50 poikilotherm
I hope so :-D
22:51 poikilotherm
Can you leave a comment on #6070? I really need to get some sleep... 1am over here...
22:52 pdurbin
I'll leave myself a reminder to do so.
22:52 poikilotherm
Thank you so much! Awesome :-)
22:52 pdurbin
reminder made :)
22:53 poikilotherm
Have a good night - maybe read you tomorrow... ;-)
22:53 pdurbin
get some sleep!
22:53 pdurbin
:)
