Time
S
Nick
Message
07:07 jri joined #dataverse
07:50 stefankasberger joined #dataverse
08:19 pmauduit
pdurbin: I'd say it's the same amount of work to plug both jvm into collectd
08:19 pmauduit
each will have a dedicated JMX port that will be harvested by collectd
09:54 pdurbin
pmauduit: cool but when I look at https://lucene.apache.org/solr/guide/7_3/monitoring-solr-with-prometheus-and-grafana.html there is no mention of collectd. Obviously, there's more than one way to do it. :) I'm fine with whatever. :)
10:18 pmauduit
pdurbin: yes because solr seems to have a prometheus exporter
10:18 pdurbin
yeah
10:18 pdurbin
this is why I'm excited about prometheus
10:18 pdurbin
lots of projects seem to support it
10:23 pmauduit
maybe there is a prometheus exporter in the J2EE server that dataverse uses (I still can't remember which one it is)
10:24 pdurbin
Well, it's a little complicated. :)
10:25 pdurbin
Dataverse is stuck on Glassfish 4 and there are workarounds to have a Prometheus exporter. But I want us to upgrade to Payara 5 which is a fork of Glassfish that includes a Prometheus exporter.
10:29 pmauduit
ok, nevermind, we still should be able to open a JMX endpoint where we should have at least the default JVM metrics (which can then harvested by collectd)
10:30 pdurbin
Sounds good. :)
11:13 pdurbin
pmauduit: does this help with how to get JMX metrics from Glassfish? https://docs.oracle.com/cd/E19798-01/821-1751/giwqm/index.html
11:15 pdurbin
This is what I see in server.log: JMXStartupService has started JMXConnector on JMXService URL service:jmx:rmi://ip-172-31-36-60.ec2.internal:8686/jndi/rmi://ip-172-31-36-60.ec2.internal:8686/jmxrmi|#]
11:15 pmauduit
pdurbin: you probably have somewhere a file to edit to add some JVM options, right ?
11:15 pdurbin
Yes, we put JVM options in /usr/local/glassfish4/glassfish/domains/domain1/config/domain.xml
11:16 pmauduit
you could let it listen to localhost, no need to expose it on the internet (even if you've some security-groups to firewall)
11:16 pmauduit
ok
11:16 pmauduit
i'll try to plug collectd onto it then
11:16 pmauduit
(sorry I'm in a meeting, so I might lag a little bit)
11:30 pdurbin
No problem. Meanwhile I'm playing around a bit with this: echo beans | java -jar jmxterm-1.0.0-uber.jar --url localhost:8686 (but I don't really know what I'm doing) :) from https://github.com/jiaqi/jmxterm
11:42 pmauduit
pdurbin: I've got some notes about using jmxterm but it might be 1. on our private confluence 2. in french unfortunately :P
11:50 pmauduit
nothing really interesting in my notes though, but first after having opened the JMX endpoint, you can 1. select a domain 2. list / select a MBean 3. enjoy
11:51 pmauduit
you can launch jmxterm as a shell, for explorational purposes
11:51 pmauduit
*exploratory
11:56 pmauduit
yum packaging system is weird sometimes, I'd expect that jmx plugin for collectd will need the java plugin also, but I had to setup it also by hand explicitely
11:57 pmauduit
pdurbin: done
12:02 pmauduit
pdurbin: http://ec2-3-81-53-52.compute-1.amazonaws.com/grafana/d/COPs6vKZk/overall-metrics?orgId=1&from=now-5m&to=now&fullscreen&panelId=6
12:12 pdurbin
pmauduit: fantastic! I just commented on the issue after you and tried to save the updated dashboard config: https://github.com/IQSS/dataverse-ansible/issues/99#issuecomment-525272836
12:14 pmauduit
I can see a overall jmx usage in the json, so I'd guess so :)
12:14 pmauduit
s/jmx/heap/ sorry
12:16 poikilotherm joined #dataverse
12:24 pdurbin
:)
12:25 pmauduit
pdurbin: FYI in the legend if you click onto "heap-used" you will see only this metric
12:26 pmauduit
we can also change this in the config so that we have only heap-used selected
12:27 poikilotherm
Hey guys :-)
12:29 pdurbin
hey poikilotherm
12:30 poikilotherm
Been busy yesterday with an intern from university of Stuttgart :-)
12:30 poikilotherm
She was very eager about our stuff :-D
12:31 pdurbin
pmauduit: ok. I don't know what I want exactly. At a high level, I want memory usage of glassfish/dataverse to be tracked in prometheus/grafana and then I want to hit Dataverse with curl over and over to exercise this memory leak (or resource utilization at least, if not a memory leak): https://github.com/IQSS/dataverse/issues/6035
12:32 pdurbin
poikilotherm: great! Would she like to be contributor 100 for Dataverse? :)
12:32 poikilotherm
I don't think so
12:32 pdurbin
ok
12:33 pdurbin
poikilotherm: how interested are you in all this monitoring stuff?
12:33 poikilotherm
Very :-D
12:33 pdurbin
Would you want to put some of it in dataverse-kubernetes?
12:34 poikilotherm
Sure!
12:34 pdurbin
cool
12:34 poikilotherm
You should be running monitoring for K8s already when going into production
12:34 pdurbin
sure
12:34 poikilotherm
And these days, Prometheus is very popular for this
12:35 pdurbin
yeah :)
12:35 pdurbin
we have pmauduit to thank for kickstarting us :)
12:35 pdurbin
pmauduit: I sent a nice email yesterday internally about what a great contributor you are :)
12:37 pmauduit
pdurbin: I'm honoured ! :)
12:37 pdurbin
poikilotherm: any opinions on where I should stay in Brussels? Downtown or near the university where FOSDEM takes place? I've never been there.
12:37 pmauduit
glad to be able to help anyway
12:37 poikilotherm
I have absolutely no idea
12:37 poikilotherm
Sry, never spent a night there
12:38 pdurbin
ok, no worries
12:40 poikilotherm
:-)
12:41 pdurbin
poikilotherm: can you please remind me where "Can one of the admins verify this patch?" is configured? I got a question about this yesterday.
12:42 poikilotherm
Lemme check the config
12:43 pdurbin
thanks
12:43 poikilotherm
Hmm someone seems to have created some more jobs
12:43 pdurbin
oh?
12:43 poikilotherm
https://jenkins.dataverse.org/job/IQSS-Dataverse-PullRequest/
12:44 pdurbin
oh, yes, that's new
12:44 pdurbin
Is that why dataversebot started commenting on pull requests?
12:44 poikilotherm
Most certainly
12:45 poikilotherm
Maybe those have been created be the plugin automatically
12:45 poikilotherm
Because I see some of my PRs there
12:45 pdurbin
sorry, what plugin?
12:46 pdurbin
Oh, I see... 'When a new pull request is opened in the project and the author of the pull request isn't white-listed, builder will ask "Can one of the admins verify this patch?".' https://wiki.jenkins.io/display/JENKINS/GitHub+pull+request+builder+plugin
12:46 pdurbin
So you didn't configure this message.
12:47 pdurbin
it seems to come from here: https://github.com/jenkinsci/ghprb-plugin/blob/ghprb-1.42.0/src/main/resources/org/jenkinsci/plugins/ghprb/GhprbTrigger/global.groovy#L32
12:47 poikilotherm
Yes
12:48 poikilotherm
But the whole thing needs to be activated for the repo
12:51 pdurbin
I guess I'm still confused. How does https://jenkins.dataverse.org know about dataversebot? Is this something you configured?
12:52 poikilotherm
Yeah
12:52 pdurbin
Where?
12:52 poikilotherm
I setup this for dataverse-kubernetes
12:52 poikilotherm
I setup credentials accessiable for all admins
12:53 poikilotherm
And used those in the global config and in the config for the job
12:53 pdurbin
Did you mean for dataversebot to comment on the main "dataverse" repo too? Here, for example: https://github.com/IQSS/dataverse/pull/6110#issuecomment-523727349
12:53 poikilotherm
Nope
12:54 pdurbin
Ok. Are we in control of this? Can we choose which repos dataversebot comments on? :)
12:54 poikilotherm
We should be :-D
12:54 donsizemore joined #dataverse
12:54 pdurbin
Good. :)
12:54 poikilotherm
The PR plugin has some capabilities to autoadd jobs IIRC
12:54 poikilotherm
This might have happened here
12:54 pdurbin
I don't even have a login to https://jenkins.dataverse.org but donsizemore does. :)
12:54 pdurbin
donsizemore: mornin'
12:55 donsizemore
good morning. i thought i made you an admin on jenkins.dataverse?
12:56 pdurbin
donsizemore: I can ssh in.
12:58 poikilotherm
I'm just going to delete the job
12:58 poikilotherm
https://jenkins.dataverse.org/job/IQSS-Dataverse-PullRequest/
12:58 poikilotherm
It has the GH PR plugin activated
12:58 pdurbin
poikilotherm: no no
12:59 pdurbin
poikilotherm: donsizemore and Gustavo created it while I was on vacation.
12:59 poikilotherm
OK
12:59 poikilotherm
So they will have activated the plugin in it :-D
12:59 pdurbin
I guess. I don't know.
12:59 poikilotherm
And they used the credentials I left there on purpose
13:00 poikilotherm
For me this is fine, these accounts where meant to be used by you/us/community/...
13:01 poikilotherm
Ah they didn't even need to use the credentials
13:01 poikilotherm
The global config is used
13:03 pdurbin
Ok. Again, I don't know what they did. But I have a better understanding now of what's going on. So thank you. :)
13:04 poikilotherm
You're welcome :-)
13:04 pdurbin
donsizemore: how much capacity do you have to work with me and pmauduit (and poikilotherm if he's interested) in getting the collectd/prometheus/grafana config into dataverse-ansible?
13:11 pdurbin
poikilotherm: do you want anything added or removed from your "todo" column on my board? This: https://github.com/orgs/IQSS/projects/3#column-5999026 :)
13:12 poikilotherm
Hehe, no that's ok for now.
13:13 pdurbin
poikilotherm: ok, I was thinking I could create an issue in dataverse-kubernetes about prometheus/grafana if you want
13:14 poikilotherm
Sure, go ahead :-) It will mostly be an docs issue
13:14 poikilotherm
Because there are already operators, etc to use for this.
13:14 poikilotherm
Not necessary to bundle your own
13:15 poikilotherm
Although, as long as we are not on Payara, a sidecar might be usefull for exporting the JMX things
13:16 pdurbin
Is Glassfish 4 hard coded into dataverse-kubernetes?
13:16 poikilotherm
Nop
13:17 poikilotherm
Nope
13:17 pdurbin
phew
13:17 poikilotherm
There are other images with Payara
13:22 pdurbin
cool
13:22 pdurbin
For now I just use dataverse-ansible to spin up Dataverse on Payara. At least the Dataverse APIs work. :)
13:23 pdurbin
poikilotherm: is https://github.com/orgs/IQSS/projects/4 still accurate in terms of "in focus" etc?
13:24 poikilotherm
Yeah :-)
13:27 pdurbin
ok :)
14:03 donsizemore
@pdurbin hey hey, i took yesterday for the dataverse upgrade and need to do one more "odum thing" this morning... i could start a branch this afternoon?
14:04 donsizemore
@poikolotherm feel free to clobber, correct, or undo anything i've done on jenkins.dataverse.org, particularly with your credentials
14:05 donsizemore
@poikilotherm i just saw the plugin installed and thought "oh, nice"
14:44 pdurbin
A new branch sounds perfect.
14:59 donsizemore joined #dataverse
15:02 donsizemore
@pdurbin do we care how much of the prometheus config is group_var configurable? just make it work up front and augment as needed?
15:06 pdurbin
donsizemore: augment as needed sounds just fine. Thanks!
15:07 pdurbin
Do you think you'll use this yourself? In production? Or on test servers?
15:17 donsizemore
definitely on a couple test servers. in production if need be
15:18 donsizemore
i'm running through prometheus and collectd on the branch, will add in grafana after lunch
15:27 pdurbin
donsizemore: awesome
15:30 pdurbin
bjonnh bricas_ jri pmauduit stefankasberger donsizemore xarthisius: community call in half an hour: https://dataverse.org/community-calls
18:07 donsizemore joined #dataverse
19:50 donsizemore
@pdurbin i'm not done but you can see where i'm at https://github.com/IQSS/dataverse-ansible/tree/99_collectd_grafana
19:52 * pdurbin
looks
19:53 pdurbin
donsizemore: cool. I think this is all making sense. We'll need to add the JSON file that defines a Grafana dashboard.
19:55 donsizemore
i've got that in place; i'm looking at the datasource stuff
19:55 donsizemore
and need to get a grafana API key programatically
19:56 donsizemore
or i suppose i can just do it as admin/admin for now
19:56 pdurbin
admin/admin is fine for me for now
20:17 donsizemore
@pdurbin the create datasource dies "Unable to switch to organization 1" but I just pushed the rest of what i had to the branch
20:22 pdurbin
great! now I see that JSON file
20:23 pdurbin
pmauduit: any ideas on that "Unable to switch to organization 1" error?
