Time
S
Nick
Message
01:13 sivoais joined #dataverse
06:57 juancorr joined #dataverse
10:21 stefankasberger joined #dataverse
12:25 donsizemore joined #dataverse
12:31 pdurbin
mornin' all
12:39 donsizemore
@pdurbin morning! my vagrant run succeeded after i left yesterday; testing in EC2 now and we may have a grafana
12:44 pdurbin
donsizemore: fantastic! collectd too?
12:44 donsizemore
we'll see in a minute =)
12:48 pdurbin
nice
13:04 donsizemore
well, we're closer http://ec2-54-227-218-175.compute-1.amazonaws.com/grafana/d/COPs6vKZk/overall-metrics?orgId=1
13:04 donsizemore
TASK [dataverse : create grafana datasource] *********************************** changed: [localhost] => {"changed": true, "id": 1, "msg": "Datasource prometheus created : Datasource added", "name": "prometheus"}
13:09 donsizemore
poop. i see what happened.
13:10 pdurbin
oh good
13:10 pdurbin
I was going to ask pmauduit to take a look :)
13:10 pmauduit
oh ansible :)
13:10 donsizemore
this will not be a "production-ready" install but can easily be tweaked. just want to get it up for IQSS' testing
13:11 pdurbin
pmauduit: I think donsizemore is right. We're close!
13:12 donsizemore
@pmaudit i can go back and parameterize stuff; but i think the main interest now is in monitoring development branch performance
13:12 pmauduit
ok, i'd have liked to have a bit more time to help you though, as I did some ansible fiddling on other topics recently
13:12 donsizemore
@pmauduit i'm always looking for my 'round tuit
13:12 pmauduit
"round tuit" ?
13:13 donsizemore
american pronunciation on the latter: "when i get a round to-it"
13:13 pdurbin
pmauduit: a "round tuit" is from Perl, I think. sivoais might know. :)
13:13 pmauduit
ok :)
13:13 donsizemore
i used to have one. a wooden oversized coin labelled "tuit"
13:48 donsizemore
so we're getting memory but not CPU http://ec2-34-228-225-233.compute-1.amazonaws.com/grafana/d/COPs6vKZk/overall-metrics?orgId=1
13:52 juancorr joined #dataverse
14:07 pdurbin
donsizemore: having memory working is amazing!! Great stuff!
14:07 donsizemore
well, i can't find what the problem is with the CPU monitor
14:07 donsizemore
but if you want the memory stuff up front i can merge and y'all can start on the JMX bits
14:10 pdurbin
donsizemore: sure! Want to move the memory graph to the top?
14:11 donsizemore
i think i found it. capital "P"
15:09 pdurbin
heh
15:09 pdurbin
I just sent a link to Slava. Of the Grafana chart.
15:20 donsizemore
@pdurbin http://ec2-52-90-73-196.compute-1.amazonaws.com/grafana/d/COPs6vKZk/overall-metrics?orgId=1 =)
15:20 donsizemore
@pdurbin i can merge that branch and we can make the JMX stuff another issue, or we can keep going in the same branch?
15:56 pdurbin
donsizemore: merge it, merge it!1!! :)
16:51 donsizemore
Done. I'll open issue #100(!) to track the JMX stuff?
17:10 pdurbin
donsizemore: sure!
17:40 donsizemore
@pdurbin p.s. I installed the Jenkins Test Results Analyzer plugin to address #3 on my to-do list from last week's testing meeting https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-testSubset/test_results_analyzer/
17:41 pdurbin
donsizemore: wow, very fancy. And we're planning on calling you on time today. Sorry about last week.
17:41 donsizemore
@pdurbin i have to beg off today (I thought I e-mailed?) - double-booked with the project that pays half of my salary
17:42 pdurbin
Oh! Yes, I think you mentioned this yesterday in IRC .
17:42 pdurbin
Do you mind dashing off an email too?
17:49 pdurbin
donsizemore: great email! Thanks!
17:57 donsizemore
@pdurbin ooh, check out https://jenkins.dataverse.org/job/IQSS-dataverse-develop/test_results_analyzer/ (expand edu.harvard.iq.dataverse.api)
17:58 pdurbin
donsizemore: it's so beautiful I want to cry. T-T
17:59 pdurbin
when you get a fancy job in silicon valley doing this stuff, please don't forget about us
18:06 donsizemore
10 years ago i was in love with the bay area.
18:43 donsizemore joined #dataverse
20:38 pdurbin
My brother moved out there recently. I plan to get out there every one in a while.
20:51 pameyer joined #dataverse
20:52 pameyer
pdurbin: it sounded like integration test coverage was on the menu
20:52 pdurbin
pameyer: yes! I'm working on https://github.com/IQSS/dataverse/issues/6124
20:53 pdurbin
Can you ssh into centos
20:53 pdurbin
oh, whoops
20:53 pdurbin
sorry, sorry
20:53 pdurbin
can you ssh into centos
20:54 pameyer
looks like
20:54 pameyer
is aws the target executation environment for 6124?
20:54 pdurbin
great! can you please `su -i` to root and then `su - glassfish`?
20:54 pdurbin
for now, yes
20:55 pdurbin
Are you going to tell me I should be testing in Docker? :)
20:55 pameyer
no
20:55 pdurbin
phew :)
20:55 pameyer
but `sudo su ; su -i glassfish` works
20:55 pdurbin
ah
20:55 pameyer
ah; was answering a different question ;)
20:56 pameyer
if you get the paths sorted out, it _shouldn't_ matter which execution environment - but making it generic would be trickier
20:56 pdurbin
Yeah. I'm worried I have the paths wrong. I'm trying to follow http://guides.dataverse.org/en/4.16/developers/testing.html#measuring-coverage-of-integration-tests
20:57 pameyer
well, that was accurate when I wrote it ;)
20:57 pdurbin
http://guides.dataverse.org/en/4.16/_downloads/instrument_war_jacoco.bash has jacoco-0.8.2 in it but I'm using jacoco-0.8.1 because I had to downgrade us shorting after you wrote that.
20:57 pameyer
using the script to do war instrumentation makes me think that using the jacoco api to post the war to glassfish won't work; unless the method too large stuff is fixed
20:58 pdurbin
Otherwise I'm following the steps.
20:58 pameyer
re-reading it, step 2 is ambigious :(
20:58 pameyer
it meant put jacocoagent.jar in the glassfish lib, then deploy the dataverse war
20:59 pameyer
but I could see somebody trying asadmin deploy jacocoagent.jar
20:59 pameyer
do those steps make sense to you?
20:59 pdurbin
No, I got that part ok, I think.
20:59 pdurbin
In /root/prepare-glassfish.sh I cp /home/glassfish/local/jacoco-0.8.1/lib/jacocoagent.jar /usr/local/glassfish4/glassfish/lib
20:59 pdurbin
Actually, I couldn't deploy the instrumented war file without that step. :)
20:59 pameyer
me either
20:59 pdurbin
I'm not surprised. :)
21:00 pdurbin
I'm stuck on the last step.
21:00 pameyer
is that because it doesn't work anymore, or because it's incomprehensible?
21:00 pdurbin
Can you please look at /home/glassfish/dataverse/run-api-tests.sh ?
21:01 pdurbin
I cd to /home/glassfish/dataverse first. Then I ./run-api-tests.sh
21:02 donsizemore
@pameyer!
21:02 pdurbin
You'll see several commented out missteps. Me messing the the paths. My confusion is probably palpable.
21:02 pameyer
@donsizemore - hi
21:02 pdurbin
donsizemore: should we show pameyer Prometheus and Grafana measure Dataverse stuff? :)
21:02 pameyer
@pdurbin reading
21:02 pdurbin
measuring*
21:03 pdurbin
donsizemore: actually, let's not distract him... yet!
21:04 pdurbin
pameyer: part of what that script does is rsync all of "target" to http://ec2-3-81-78-209.compute-1.amazonaws.com/target/ ... at the end I mean.
21:05 pameyer
I wouldn't be shocked if that script threw some errors trying to generate the report
21:05 pdurbin
me neither
21:06 pameyer
I _think_ that it wants the un-instrumented class files for some reason
21:06 pameyer
do you have stdout/stderr around?
21:06 pdurbin
You're welcome to go ahead and run it.
21:07 pdurbin
I'm not running the whole test suite.
21:07 pdurbin
just a bit of InfoIT, I think
21:07 pdurbin
so the whole script should take maybe 20 seconds or so
21:07 pameyer
well, let's find out
21:07 pdurbin
thanks!!
21:08 pameyer
yup; fun with exceptions
21:09 pdurbin
oh?
21:09 pameyer
rat0.err
21:09 pameyer
fiddling it slightly, seeing if it'll help
21:09 * pdurbin
crosses fingers
21:10 pameyer
hopefully not killing glassfish
21:10 pameyer
no lynx on that box :(
21:10 pdurbin
You're welcome to yum install whatever you want.
21:10 pameyer
but there are coverage reports
21:11 pdurbin
but 0% coverage in those reports
21:12 pameyer
so there are :(
21:12 pameyer
do ITs work with no baseurl now?
21:14 pdurbin
well, localhost:8080 is the default
21:14 donsizemore
:8084?
21:14 pameyer
ok; so it's not that
21:14 pdurbin
8084 is what docker-aio uses
21:15 pdurbin
yeah, the test seems to execute ok, InfoIT, I mean
21:16 pameyer
size 0 data file makes me think something's up with the instrumenting
21:17 pdurbin
To instrument, I ran your script. :)
21:17 pameyer
well, size 0 and modifcation time from a few hours ago. restarting glassfish on general principles
21:18 pdurbin
perfect
21:18 pdurbin
Shall we run a `mvn clean` too?
21:18 pameyer
may not matter
21:19 pameyer
already more than size 0 after a restart; let's see what the tests do
21:19 pameyer
ah - makes sense
21:19 pameyer
data file isn't getting flushed until a gf restart
21:20 pameyer
congradualtions pdurbin - InfoIT has 4% coverage
21:20 pameyer
... and I can't spell
21:20 pdurbin
where where?
21:21 pameyer
lynx -nocolor target/coverage-it/index.html
21:21 pameyer
well, I only read the top-line - but it's non-0 from that run
21:24 pameyer
Total 303,791 of 317,504 4% 28,85
21:30 pdurbin
sorry, my wifi is acting up. I see green! Thanks, pameyer!!
21:32 pdurbin
And yes, 4%. Interesting that simply getting the version of Dataverse via API exercises 4% of our code. :)
21:32 pameyer
glad it helped!
21:32 pdurbin
So to be clear, what did you do? :)
21:33 pdurbin
You didn't have to mess with this line, right? java -jar ${JACOCO_HOME}/jacococli.jar report --classfiles target/classes --sourcefiles src/main/java --html target/coverage-it/ /usr/local/glassfish4/glassfish/domains/domain1/config/jacoco.exec
21:33 pameyer
three things ; so not 100% sure which changed it
21:33 pameyer
changed to use the non-instrumented classes, glassfish restart, and set dataverse.test.baseurl
21:33 pameyer
but baseurl was to the default; so that's almost certainly not it
21:34 pameyer
glassfish restart was definately necessary; but I'm less sure about switching to non-instrumented classes for the report generation
21:34 pdurbin
sorry, are you talking about the line above? You changed --classfiles?
21:35 pameyer
yup
21:35 pdurbin
Is the line above how it should be? That's what you used?
21:35 pameyer
it was pointing to the glassfish one
21:35 pdurbin
ok
21:35 pdurbin
I tried a bunch of variations.
21:35 pameyer
if I'm remembering correctly, when I tried report generation with instrumented classes something was unhappy
21:36 pdurbin
I changed something too. I confused myself with all my rsyncing.
21:36 pameyer
which doesn't make much sense, so it may be inaccurate
21:36 pameyer
well, take a quick snapshot before it changes again ;)
21:36 pdurbin
I'm going to get this set up on another box. :)
21:36 pdurbin
And extra document it. :)
21:36 pdurbin
thanks again and sorry that this was way more than 10 minutes :)
21:37 pameyer
well, sounds like it got things unblocked :)
21:37 pdurbin
yes!
21:37 pdurbin
thank you!
21:37 pameyer
no problem - glad to help
21:38 pameyer
and now I'm out for the night .... I should stop in every now and then though
