Time
S
Nick
Message
00:35 icarito[m] joined #dataverse
01:22 rigelk joined #dataverse
06:28 juancorr joined #dataverse
06:57 poikilotherm joined #dataverse
10:04 pdurbin joined #dataverse
10:24 pdurbin
pmauduit: for French speakers like you, a nice 4 minute video about a new installation of Dataverse: https://twitter.com/IST_IRD/status/1167108644695138305
10:25 poikilotherm
Mornin' pdurbin :-)
10:25 pdurbin
poikilotherm: for German speakers like you, two talks by Adam Bien in early November: https://jax.de/serverside-enterprise-java/tipps-tricks-und-workarounds-mit-jakarta-ee-microprofile-slideless/ and https://jax.de/web-development-javascript/web-apps-ohne-frameworks-slideless-nomigrations/
10:42 pdurbin
poikilotherm: mornin'. How are things?
10:43 poikilotherm
Pretty good :-)
10:43 poikilotherm
Working on providing input for https://github.com/IQSS/dataverse/issues/6070
10:43 poikilotherm
Currently getting coverage report for the affected classes
10:44 pdurbin
Cool, speaking of coverage, would you like to help me with https://github.com/IQSS/dataverse/issues/6124 ?
10:44 poikilotherm
I can take a look :-)
10:45 pdurbin
Thanks! My plan is to instrument the war as described at http://guides.dataverse.org/en/4.16/developers/testing.html#measuring-coverage-of-integration-tests
10:49 poikilotherm
Sounds like a plan
10:49 poikilotherm
And sounds like I should do sth for this, too ;-)
10:49 pmauduit
pdurbin: thanks, I'll have a look
10:50 pdurbin
Do you want me to create an issue in dataverse-kubernetes?
10:50 poikilotherm
Sure :-) You could mention the relation to 6124
10:50 pdurbin
pmauduit: thanks, I don't understand a word of it (except "dataverse") but they seem to be saying nice things. :)
10:55 pmauduit
pdurbin: after explaining what a "datastore" software provides, they say that they set up a dataverse at the IRD (french research institute)
10:56 pdurbin
Nice. And they started following me on Twitter. So I sent them a DM and asked if they'd like to have their installation on the map at https://dataverse.org (which they do).
11:03 pdurbin
poikilotherm: ok, I just created https://github.com/IQSS/dataverse-kubernetes/issues/84
11:06 poikilotherm
Thx
11:08 pdurbin
poikilotherm: on a related note, check this out: https://quarkus.io/guides/tests-with-coverage-guide
11:09 poikilotherm
Actually this stuff would be much easier using an embedded server...
11:09 poikilotherm
Which is more or less what you do, when you use quarkus
11:10 pdurbin
yeah
11:10 poikilotherm
This is absolutely possible with Payara, too
11:10 poikilotherm
But not with a custom patched Glassfish 4.1 :-D
11:10 pdurbin
sigh
11:10 poikilotherm
We can go the agent way
11:10 pdurbin
I spent a fair amount of time over my holiday weekend playing with Quarkus. It's fantastic.
11:10 poikilotherm
Looking into that
11:11 poikilotherm
Yeah, I fear it is :-D
11:11 pdurbin
poikilotherm: did you see those talk I mentioned by Adam Bien above? I'm going to guess that he's going to at least mention Quarkus. :)
11:12 poikilotherm
I think he mentioned those in one of his latest podcasts
11:14 pdurbin
Yes! He has a number of episodes where he goes deep on Quarkus, interviewing the creators. Plus he has recorded a number of screencasts about Quarkus. I've been re-listening and watching. :)
11:21 poikilotherm
pdurbin did you take a look at the tcpserver or tcpclient mode of JaCoCo?
11:22 pdurbin
poikilotherm: no, I didn't know those modes exist
11:23 poikilotherm
The maven plugin can grab the IT test coverage stuff from the app server that way
11:23 poikilotherm
In client mode the agent will push it to a remote location
11:24 poikilotherm
Especially for running on K8s this should be a lot easier than fiddling with a file stuck inside a container
11:27 pdurbin
Sounds fantastic. What's the first step?
11:31 poikilotherm
Provide a configuration :-)
11:31 poikilotherm
I'm trying to read up some things about wheter we need instrumentation etc
11:31 poikilotherm
(Most likely we will)
11:32 pdurbin
Whatever works. :)
11:33 poikilotherm
We should definitly think about creating a different Maven profile for this
11:34 pdurbin
poikilotherm: I was wondering where the cleanest place to add it would be. I'm fine with a Maven profile.
11:34 poikilotherm
I think there is already some todo comment about this in pom.xml :-D
11:35 poikilotherm
I found a nice example for the plugin configuration to download the coverage from remote
11:35 poikilotherm
https://github.com/piczmar/maven-jacoco-remote
11:36 pdurbin
Looks like they call the profile "staging". What should we call ours?
11:37 poikilotherm
Whatever sounds good for you :-)
11:38 pdurbin
Well, we have all-unit-tests already. Maybe it should be all-integration-tests.
11:38 poikilotherm
Ok we don't need to handle instrumentation ourselfs
11:38 poikilotherm
https://www.eclemma.org/jacoco/trunk/doc/offline.html
11:39 poikilotherm
We are able to use the agent, so it takes care of this
11:42 pdurbin
From looking at pameyer's script. It looks like we need to copy jacococli.jar into WEB-INF/classes. Does that sound right? I'm taking about instrument_war_jacoco.bash (linked from the guides).
11:44 poikilotherm
Err... the agent should take care of this for us...
11:44 pdurbin
oh :)
11:44 poikilotherm
At least if I understand the JaCoCo docs right
11:45 poikilotherm
Have a look at the offline.html I linked above
11:45 poikilotherm
But beware - I don't think we need to use offline mode
11:45 poikilotherm
Because we do have the option to use the agent jar
11:46 pdurbin
You're saying you'd rather use jacocoagent.jar than jacococli.jar?
11:46 poikilotherm
Yeah
11:47 poikilotherm
Why should we bother about instrumenting?
11:47 poikilotherm
Are there any old issues for this
11:47 poikilotherm
Maybe there are reasons we are not aware of
11:48 poikilotherm
Doing all this instrumentation etc is a lot more work than "just use the agent" and it would be pretty cool if this could be simplified
11:49 pdurbin
I'm looking at "JaCoCo has two modes" at https://quarkus.io/guides/tests-with-coverage-guide#instrumenting-the-classes-instead
11:50 poikilotherm
Yeah
11:50 poikilotherm
Unfortunately, this is incompatible with the dynamic classfile transformations that Quarkus does
11:50 poikilotherm
But we do not use Quarkus for Dataverse (at least not right now)
11:51 pdurbin
:)
11:52 poikilotherm
We can do offline instrumentation if you prefer that...
11:52 poikilotherm
But lets use the maven target then, not the cli jar ;-)
11:52 pdurbin
Right now I just want whatever is easiest.
11:53 poikilotherm
Using the agent *seems* easier
11:53 poikilotherm
I haven't done this before with remote
11:53 pdurbin
Never a dull moment. Always something new. :)
11:54 poikilotherm
Loosing my innocence :-D
11:54 donsizemore joined #dataverse
12:03 pdurbin
poikilotherm: so are you interested in making a pull request for this?
12:05 poikilotherm
I am interested, but let me first finish collecting my infos for #6070
12:05 pdurbin
ok
12:05 poikilotherm
And I would need to test and experiment on this first
12:05 pdurbin
of course
12:17 pdurbin
donsizemore: mornin'
12:41 donsizemore
morning - i'm going to have to beg off tomorrow afternoon - we have a second-round demo of the MVP on a project that pays 50% of my salary
12:41 pdurbin
may the demo gods be with you
12:43 donsizemore
the "real" MVP demo is friday
12:44 pdurbin
gotcha
12:45 pdurbin
donsizemore: I hacked a bit on the grafana dashboard stuff. I forget where we are with prometheus, grafana and friends.
12:55 donsizemore
@pdurbin ansible's crabbing at the "import dashboard" step, but we had MVP demo #1 on friday so that consumed my afternoon
12:56 donsizemore
@pdurbin i pushed everything i have in branch 99 if you want to play along at home
12:57 pdurbin
Right. That's the branch I was playing with. I think we need to tweak the JSON for the dashboard a bit.
12:57 juancorr joined #dataverse
13:01 poikilotherm
pdurbin is this what you expected to see? https://github.com/IQSS/dataverse/issues/6070#issuecomment-527371642
13:03 pdurbin
poikilotherm: it's hard for me to say. I don't reach for Apache Commons stuff very often.
13:03 poikilotherm
You wanted me to provide a list of what needs testing
13:04 poikilotherm
I just saw that I accidentially killed a list
13:04 poikilotherm
Let me restore that
13:04 pdurbin
I said, "I'd like to back up to what problem we're trying to solve here." :)
13:05 pdurbin
donsizemore: when you're ready, I'll explain. :) We need to remove "id" and one other change.
13:06 poikilotherm
pdurbin: its that refactoring game...
13:07 poikilotherm
Get rid of old, unmaintained stuff
13:07 pdurbin
I also wrote, "I'm not personally very enthusiastic about the benefits above (no new features or bug fixes)"
13:07 poikilotherm
I re-added the other list. Must have been dropped due to some caching issue
13:08 poikilotherm
Yeah.
13:08 poikilotherm
I know
13:08 poikilotherm
It's a small change
13:08 poikilotherm
Housekeeping
13:08 poikilotherm
Housekeeping most of the time doesn't pay off directly
13:08 pdurbin
How much effort for QA?
13:09 poikilotherm
IMHO most stuff is about StringUtils
13:09 poikilotherm
66 occurences
13:09 poikilotherm
From what I read in the upstream docs, this did not change
13:10 poikilotherm
So most of this stuff can be changed by replacing it and leave it at that
13:11 pdurbin
I think we're talking past each other. I'm trying to ask about QA effort. QA needs to know what to click to test the change. Or at least which Dataverse features are affected.
13:12 pdurbin
Does that make sense?
13:12 poikilotherm
Yeah.
13:13 poikilotherm
But that's not so easy to gather
13:14 poikilotherm
E.g. StringUtils is used in 66 classes, but 264 calls done to it.
13:14 pdurbin
I know it's not easy. :) It's a lot of effort.
13:14 pdurbin
A lot of cost in developer time.
13:14 poikilotherm
It always doesn't pay of not having Unit tests ;-)
13:15 pdurbin
Yeah.
13:16 pdurbin
On a related note, did you see "Unit tests vs Integration tests" at https://groups.google.com/d/msg/dataverse-dev/k6nZ_icK9fw/Qf6f2-NJEgAJ ?
13:16 pdurbin
the whole email I mean
13:17 pdurbin
I'm sure Gustavo would appreciate any feedback.
13:17 poikilotherm
Sorry I think you sent me the wrong link...
13:18 poikilotherm
This is about controlled vocabs
13:18 poikilotherm
Oh wait a moment
13:18 poikilotherm
Mixed up sth
13:18 poikilotherm
Sry for the noise
13:18 pdurbin
no worries
13:18 poikilotherm
Ah I saw that mail a while ago...
13:18 poikilotherm
I hadn't a chance to reply yet
13:19 pdurbin
Now is a good time to reply. Or at least sometime during this sprint.
13:32 pdurbin
... since this whole sprint (just started) has a focus on improving our automated testing: https://github.com/orgs/IQSS/projects/2#column-5298408 :)
13:35 poikilotherm
Yeah
13:35 poikilotherm
I just posted a reply
13:35 * pdurbin
looks
13:36 pdurbin
great reply, thanks
13:36 poikilotherm
I dunno if I should add the goat...
13:36 poikilotherm
https://www.obeythetestinggoat.com/
13:37 pdurbin
If you feel like it, you could also reply on my "thoughts on automated testing" post at https://groups.google.com/d/msg/dataverse-dev/ISot5k4VjZQ/t-hzPk8tAwAJ
13:38 poikilotherm
Heh, I could just post a link to my other post :-D
13:38 poikilotherm
Automation is cool, but recently there is not much to automate :-D
13:39 pdurbin
Not much to automate? Rly? :)
13:40 donsizemore
@pdurbin i think i'm good on MVP front. jon wants coffee, then i'll pick grafana and jenkins testing back up
13:41 pdurbin
donsizemore: great. I spun up a new server. Want me to send you the ssh key?
13:41 pdurbin
poikilotherm: I feel like there's tons of stuff that should be automated, especially with regard to testing.
13:41 poikilotherm
Yeah, I wasn't precise
13:42 poikilotherm
There are unit tests and some integration tests
13:42 poikilotherm
But the amount and coverage of tests is not very high
13:42 pdurbin
How much coverage do we have for integration tests? (This is a trick question.) :)
13:43 poikilotherm
None currently
13:43 pdurbin
No, we have some.
13:43 pdurbin
And more than we do for unit tests.
13:43 poikilotherm
But no report right?
13:43 pdurbin
right
13:43 pdurbin
That's what I'm asking you to help me with. :)
13:43 poikilotherm
So it's none
13:45 pdurbin
It's not none. You aren't being very precise. :)
13:45 poikilotherm
Ok its n/a ?
13:45 poikilotherm
Unknown
13:45 poikilotherm
Maybe Null
13:45 pdurbin
It was measured once. Months ago.
13:46 poikilotherm
Oh I didn't know that
13:46 poikilotherm
pdurbin do you want to think about things like Testcontainers?
13:47 pdurbin
Eventually. In my thoughts on automated testing I gave my preference of what to work on in what order.
13:47 poikilotherm
Totally fine :-)
13:47 poikilotherm
Let me do a quick test if I can get that agent running...
13:48 pdurbin
Thanks! Meanwhile I've been getting Apache set up to show the Jacoco reports.
13:48 poikilotherm
?
13:49 poikilotherm
Why don't you just expose them via Jenkins?
13:49 poikilotherm
We can have nice little icons from this later
13:49 poikilotherm
And stop using coveralls
13:49 pdurbin
That's a good idea and I'm sure we will eventually. But I don't have Jenkins installed on this box.
13:49 poikilotherm
Ok, that's another story ;-)
13:50 pdurbin
I found where pameyer gave us the one time code coverage number for integration tests. "slightly over 30%" http://irclog.iq.harvard.edu/dataverse/2018-09-06#i_72128
13:51 pdurbin
Which is better than the 19% we have for unit tests on the develop branch right now.
13:54 poikilotherm
Do you think I should add the agent to the development image by default?
13:54 poikilotherm
Most likely that would make sense, right?
13:54 pdurbin
Sure!
13:56 pdurbin
My first thought was to add it to the insecure setup stuff: https://github.com/IQSS/dataverse/blob/v4.16/scripts/api/setup-all.sh#L86
13:56 pdurbin
But I think you're right that it's better to add it at a lower level, better to add it in Maven.
14:07 poikilotherm
*sigh*. pdurbin I just re-ran my stuff and stumbled over that endorsed api stuff again
14:08 pdurbin
sorry, which stuff
14:08 poikilotherm
The thing I reverted for you
14:08 pdurbin
oh
14:08 poikilotherm
As it is not listed as a direct dependency, it needs be handled like a dynamically loaded lib
14:09 pdurbin
sounds like I should pull it out of QA
14:09 poikilotherm
Like it may be necessary for Surefire
14:09 poikilotherm
*sigh*
14:09 poikilotherm
Yeah it is not sufficient as is
14:10 poikilotherm
Forks do have their advantages
14:11 pdurbin
:)
14:11 pdurbin
You can move faster with a fork.
14:12 poikilotherm
Yeah
14:12 poikilotherm
It's like not adding test
14:12 poikilotherm
s
14:12 poikilotherm
Doesn't pay of in the long run
14:13 poikilotherm
Well what shall I do now
14:13 pdurbin
jacoco agent
14:13 poikilotherm
Yeah, I was trying to do so
14:13 pdurbin
cool
14:13 poikilotherm
Then that endorsed stuff bit back
14:13 poikilotherm
Stopping my builds :-(
14:13 pdurbin
I'd love to see https://github.com/IQSS/dataverse-kubernetes/issues/84 under "in focus" :)
14:27 poikilotherm
pdurbin I have about 20 minutes left
14:27 poikilotherm
What shall I do about #6070 and e0646cd
14:27 poikilotherm
Open another issue and PR?
14:31 poikilotherm
Whoops
14:31 poikilotherm
Wrong issue
14:31 poikilotherm
I meant https://github.com/IQSS/dataverse/issues/6132
14:34 pdurbin
poikilotherm: an issue dedicated to the endorsed stuff? Sure
14:35 pdurbin
poikilotherm: but I would love for you to focus on automated testing during this sprint. Like the rest of us are. :)
14:41 poikilotherm
pdurbin shall I put the PR or the issue to Code Review?
14:42 poikilotherm
I will be not around tomorrow - getting new windows in my house
14:42 poikilotherm
I will try to help you with the testing, but I need at least a bit movement on the Auth stuff...
14:44 poikilotherm
https://github.com/IQSS/dataverse/issues/6140
14:45 pdurbin
poikilotherm: PRs should go to code review, please
14:45 poikilotherm
Done
14:45 poikilotherm
That differs from the old process on Waffle
14:45 poikilotherm
I don't have in mind if this is documented... ;-
14:46 pdurbin
There's an issue to document the new process.
14:46 pdurbin
this one: Github Projects Automation and Workflow Changes https://github.com/IQSS/dataverse/issues/5845
14:48 poikilotherm
Ok
14:48 poikilotherm
Gotta go, catch some kids
14:48 poikilotherm
Maybe I am online later this evening...
14:48 poikilotherm
Cu
14:48 poikilotherm
(on Thursday)
15:24 donsizemore joined #dataverse
16:54 donsizemore joined #dataverse
17:38 donsizemore
@pdurbin okay, NOW i'm back on the grafana train. and trying again with 6.3.5 out today
17:38 pdurbin
donsizemore: great! Can I show you those two changes?
17:39 donsizemore
yes yes
17:39 donsizemore
though the branch thinks its up to date
17:40 pdurbin
Yeah, I haven't touched you branch. I can just push a commit if that's easier.
17:40 pdurbin
your*
17:40 donsizemore
also, any particular sub-tests in DatasetsIT you want me to run first?
17:40 donsizemore
commit would be great, fix me fix me
17:40 pdurbin
wasn't there a failing sub test?
17:41 donsizemore
testCreateDeleteDatasetLink - i'll start with it
17:41 pdurbin
cool
17:44 donsizemore
what'd ya fix? what'd ya fix?
17:46 donsizemore
oh, poop. i need to update our testing branch to 4.16
17:46 donsizemore
^fork
17:46 pdurbin
donsizemore: this is what I fixed: https://github.com/IQSS/dataverse-ansible/pull/100
17:47 donsizemore
excellent. gonna try it
17:47 pdurbin
thanks!
17:55 donsizemore
@pdurbin DatasetsIT#testCreateDeleteDatasetLink succeeds by itself
17:56 pdurbin
interesting
17:57 pdurbin
maybe try all of DatasetsIT next
17:59 donsizemore
dey all failed. but let me update the fork first
18:00 pdurbin
ok
19:03 donsizemore
@pdurbin all of DatasetsIT succeeded on current develop
19:04 pdurbin
donsizemore: interesting. I guess I'd suggest that you keep running more and more at a time.
19:04 pdurbin
until you build up the full comma separated list
19:05 donsizemore
they're not consistent, though. i did repeated runs which failed under different tests
19:05 pdurbin
ok
19:05 pdurbin
bummer
19:33 poikilotherm joined #dataverse
19:35 poikilotherm
Good evening gentleman
19:35 poikilotherm
-a+e
19:43 poikilotherm
Anyone still around?
19:56 pdurbin
poikilotherm: hi! Sorry, I was just in tech hours talking about Quarkus.
19:56 pdurbin
donsizemore: can you please ask Akio if he's heard of Quarkus?
19:56 poikilotherm
Hihihi
19:58 poikilotherm
Were they excited? Thrilled?
19:58 pdurbin
Heh. I wish. :(
19:58 poikilotherm
:-D
19:58 poikilotherm
I knew that...
19:59 poikilotherm
Hey, it took about 1 year, now there is a sprint and discussion about testing.
19:59 pdurbin
The feeling in the room was that it will be too hard to migrate Dataverse to Quarkus. Specifically it was asked if EJBs are supported and I don't believe they are. But I thought I heard CDI can be used in many places where EJB is used. Do you know anything about this?
19:59 poikilotherm
Well you more or less will need to replace EJBs with CDI
20:00 poikilotherm
Spring uses CDI only
20:00 poikilotherm
I think I read some stuff about this transistion
20:00 poikilotherm
https://developers.redhat.com/blog/2019/04/12/migrating-java-applications-to-quarkus-lessons-learned/
20:01 poikilotherm
There is a section "code changes", where some details are shared
20:02 pdurbin
yeah, that's the post I'm referring to in a follow up email
20:02 pdurbin
I didn't know Spring uses CDI only.
20:08 poikilotherm
At least with Java EE 7 EJBs were still easier to use over RMI
20:09 pdurbin
we don't use RMI
20:09 pdurbin
I used it in a class once.
20:09 poikilotherm
And I am not sure if you have role based security in Java EE 8 with CDI
20:09 pdurbin
I don't know. We don't use role based security.
20:09 poikilotherm
Spring has this, of course, but that is outside Java EE
20:10 pdurbin
Yeah, we implemented our own security, our own roles and whatnot.
20:10 donsizemore
@pdurbin woo woo, further: KeyError: 'grafana_url' (which is supposed to be valid)
20:11 pdurbin
huh
20:11 pdurbin
donsizemore: weird
20:11 pdurbin
poikilotherm: what do you think? Should I have hope that Dataverse could move to Quarkus eventually?
20:12 poikilotherm
pdurbin: dunno. Maybe cut out stuff first to make it lighter
20:13 poikilotherm
The bigger the monolith, the more burden who get for such a HUGE transistion
20:13 poikilotherm
This should definitely be possible
20:13 poikilotherm
Although some things might need big refactoring
20:13 pdurbin
I still think the first step is to get off Glassfish 4.1.
20:14 poikilotherm
I am not familiar enough with the complete codebase to actually give serious answer...
20:14 poikilotherm
Yeah, that sounds like a good plan :-D
20:14 pdurbin
I guess I was hoping that by playing with Quarkus, we would at least start being able to use MicroProfile APIs such as the "config" API . And maybe that would help motivate us to get off Glassfish 4? I dunno.
20:14 poikilotherm
I don't know if JSF is working OK with Quarkus
20:15 poikilotherm
Maybe one should discuss moving the UI to webcomponents
20:15 poikilotherm
And only use the excellent already present REST API
20:19 pdurbin
I'm definitely interested in Web Components. It would be nice to use a standard like that instead of thinking about this or that new Javascript framework.
20:20 poikilotherm
Definitely
20:20 poikilotherm
But I don't know how easily one could get IQSS devs and community into using it
20:21 pdurbin
yeah
20:21 poikilotherm
Changing the UI from JSF to Webcomponents and lit html might be too much
20:21 pdurbin
yeah
20:22 pdurbin
poikilotherm: but what if we rewrote https://github.com/IQSS/dataverse-installations from Django to Quarkus+Web Components? Are you with me? :)
20:22 poikilotherm
I would be with you if I had more time :-D
20:22 pdurbin
yeah
20:22 pdurbin
I would be with me too.
20:23 poikilotherm
LOL
21:09 poikilotherm
pdurbin: need to get some sleep now...
21:10 poikilotherm
2300 over here and I need to be on construction site by 7am
21:10 poikilotherm
Cu
21:19 donsizemore joined #dataverse
21:48 pdurbin joined #dataverse
