Time
S
Nick
Message
00:26 jri joined #dataverse
04:27 jri joined #dataverse
05:52 nanoz joined #dataverse
08:06 jri joined #dataverse
08:23 MrK joined #dataverse
08:39 MrK
Hi, me and me friend wanna work on some new features and I was wondering if we could be added to the contributor list and our issue assign to us?
09:26 poikilotherm joined #dataverse
10:26 nanozz joined #dataverse
11:06 pdurbin
MrK: hi! I'm looking through issues. How do you feel about this one? https://github.com/IQSS/dataverse/issues/5137
11:08 poikilotherm
Morning pdurbin :-)
11:08 poikilotherm
Sorry I couldn't make it to the community call yesterday (evening for me)
11:10 pdurbin
poikilotherm: no problem. During tech hours we talked about deps and payara.
11:13 poikilotherm
Oh cool :-) Did you guys came up with something or still discussing things?
11:25 msy_inist joined #dataverse
11:31 pdurbin
poikilotherm: Gustavo and Matthew are going to talk more about which application server to target.
11:34 poikilotherm
Ok
11:34 poikilotherm
Then I'll wait for whatever comes up from that...
11:34 poikilotherm
In the mean time, I'm starting to get my hands dirty with dependencies...
11:35 poikilotherm
There is plenty of room for housekeeping business...
11:35 poikilotherm
Should I look more into that or what was the result of tech hours for that?
11:40 pdurbin
I guess people were wondering how you were planning to keep dependencies up to date when you put them in the a "lib" directory in Glassfish.
11:41 poikilotherm
Everytime the container is rebuilt, the deps are updated from what is defined within the maven project
11:41 poikilotherm
Then the new container image needs to rolled out on docker/kubernetes and of course containers need to restart
11:42 poikilotherm
But that's the hole thing about the uber/fat/hollow jar topic...
11:43 poikilotherm
I kinda ship around that by sticking with the WAR concept of deployments, but adding the dependencies on another level
11:44 poikilotherm
By that you still need the application server. Getting rid of that is even more to the fat/uberjar side
11:44 poikilotherm
(Because you actually package both deps AND app server within the JAR)
11:44 pdurbin
Sure, there are multiple dimensions here.
11:45 pdurbin
The feeling in the room was that we're happy for you to experiment with stuff.
11:45 poikilotherm
The nice thing is: by sticking with the WAR concept, the maven build is not that much different for container and "classic mode"
11:45 poikilotherm
Ok
11:47 poikilotherm
Then I will stick with fixing the dependency stuff for AWS by using BOMs for AWS and Jackson and stop there, right? Or shall I continue with some more dependency convergence conflicts?
12:09 MrK
pdurbin: Oh sry I wasn't precise, we are from the branch of University of Warsaw and we wanted to commit to the project in order to use it for our own requirements. Since dataverse has quite a lot of features we thought that why not just join and add the features we need, since it will benefit everyone. Would it be possible to assign the issue to the creator? https://github.com/IQSS/dataverse/issues/5280 .
12:14 pdurbin
MrK: oh! Great! So you have a specific feature in mind. That's fine but our design team would probably want to give their opinion on that feature. I wouldn't want you to spend hours and hours coding it up and then hear that it's not what they want.
12:15 pdurbin
MrK: the design team is currently working on this issue that relates to the one you opened: https://github.com/IQSS/dataverse/issues/3404
12:16 pdurbin
MrK: so I would suggest leaving a comment on #3403 introducing yourself and linking to your suggestion at #5280
12:16 pdurbin
MrK: I hope that makes sense. I don't want to discourage you! I can try to remember to mention you during the design meeting this afternoon.
12:18 pdurbin
poikilotherm: I'm kind of thinking you should talk to Gustavo and Matthew, to be honest. Again, our main interest is not being stuck on an a specific version of Glassfish from years ago.
12:20 pdurbin
MrK: I should add that users could really use a way to organize their files so I'm glad you're thinking about this!
12:21 MrK
pdurbin: ah no worries we are doing research how the process looks so it's good to know, I'm also gonna have to talk to our decision makers if they are fine with it since in the future there are going to be deadlines :D. I was also wondering if ofc it's even possible to predict, how much time does it usually take to review the new feature we would like to add?
12:23 pdurbin
Well, it really depends on the feature. Generally speaking, API only features are less controversial. Features that don't affect the UI or UX, I mean.
12:23 pdurbin
MrK: Do you have ideas for other features?
12:24 pdurbin
It's kind of sad but sometimes users resort to naming their files 0foo.txt and 1.bar.txt to keep them in the order they want. :/
12:27 MrK
pdurbin: Right now we have one more which is a bit more controversial, it's pretty much an ability to apply additional CC licenses.
12:28 pdurbin
MrK: funny you should mention that because there's been some chatter about that in Slack within the last week or so.
12:29 pdurbin
It's less controversial than you think. I could probably dig up some recent mockups with multiple CC license options.
12:30 poikilotherm
pdurbin: do they usually hang around on your slack channel? willing to go over there and talk to them, but most certainly I will need an account.
12:31 pdurbin
poikilotherm: sure, the #dv-design channel is quite active
12:31 MrK
oh interesting, that's great we have the plans how it should look, so it would be great to coordinate and chat about details
12:31 pdurbin
MrK: can you see the Google doc linked from the most recent comment at https://github.com/IQSS/dataverse/issues/1990 ?
12:32 poikilotherm
What do I need to get inside that channel? Google doesn't spit out much about where and how to get access...
12:32 MrK
pdurbin: oh so on this slack channel there are people from the design team?
12:33 pdurbin
MrK: right, they have their own channel
12:33 poikilotherm
Drop a mail in case you don't want that to be logged publically
12:33 poikilotherm
;-)
12:34 MrK
pdurbin: sadly I need to reques the access in order to view the file
12:34 pdurbin
MrK: ok, I would suggest leaving a comment on the issue asking if that Google doc can be opened up.
12:36 pdurbin
poikilotherm: I don't know what the process is for people joining Slack but I guess you could try emailing supportIRC because it's free. :) And based on an RFC . :)
12:37 pdurbin
MrK: leaving a comment on https://github.com/IQSS/dataverse/issues/1753 would be good too.
12:37 poikilotherm
You are aware about free plans for OSS?
12:37 pdurbin
poikilotherm: buh, I'm more of an IRC guy
12:38 poikilotherm
*G* I like the capabilities like image/site preview, markdown support etc.
12:38 poikilotherm
And there is a IRC bridge IIRC
12:38 pdurbin
poikilotherm: no, Slack recently killed their IRC and XMPP gateways
12:38 poikilotherm
But I'm fine with both
12:38 pdurbin
poikilotherm: how do you feel about Gitter?
12:38 poikilotherm
Oh sad to hear that
12:39 poikilotherm
Gitter is far from usable. Used it for a project. Unstable UI, unusable apps losing messages and notifications, overall a very unpleasant XP
12:40 poikilotherm
Mattermost and Rocketchat seem to be a good alternative to Slack
12:40 poikilotherm
We use Rocketchat here at FZJ and it keeps getting attraction
12:40 pdurbin
poikilotherm: huh, ok. Gitter works fine from its IRC gateway for me. And I recently installed the Android app and it seems to work well enough. Gitter is free and open source and good for transparency in open source, which Slack is not.
12:41 poikilotherm
Sure :-) That's why we choose Rocketchat ;-)
12:41 pdurbin
Can Rocketchat have public logs like Gitter?
12:42 * poikilotherm
reads docs
12:43 pdurbin
The nice thing about Gitter is that public logs are the default. It's part of the culture of using Gitter.
12:44 poikilotherm
You can create public channels in Rocketchat and give write permissions to just a bunch of people
12:45 MrK
what's the slack adress for dataverse?
12:45 pdurbin
poikilotherm: cool. Can you create one so I can see how it looks?
12:45 poikilotherm
Most likely the broadcast channels should be a good match
12:45 poikilotherm
https://rocket.chat/docs/user-guides/channels/#creating-channels
12:45 poikilotherm
Sorry - our Rocketchat is on the intranet only due to regulation issues like GPDR etc
12:45 poikilotherm
Will take time to solve that
12:46 poikilotherm
Oh and I am not sure if you can see the channel without a user account
12:46 poikilotherm
But of course you can hook it to something like Github and the like
12:50 poikilotherm
Riot looks promising too - world readable chat is a baked in feature
12:51 poikilotherm
Oh that's based on Matrix...
12:51 poikilotherm
https://matrix.org
12:51 pdurbin
MrK: https://iqss.slack.com
12:52 pdurbin
poikilotherm: does RocketChat have an IRC gateway?
12:53 poikilotherm
IIRC yes
12:54 jri joined #dataverse
12:54 pdurbin
cool
12:55 poikilotherm
Jepp, there are reports about people using it, but it is still a "work in progress"
12:56 pdurbin
I'm bummed you don't like Gitter.
12:57 MrK
pdurbin: is it possible to join the slack? In sign up form I see you need an harvard email. Also I had a question about changing schema, if you guys change anything in schema, you are just including it in upgrade sql and release it with new version?
12:59 poikilotherm
I had some bad experience back in April with it... I would be glad to try again...
12:59 pdurbin
MrK: you could try asking at support
12:59 poikilotherm
pdurbin: just did that... ;-)
12:59 pdurbin
poikilotherm: Gitter is hiring. Maybe it'll be better in the future. :)
13:04 jri joined #dataverse
13:16 poikilotherm
pdurbin: current outcome of my research about the maven dependency problems... https://github.com/poikilotherm/dataverse/commit/0b19800fd699fe8159478629c7b9da2170b3e5b4
13:19 poikilotherm
Maybe I should a word or two about dependency management to the dev docs, too
13:49 poikilotherm
+write
13:49 poikilotherm
Something is wrong in my head today. Skipping words...
13:54 isullivan joined #dataverse
14:02 donsizemore joined #dataverse
14:16 poikilotherm
Hey pdurbin... Just found an excellent case of why that dependency management is important... Moved from JaCoCo 0.7.5... to 0.8.2 and removed it from the direct dependencies. That lead to errors about missing libraries for Dataverse code, that has been available via the transitive dependency before...
14:20 poikilotherm
Whoops. You relied on edu.emory.mathcs.backport.java.util.Collections within GitHubOAuth2AP.java and util.json.JsonPrinter
14:21 poikilotherm
That most certainly was by accident
14:24 poikilotherm
And I fixed two places using org.codehaus.plexus.util.StringUtils instead of org.apache.commons.lang.StringUtils
14:24 poikilotherm
Most certainly these two packages have been around because of the direct dependency to JaCoCo
14:25 poikilotherm
And then during coding the IDE offered it as a quick import
14:35 nanoz joined #dataverse
14:45 pdurbin
poikilotherm: yeah Netbeans keeps suggesting that package from emory. It's annoying. I don't know if we even use it.
14:46 poikilotherm
I hope Netbeans only will offer it because of the former direct JaCoCo dependency
14:47 poikilotherm
At least IDEA analyzes the Maven POM and offers stuff you define as depenendcy in there
14:48 pdurbin
Well, Netbeans is definitely operating off the POM. But I don't know how it decides what to suggest.
14:49 pdurbin
poikilotherm: your commit above makes sense. Does it break anything? Or change how we deploy the war file? Do you want to go ahead and make a pull request?
14:50 poikilotherm
Do you think I should split up? We talked about that a few days ago.
14:50 poikilotherm
Splitting is not difficult, I'm just unsure.
14:51 poikilotherm
What do you prefer?
14:51 poikilotherm
I hadn't a chance to try agains some S3 yet
14:51 pdurbin
smaller chunks are almost always better
14:51 poikilotherm
Maybe you guys can give this a shot?
14:52 pdurbin
It's great when developers let QA know if they have concerns about any particular functionality, such as S3 or anything else.
14:53 poikilotherm
Should I add some docs about the maven deps management first?
14:54 poikilotherm
Maybe it makes sense to create a successor to my AWS issue describing a more general dependency cleanup
14:54 poikilotherm
Actually I made the first steps towars this...
14:54 pdurbin
Yes, a new issue might be best. And yes to docs. Always docs, if you're willing to write them!
14:55 poikilotherm
Sure. Any suggestions about a place within the dev docs? Browsed through them, but unsure where this might fit best
14:56 pdurbin
Hmm. http://guides.dataverse.org/en/4.9.4/developers/tips.html maybe?
14:56 pdurbin
The tips page is kind of a dumping ground. :/
14:56 pdurbin
http://guides.dataverse.org/en/4.9.4/developers/coding-style.html maybe? A new section about Maven?
14:57 pdurbin
Or a new page?
14:57 pdurbin
A new page called "dependency management"?
14:58 poikilotherm
Let's make a new page... Tips and Code Style seem to cover other topics, but this is definitly something noteworthy to any developer
14:58 poikilotherm
Thus I think it deserves its own entry in the TOC on the left
14:59 pdurbin
Sure, please go for it.
15:00 pdurbin
Eventually we might need some nesting in that sidebar.
15:01 pdurbin
MrK: at standup in about an hour I'm planning on mentioning how you'd like to contribute. Thanks again for the offer!
15:09 MrK
pdurbin: oh thank you! Would it be possible to mention an access to slack so there would be an possibilty to talk with the design team? Btw. There are 2 developers from us assigned to this project at the moment.
15:20 poikilotherm
pdurbin: https://github.com/IQSS/dataverse/issues/5288
15:23 pdurbin
poikilotherm: cool. Thanks! I don't know what a BOM is.
15:24 pdurbin
oh, Bill Of Materials
15:25 poikilotherm
And to be precise: https://github.com/IQSS/dataverse/pull/5289
15:26 pdurbin
poikilotherm: thanks! Dragged to the "Community Dev" column at https://waffle.io/IQSS/dataverse
15:26 poikilotherm
:-)
15:27 poikilotherm
Maybe you can assign matthew, scolapasta and me?
15:27 poikilotherm
More workload on people in charge... :-P
15:27 pdurbin
They aren't in the office yet.
15:27 poikilotherm
Oh! Alright...
15:27 pdurbin
I'll mention it at standup in 45 minutes.
15:28 poikilotherm
Thank you very much! :-)
15:28 pdurbin
sure, I have a list of three things to bring up
15:28 pdurbin
If anyone has anything else, please let me know. :)
15:28 poikilotherm
Lets see if I can add more to that... :-D
15:28 poikilotherm
What's already on the list?
15:29 pdurbin
1. MrK and another developer want to contribute features with UI/UX impact.
15:29 pdurbin
2. AccessIT is failing: https://github.com/IQSS/dataverse/issues/5287
15:30 pdurbin
3. Oliver wants to talk to Gustavo and Matthew about upgrading Glassfish, deps, etc.
15:30 poikilotherm
About Nr. 2: isnt that one of the tests that fail randomly?
15:31 pdurbin
No, I'm hoping it's just the Solr schema change I added.
15:31 poikilotherm
OK
15:31 * poikilotherm
crosses fingers
15:32 pdurbin
:)
15:32 poikilotherm
Oh about the Community Dev label: maybe that was a bit to quick? I dunno if matthew and scolapasta might prefer to do this on their own, as it is kinda reaping out body parts replacing it with fresh stuff
15:33 MrK joined #dataverse
15:33 poikilotherm
I'm not sure if the trust an outside community dev enough for this
15:33 pameyer joined #dataverse
15:34 poikilotherm
This is no prio for me, my primary goal is still #5274
15:39 pdurbin
Oh, I thought 5288 was something you wanted.
15:40 poikilotherm
Sure - I will need it as a base for 5274
15:41 poikilotherm
But if there is no interest in the rest of 5287, I'm happy to leave it "as is"
15:41 poikilotherm
By the way: the WAR is at 125MB now... ;-) -21MB, yeah!
15:42 donsizemore joined #dataverse
15:45 pdurbin
I don't think you mean 5287.
15:45 poikilotherm
Argh.
15:45 poikilotherm
You are right
15:46 poikilotherm
The rest of 5288
15:46 pameyer
pdurbin: I'd thought there was another one of the ITs failing too; but from a quick look don't see the issue. any ideas if that one was fixed?
15:48 pdurbin
pameyer: you might be thinking of https://github.com/IQSS/dataverse/issues/5260
15:49 pdurbin
poikilotherm: yeah, I'm not sure. I trust you three will decide what's important to you. I'm interested in the ability to move off of the old version of Glassfish we're stuck on.
15:49 pameyer
pdurbin: yes; thanks
15:52 pdurbin
poikilotherm pameyer: for the record, the AccessIT thing was just the Solr schema. Thanks for crossing fingers. Sorry for the noise.
15:54 pameyer
nice to see I'm not that sometimes opens issues where the resolution is "requestor got mixed up" :)
15:56 pdurbin
I forgot I even messed with the Solr schema.
15:56 pdurbin
But I just emailed dataverse-dev to spread the word.
15:56 pameyer
you managed to include it in docker-aio, so it passed there
16:03 pdurbin
sure, docker-aio is smarter about setting up solr automatically than we developers are on our laptops :)
16:06 donsizemore joined #dataverse
16:25 poikilotherm
Somebody asked about security scans at community call... Maybe https://projects.eclipse.org/proposals/eclipse-cognicrypt can help out with scanning for crypto security?
16:42 isullivan joined #dataverse
16:50 pdurbin
poikilotherm: maybe, pameyer has some tools he uses too
16:50 pdurbin
poikilotherm: I let those guys know you want to talk app servers and deps
16:55 pameyer
pdurbin poikilotherm there are some security scanning tools I use; but with my paranoia level I'm not a bit fan of going into too much detail on those in a world-visible publicly logged forum
16:56 pdurbin
pameyer: excellent point. poikilotherm and others, if you find vulnerabilities, please report them to security
16:57 poikilotherm
Yes Sir! :-)
16:58 poikilotherm
Thx for mentioning at stand up :-)
16:59 poikilotherm
I'm back on work on 4172 and getting the docker based stuff inside the pom. It is such a blessing not to worry about the hacky stuff anymore - that gave me some headaches...
17:01 pdurbin
Cool, for what it's worth nobody seemed to object to Payara at tech hours yesterday.
17:04 poikilotherm
Glad to hear that :-)
17:05 pdurbin
:)
17:13 nanoz joined #dataverse
17:29 poikilotherm
pdurbin: IIRC you where the one in charge to write the install scripts, right?
17:30 poikilotherm
Why has the postgres JAR dependency been managed outside of the POM?
17:30 poikilotherm
Wouldn't it make sense to include that?
17:30 poikilotherm
Obviously there is some historic reason to that...
17:40 jri joined #dataverse
17:41 pameyer
there used to be several postgres JARs bundleded with the installer, and which one was used with glassfish depended on the postgresql version
17:42 pameyer
then the postgres jdbc driver supported all the postgresl server/client versions dataverse was targeting, and there wasn't a need for multiple jars
17:43 poikilotherm
Alright - and why has this not moved inside the WAR?
17:43 poikilotherm
Wouldn't that be easier for maintenance?
17:44 poikilotherm
(Sometimes new versions of the jdbc JAR are released for various reasons. That will never be updated or only when the install script is re-run on an update of dataverse)
17:44 poikilotherm
Packaging it inside the WAR might be easier then, isn't it?
17:51 poikilotherm
Alright guys, that's it for me for today... Cu all tomorrow :-)
17:51 poikilotherm
Good night / Good afternoon ;-)
18:27 donsizemore joined #dataverse
18:30 pdurbin
It wasn't me! :)
18:48 pdurbin
... though I did add "use strict" :)
19:29 donsizemore joined #dataverse
20:12 jri joined #dataverse
20:29 isullivan joined #dataverse
20:48 JonathanNeal joined #dataverse
21:14 pameyer
@poikilotherm something that only occured to me later - the install script copies the postgres jar before war deployment; and some of the glassfish configuration to create the database connection pool happenes before war deployment.
21:16 pameyer
I don't have a lot of expertise w\ glassfish / j2ee connection pools, but I think that if the postgres was in the war, creating the connection pool would fail. and I wouldn't be surprised if deploying the applicaiton required the connection pool to exist (because if not, that configs could have been done elsewhere)
23:23 poiki_at_home joined #dataverse
23:24 poiki_at_home
pameyer: just read your comment about the connection pools. Maybe this was true for Glassfish 3 and 4, but at least with 5 this gets a lot easier: https://blog.payara.fish/an-intro-to-connection-pools-in-payara-server-5#id- [MARK-434]AnIntroductiontoConnectionPoolsinPayaraServer-CreatingtheConnectionPool-Manuallyviadomain.xml
23:24 * pdurbin
hopes poiki_at_home gets enough sleep tonight ;)
23:25 pameyer
@poiki_at_home good to hear
23:25 pameyer
and yeah, downtime is good :)
23:25 poiki_at_home
Ah damn, took the wrong section
23:25 poiki_at_home
https://blog.payara.fish/an-intro-to-connection-pools-in-payara-server-5#id- [MARK-434]AnIntroductiontoConnectionPoolsinPayaraServer-CreatingtheConnectionPool-PerApplicationResources
23:26 poiki_at_home
I really would like to see less asadmin commands when fiddling around with containers...
23:26 poiki_at_home
Makes the building process much easier ;-) Just deploy and go go go
23:27 pdurbin
What's the alternative? XPath madness?
23:27 pameyer
less asadmin would definately be better
23:28 pameyer
it would need code changes, but moving stuff to environmental variables still seems like a not-horrible idea (at least, for where it can be done)
23:29 pdurbin
I have no quarrel with environment variables. :)
23:30 pameyer
my turn to head out for the night ;)
23:31 poiki_at_home
Nope, the alternative is configure the JDBC stuff from inside the application
23:32 pdurbin
Sounds fancy.
23:33 poiki_at_home
http://dplatz.de/blog/2018/self-contained-jee-app.html
23:33 poiki_at_home
That's a pretty good summary of how to get there ... :-)
23:33 poiki_at_home
Payara and Wildfly lets you reuse ENV vars within annotations
23:33 poiki_at_home
Thats really cool stuff
23:34 poiki_at_home
Maybe this is an option for those other config parts, too? (Need to dig deeper into that...)
23:34 poiki_at_home
No more fiddling with asadmin... Ah, what a dream
23:35 poiki_at_home
Mainly because of asadmin not being idempotent
23:36 poiki_at_home
Oh and OmniFaces did some fancy stuff, too ;-) http://jdevelopment.nl/switching-data-sources-datasourcedefinition/
23:36 pdurbin
As long as we don't go back to a bunch of replaces with sed on an XML file, I'm fine with whatever.
23:38 pdurbin
Those were dark days.
23:39 poiki_at_home
NO WAY!
23:40 poiki_at_home
I will stand by your side, fighting everyone trying to get over that wall
23:40 poiki_at_home
That area has been sealed... ;-)
23:41 poiki_at_home
Alright - 00:40 over here. Finally heading for my bed now, more stuff tomorrow.
23:41 poiki_at_home
Cu all
23:41 * pdurbin
hands poiki_at_home a halberd
23:42 poiki_at_home left #dataverse
23:48 pdurbin
huh, this dplatz solution does look pretty good
