Time
S
Nick
Message
07:32 Virgile joined #dataverse
08:36 juancorr joined #dataverse
09:49 Virgile joined #dataverse
12:45 Virgile joined #dataverse
13:14 donsizemore joined #dataverse
13:23 donsizemore
@poikilotherm @pameyer it patches the installer because most of the summer develop wanted payara but "release" wanted glassfish4 https://github.com/GlobalDataverseCommunityConsortium/dataverse-ansible/blob/master/tasks/dataverse-install.yml#L30
13:25 donsizemore
now that the dust has settled, i'd imagine we still want to grab updated everything when building from a branch
14:53 donsizemore joined #dataverse
15:09 pdurbin joined #dataverse
15:17 poikilotherm
Hey pdurbin :-)
15:17 pdurbin
hey
15:27 poikilotherm
I talked to Jim about introducing configuration of S3 access credentials via configuration of the app in addition to the profiles.
15:27 poikilotherm
Of course I would like to do this via MPCONFIG
15:29 poikilotherm
What do you think would be a good scope for a PR down the road? Should it include the mpconfig libs in pom.xml or should I create a separate pr for this? Just introducing the libs doesn't do anything in the app. We need to use the API before that... ;-)
15:30 poikilotherm
(I meant if the PR for changing config of S3AccessIO should include the details about introducing MPCONFIG to the codebase)
15:40 pdurbin
I was just thinking about how we have 3 configuration options now: JVM options, database settings, and MicroProfile Config. Yes, we should have a bit of a strategy about when to use each. Maybe a pull request to the dev guide that gives some guidance to devs?
15:53 pameyer joined #dataverse
15:55 pameyer
@donsizemore - thanks, that makes sense
15:56 pameyer
@pdurbin - the property files are kind of configuration too, right?
15:58 pdurbin
I don't think they're supposed to be but yes, in a place or two we have some config stuff, maybe even a regex, in a properties file.
15:59 pdurbin
pid.allowedCharacters=^[A-Za-z0-9._/:\\-]*
16:00 pdurbin
Odd to me that it's mixed in to Bundle.properties with otherwise only text.
16:01 pameyer
those, but also (maybe) trying to use languages that don't have corresponding property files
16:02 pameyer
.... this tangent was me thinking that your idea of a strategy for what config options go where was good one and going off the rails
16:03 pdurbin
Well, language configuration is a good point. Something to consider.
16:26 poikilotherm
pdurbin MPCONFIG relies on and reuses the system properties. And we can create a configsource to eat the database settings. MPCONFIG primarily opens up that you can do configuration in multiple spaces.
16:26 poikilotherm
pdurbin consolidation would mean a big bang thing. Time shows that you folks don't like that. So I would try to do this step by step, refactoring where it fits.
16:27 pdurbin
Multiple spaces? What do you mean?
16:28 poikilotherm
pameyer pdurbin good catch with those configs in Bundle files! Easy to migrate those to the microprofile-config.ini (already in the codebase) and change code as necessary to use MPCONFIG for retrieval. Get's much easier to override that way...
16:29 poikilotherm
pdurbin: spaces: maybe wrong term. Better use "sources". You can have multiple sources for config data, mix'n'match as the are ordered by an ordinal.
16:31 poikilotherm
One of those sources is system properties, another is env vars, another is a directory, a default one is the microprofile-config.ini inside the WAR
16:31 pdurbin
Ok, and maybe env var always "win". Higher precedence.
16:32 poikilotherm
So retrieving a configuration value for a property called "foo.bar.example" is much more flexible that way
16:32 poikilotherm
https://docs.payara.fish/community/docs/5.2020.6/documentation/microprofile/config/README.html#config-sources
16:33 poikilotherm
Plus the default ones: https://github.com/eclipse/microprofile-config/blob/master/spec/src/main/asciidoc/configsources.asciidoc#default-configsources
16:33 poikilotherm
So system property wins due to ordinal 400 over env var with ordinal 300
16:33 poikilotherm
(You can change ordinals)
16:35 pdurbin
Ok, I get tht ConfigSources is a thing. Thanks.
16:35 poikilotherm
OMG 5.3 on the horizon... https://github.com/IQSS/dataverse/issues/7439
16:36 poikilotherm
Hopefully I can get everything merged before the window closes... O.O
16:37 pdurbin
poikilotherm: did you change the password for dataverseAdmin? admin1 doesn't work for me.
16:38 pdurbin
Oh, it's admin. I think this is normal behavior when you run the installer. Nevermind.
16:39 poikilotherm
Yes, the normal password is admin :-D And I didn't touch anything in that area...
16:39 pdurbin
Is src/main/resources/META-INF/microprofile-config.properties used? I'm asking because is has dataverse.db.name=dataverse but I'm using a database called "dvndb" like usual.
16:40 poikilotherm
pdurbin any inside info about when you want to cut the edge for the 5.3 release?
16:41 poikilotherm
Yes it is used
16:41 poikilotherm
But you can override
16:41 poikilotherm
Just configure the system properties as usual
16:42 poikilotherm
src/main/resources/META-INF/microprofile-config.properties is just providing some default values
16:43 poikilotherm
I wrote some bits about it in the release notes docs I added for 7418
16:43 pdurbin
So the installer must be setting a system property for dvndb.
16:44 pdurbin
Yeah. <system-property name="dataverse.db.name" value="dvndb"></system-property>
16:44 poikilotherm
https://github.com/IQSS/dataverse/blob/aa1cc0feb4b18f10e5e014085ea662c925ee4b99/scripts/installer/as-setup.sh#L157
16:46 poikilotherm
I choose to go with "dataverse" to make it easier to use in cloud instances, where you deploy the postgres server etc. Classic installations should be fine with setting the property.
16:48 pdurbin
Why is "dataverse" easier? I'm wondering. But I'm fine with it. Not trying to fight you. :)
16:49 poikilotherm
I simply dump the database name inside the postgres container as an env var. Creates the database, user etc for me. dvndb is using old names from good old DVN 3.x series ;-)
16:50 poikilotherm
BTW pdurbin would you like to see the option to use a properties file for the miriards of options Dataverse has?
16:51 poikilotherm
We could create a ConfigSource inside the app and read from a single file.
16:51 poikilotherm
That might be interesting for classic installations :-D
16:52 poikilotherm
(That file could for example file inside /etc/dataverse/config.properties)
16:52 pameyer
will changing the db name break all existing installations, or make it harder for them to upgrade?
16:53 poikilotherm
s/file/live/
16:53 pdurbin
Well, one thing I'm confused about is how to make use of the MicroProfile Config stuff from a classic installation. An etc file sounds fine. But you're saying you'd need to add that?
16:53 poikilotherm
pameyer: part of the upgrade process will be to set a system property. So this will work fine :-)
16:54 poikilotherm
pdurbin: yeah. There is no out-of-the-box configsource for that.
16:54 poikilotherm
Dunno if Payara Ltd would accept a PR. Just today I fixed their directory config source...
16:59 pdurbin
poikilotherm: why did you add <include>**/*.properties</include>?
17:02 poikilotherm
pdurbin otherwise no default values.
17:03 poikilotherm
Without it, the microprofile-config.properties would not be included in the WAR.
17:05 pdurbin
Ok, and you definitely want default properties? Is it crazy talk to delete that file? Not that I want to but I'm being devil's advocate and wondering if less is more.
17:20 poikilotherm
You want it. Otherwise no easy way to define useful defaults. You can do in code, but not in annotations.
17:20 poikilotherm
Will be used for more brown the road.
17:21 poikilotherm
Keep and use.
17:21 poikilotherm
Next use: mail session definition.
17:22 poikilotherm
Smart move moving the regex pameyer mentioned above in there.
17:49 pdurbin
poikilotherm: ok but I think we need some docs about microprofile-config.properties
17:51 donsizemore joined #dataverse
18:40 pdurbin
poikilotherm: another thought is to only document that file for devs. Maybe people reading the Installation Guide don't need to know about it.
19:24 pdurbin
poikilotherm: Is there some kind of automatic mapping between DATAVERSE_DB_NAME and dataverse.db.name? This is a MicroProfile Config API thing?
19:29 pdurbin
I think I found the answer at https://github.com/eclipse/microprofile-config/blob/2.0/spec/src/main/asciidoc/configsources.asciidoc#environment-variables-mapping-rules
19:29 pdurbin
The COM_ACME_SIZE example.
19:31 poikilotherm
Pdurbin: yes, the env vars are transformed because . is no valid char in them...
19:31 poikilotherm
Some magic in the background
19:31 pdurbin
For devs I think we should explain the magic.
19:32 poikilotherm
Pdurbin dev guide sounds reasonable. Do you think this should be done in this PR or should we just get things going for now and stick with #7000 etc for more details on. MPCOnFIG
19:32 pdurbin
I could go either way. I'm not going to hold this up for dev docs. But clearly there is some magic going on. :)
19:33 pdurbin
What's going on with the change from jdbc/VDCNetDS to java:app/jdbc/dataverse? Is this a necessary change?
19:34 poikilotherm
Pdurbin I don't think so regarding the magic. This isn't important for them, they deal with mpconfig in code. Did you see my explanation in the new properties about using ENV var for it?
19:35 poikilotherm
I agree on explaining the basic concepts and link to resources and where the properties file comes from etc in the dev guide
19:35 pdurbin
Sorry, what explanation? I'm looking.
19:36 poikilotherm
The JNDI name change was necessary. Couldn't make it to work with the old name, resource not found.
19:36 poikilotherm
I think I wrote about this in the commit message.
19:37 pdurbin
I don't usually look at commit messages. I look at docs and comments. :)
19:37 poikilotherm
Explanation: installation guide, config options. How to override from env var
19:38 pdurbin
Oh. yes, I saw that.
19:39 pdurbin
Is it even possible for me to override with an env var from my classic setup?
19:40 poikilotherm
That might be possible if you make it happen for the payara user profile via export. Otherwise, the appserver will not pick it up.
19:40 pdurbin
Ah, when I run asadmin start-domain, you mean. Yeah, that might work.
19:41 pdurbin
Hey these commit messages are great.
19:43 poikilotherm
I'm always trying to put effort in those so it gets easier when devs use git blame
19:43 poikilotherm
Why did this change happen etc
19:44 poikilotherm
This is hard to track in docs
19:44 pdurbin
Sure, I try to do this too but no one seems to notice. :)
19:46 poikilotherm
Oh. I do. Lots of stuff during refactoring is ending in git blame for me. And often it does not become clear why things happened when it says 'refactiring.' when you're lucky you might find an issue number n:-D
19:46 pdurbin
So the biggest problem is that this is failing: https://jenkins.dataverse.org/job/IQSS-Dataverse-Develop-PR/view/change-requests/job/PR-7422/
19:46 pdurbin
Don and I are working on it.
19:47 pdurbin
I guess I could send the pull request to QA before we fix it.
19:47 pdurbin
But nothing is going to get merged tomorrow or Friday anyway because of the holiday.
19:48 pdurbin
Oh, you asked about the timing of 5.3 didn't you?
19:54 poikilotherm
Yeah.
19:54 poikilotherm
I hope to get all of this refactoring done for 5.3
19:54 poikilotherm
So we can finally upgrade container images
19:56 pdurbin
My guess is that Danny created the 5.3 README issue because you're stuff is moving along so well. He knows this is the last piece. So I wouldn't worry. And I don't think 5.3 will be released until mid December. And probably won't be deployed to Harvard Dataverse until the new year.
19:58 poikilotherm
Thank God. Then I have chances that Payara 5.2020.7 gets released on time
19:58 poikilotherm
It would be so cool to get this stuff done finally. Has been two years :-D
19:59 poikilotherm
And it looks like they will accept my PR for the config source \o/
19:59 pdurbin
Nice.
19:59 pdurbin
And after all this you'll update dataverse-kubernetes to 5.x? :)
20:00 poikilotherm
YES!!!! FINALLY!!!! \O/
20:00 pdurbin
:)
20:00 pdurbin
🎉
20:03 poikilotherm
The Payara Upgrade Work is finally paying of
20:04 pdurbin
absolutely
20:05 pdurbin
plus we are no longer embarrassed to be on glassfish 4.1 :)
20:07 pdurbin
poikilotherm: you don't want sysadmins editing microprofile-config.properties do you?
20:07 poikilotherm
They can't. It's in the WAR.
20:08 poikilotherm
Ok they can in exploded, but it would be overwritten on upgrade
20:08 pdurbin
Well, you'd be surprised how often people replace class files, xhtml files, etc.
20:08 pameyer
do folks deploy war's they don't build?
20:08 pameyer
usually, I mean
20:08 poikilotherm
Yikes
20:08 pdurbin
We do it quite a bit in prod here.
20:09 pdurbin
pameyer: I would say yes. They probably treat the war like a binary. They trust it.
20:10 pdurbin
poikilotherm: anyway, I was thinking about removing microprofile-config.properties from the installation guide so sysadmins aren't tempted to edit it.
20:12 poikilotherm
Pdurbin that file isn't in there...
20:12 poikilotherm
Usually people won't see it as admins
20:12 poikilotherm
It's a dev thing
20:13 pdurbin
Right but you documented it in a non-dev area, the installation guide.
20:13 pdurbin
You say stuff like this: Defaults to localhost (via microprofile-config.properties).
20:14 poikilotherm
Ah ok. Now I see.
20:15 poikilotherm
Ok happy to remove the file mention
20:15 pdurbin
I can do it if you don't mind. I'm also changing e. g. to e.g. (no space).
20:16 poikilotherm
Go ahead :-)
20:16 poikilotherm
Sorry for typos and bad language.
20:16 pdurbin
it's fine
20:17 pdurbin
just little things
20:18 pdurbin
I'll probably rearrange things a bit too. dataverse.db.name is at the bottom.
20:18 pameyer
a little behind the conversation - but I appreciate having good commit messages too
20:21 poikilotherm
Pdurbin go ahead! Happy you're investing so much time :-)
20:22 pdurbin
Well, I didn't test upgrading at all. Just a fresh dev environment.
20:22 pdurbin
So I hope upgrading works.
20:24 poikilotherm
I'm confident :-D
20:27 poikilotherm
Ok pdurbin let me do a quick list of things I would like to see done for 5.3 that are still open: 1) move mail session to annotation and fix the fromaddress issue. 2) move java messaging config to annotations. 3) make s3 client configurable from config not only from profile 4) move other password for doi etc to support retrieval via Microprofile config 5) merge container creation to Dataverse repository
20:28 pdurbin
Oh dear, that's a lot.
20:29 poikilotherm
Do you think we can push all this into 5.3 when I provide the code within the next week?
20:29 pdurbin
Well, we're basically shut down until Monday. Danny is cooking already.
20:29 poikilotherm
It sound bigger than it is.
20:30 poikilotherm
Yeah that why I would try to push the code so you folks have plenty of time till cutting the release mod december
20:30 poikilotherm
S/mod/mid/
20:30 pameyer
winter recess is a little longer than usual this year
20:30 pameyer
so not as much dec to work with
20:31 pdurbin
yep, love it
20:33 donsizemore joined #dataverse
20:36 pameyer
I'm not going to complain about it ;)
20:41 donsizemore
@pameyer each of my co-workers know my catch-phrase: "the servers don't turn off"
20:47 pameyer
@donsizemore very true
21:04 pdurbin
Ok folks, I'm taking off. Big holiday here. See you Monday.
21:04 pdurbin left #dataverse
21:14 pameyer
pdurbin: have a good thanksgiving
