Time
S
Nick
Message
09:35 juancorr joined #dataverse
11:14 juancorr joined #dataverse
12:25 yoh joined #dataverse
12:28 donsizemore joined #dataverse
14:23 pdurbin joined #dataverse
14:23 pdurbin
Happy Monday, everyone.
14:48 poikilotherm
Hi pdurbin
14:48 poikilotherm
What happened?
14:49 poikilotherm
You folks are up and running early
14:49 poikilotherm
Didn't your timezone move to winter time yesterday, too?
14:50 pdurbin
not until nov 1
14:52 poikilotherm
Oh!
14:53 poikilotherm
That explains pretty much everything :-D
14:53 poikilotherm
pdurbin as you connected us on Slack about the other issue, did you see https://github.com/IQSS/dataverse/pull/7355#issuecomment-715567839 ?
14:54 pdurbin
That one is about halfway in my unread email.
14:55 poikilotherm
:-D
15:02 pdurbin
poikilotherm: now I'm looking at https://github.com/IQSS/dataverse/issues/5292#issuecomment-715532237 and wondering about "It would be a great relief not to include the bootstrap shell scripts but make Dataverse do it themself." Are you saying we would rewrite our Bash scripts into Java?
15:03 poikilotherm
Not necessarily all of 'em
15:03 poikilotherm
I wrote this 2 years ago
15:04 poikilotherm
Things like reference_data.sql are not necessary with Flyway available
15:05 poikilotherm
And setup-all.sh might be replaced by a self-bootstrapping Java class...
15:05 poikilotherm
So you don't have to run any bootstrapping scripts manually
15:06 poikilotherm
I know that setup-all.sh does a lot of stuff
15:06 poikilotherm
I'd stick with the most essential things
15:06 poikilotherm
Creating the superadmin, a root dataverse, ...
15:07 poikilotherm
#7355 isn't about removing any of this :-D It's just about reference_data.sql
15:07 poikilotherm
The only reason why Dataverse docker images contain a psql command...
15:12 pdurbin
Yeah, I agree that Flyway is a natural place to put the reference data SQL script.
15:18 pdurbin
I'm confused by "no more installing Python Postgres nor psql in containers necessary". There are other psql commands in the installer (to create the dvndb database, for example).
15:22 poikilotherm
Yeah. I don't need those. The Postgres container creates it for me :-D
15:23 pdurbin
Do you run the installer?
15:30 poikilotherm
Parts of it
15:30 poikilotherm
Like setup-all.sh
15:31 poikilotherm
The installer does loads of things not usefull/necessary/possible in a container environment.
15:31 pdurbin
yeah
15:32 pdurbin
And when you do development (your regex groups PR, for example), you do it in a container?
15:32 poikilotherm
Yeah :-)
15:33 pdurbin
you do this: http://k8s-docs.gdcc.io/en/v4.20/development/index.html
15:33 poikilotherm
Yeah :-D
15:34 poikilotherm
Some things might be a bit more advanced now that I'm adding the containers to the upstream project.
15:34 poikilotherm
Container creating is much faster that way
15:34 poikilotherm
s/ting/tion
15:35 pdurbin
Do you recommend that I switch to your way?
15:37 poikilotherm
That's a though question
15:37 poikilotherm
People not used to containers might find it hard to go that way
15:38 poikilotherm
There's a learning curve
15:38 pdurbin
My next question was going to be what you recomend for new contributors. :)
15:39 poikilotherm
If they are familiar with containers, go for it.
15:39 pdurbin
sure
15:40 poikilotherm
I try to make the container experience very smooth
15:40 poikilotherm
Like create the containers from a Maven command
15:41 poikilotherm
A dev doing development with Dataverse is most likely familiar with that
15:41 poikilotherm
There are some very promising projects like Eclipse JKube
15:41 poikilotherm
They try to take some of that brain load off the devs
15:42 pdurbin
always appreciated
15:44 poikilotherm
Making Dataverse bootstrap itself when deployed has some very neat consequences
15:44 poikilotherm
For example it would be much easier to fire up an ephemeral environment
15:45 poikilotherm
So people could create the docker images from maven
15:45 poikilotherm
And put them in docker or k8s
15:46 poikilotherm
Automatically, ideally
15:46 poikilotherm
And then just start the three containers, ready to serve
15:48 pdurbin
What if we started using containers in our automated testing? Might be a good way to get more familiar with containers.
15:50 poikilotherm
That's where I am heading to :-D
15:51 pdurbin
We tried to user docker-aio but ultimately switched to spinning up EC2 instances.
15:51 poikilotherm
Testcontainers for the win :-)
15:52 pdurbin
sure
16:03 poikilotherm
Folks I'm outta here.
16:04 poikilotherm
Read you on my mobile...
16:04 poikilotherm
Plz lemme know what you think about pointing people to https://docs.payara.fish/community/docs/5.2020.5/documentation/user-guides/upgrade-payara.html for instructions on updating their payara installation
16:42 pdurbin
I don't have a strong opinion about it. We aren't used to upgrading Glassfish very often. :)
16:57 donsizemore
for the record, "We tried to user docker-aio but ultimately switched to spinning up EC2 instances." == integration tests were failing for unrelated reasons.
17:53 pdurbin
donsizemore: yeah, but I'm not sure we want to use docker-aio if we try containers again. Maybe something fancier like TestContainers.
17:58 donsizemore
sounds good to me
19:01 nightowl313 joined #dataverse
19:07 pdurbin
poikilotherm (or anyone): did you have thoughts on " scripts for rebuilding a dev environment (with sample data) #7256 "? I see you referenced in from a couple issues. I just picked it up and my plan was to start by getting the old phoenix rebuild scripts working.
19:15 poikilotherm
Yeah, my thoughts are listed above :-D
19:16 poikilotherm
Make Dataverse build itself
19:16 pdurbin
and rebuild itself? :)
19:16 poikilotherm
It would actually be fairly easy to let Flyway load another SQL dump containing the sample data.
19:17 poikilotherm
But that might not even be necessary
19:18 pdurbin
I guess but what the scripts do now is different. And they do more. Drop the database, run setup-all, etc.
19:18 poikilotherm
Easing the deployment would be a good start
19:21 pdurbin
this is what the rebuild script does now: https://github.com/IQSS/dataverse/blob/v5.1.1/scripts/deploy/phoenix.dataverse.org/rebuild
19:21 poikilotherm
Maybe adding a few simple containers like for Postgres and Solr would already be a benefit for devs
19:21 pdurbin
...then it runs setup-all.sh here: https://github.com/IQSS/dataverse/blob/v5.1.1/scripts/deploy/phoenix.dataverse.org/post
19:21 poikilotherm
IDEs these days have nice support for running Payara inside them
19:21 poikilotherm
But you still need a database and index.
19:22 poikilotherm
Using a container for that makes less headaches than installing manually and running it.
19:24 pdurbin
We (still) aren't there yet. Using containers in dev. So I'll probably just focus on what delivers me value: getting these old scripts working with Payara.
19:34 poikilotherm
I'm sorry pdurbin but what would setup-all.sh do for you that couldn't be done by a Java class?
19:34 poikilotherm
It loads the metadata blocks, adds the built-in provider and creates dataverseadmin and root Dataverse. It adds the solr endpoint and blocks some API endpoints.
19:34 poikilotherm
And why would you run a drop database when you would simply stop and delete the container and fire up a new one?
19:36 pdurbin
All good ideas but out of scope for the current effort. :)
19:36 poikilotherm
In case of testing, all the management should done by the testing framework anyway...
19:36 poikilotherm
So what's left is how to get a setup for manual tests and initial production deployment going.
19:36 poikilotherm
Didn't you talk a lot about that great web apps in PHP that have a nice installer?
19:36 poikilotherm
Getting this stuff into Dataverse would be first step for such a thing...
19:37 pdurbin
yeah, the famous 5 minute install from wordpress
20:15 yoh joined #dataverse
20:45 bjonnh joined #dataverse
20:47 pdurbin left #dataverse
21:18 nightowl313 joined #dataverse
21:29 pameyer joined #dataverse
21:38 nightowl313
anyone know if DVUploader requires Centos 7 or 8? Trying to run it on Centos 6.6 server and getting "Bad Request {"status":"ERROR","message":"Failed to add file to dataset."}” error. Java 1.8 is running (via environment modules system).
21:41 nightowl313
i can put a ticket into github, but asking here first in case anyone happens to know =)
21:41 pameyer
@nightowl313 bad reqest should be a server-side error
21:42 nightowl313
the server we are running DVUploader from? Or our DV site?
21:42 pameyer
ah - DV site
21:42 pameyer
I have an irrational suspision that the issue may be expired SSL certificates on your cent 6.6 box
21:42 nightowl313
lol should have guessed that .. the same command runs from a different server
21:43 nightowl313
looking at logs ... should have done that
21:43 pameyer
if you run `curl -I https://$your_dataverse_host/api/info/version`, does that work or complain about certs?
21:44 nightowl313
run it from the centos 6.6 server? it works
21:45 pameyer
ok - so not that
21:45 nightowl313
centos 6.6 is on the server we are uploading from ... our DV server is centos 8
21:45 pameyer
I'd been thinking possible expired certs on the centos 6.6 system
21:46 nightowl313
don't get errors in the DV log other than "Failed to add file to dataset.|#]"
21:48 pameyer
it might be worth looking at the glassfish/payara server.log - I'm guessing that "bad request" is http 400 from the server, so there might be more info about why it failed there
21:48 pameyer
... but this is a long way of saying, "nope, I don't know" ;)
21:48 nightowl313
oh no I didn't scroll up far enough, there are some errors ... a SEVERE payara error
21:49 nightowl313
ohhhhhhhhhh the file name apparently has an unsupported character in it! hahahahahahaha
21:49 pameyer
successful troubleshooting :)
21:50 nightowl313
i'm glad you all here don't judge lol
21:51 nightowl313
although I did try the same file on a different system and it uploaded ... ubuntu
21:51 pameyer
it might be different default LANG / LC_* environmental variables
21:53 nightowl313
yea omg ... changing the name and trying again ... worked perfectly!
21:53 pameyer
great!
21:53 nightowl313
thanks so much!
21:53 pameyer
you're welcome :)
21:53 nightowl313
one day I will help with something!
21:54 pameyer
that's how it goes
21:55 pameyer
re: judging - https://github.com/IQSS/dataverse/issues/4661 might have some entertaining history :)
21:55 pameyer
not sure how to link to the bit of history where I changed the title
21:57 nightowl313
haha lol!
21:58 nightowl313
this is a fun group!
22:04 pameyer
:)
22:04 pameyer
I'd _guess_ if you compared `printenv | grep LC` or `printenv | grep LANG` on the system where it worked, and the system where it didn't, you'd see a difference
22:18 nightowl313
ah ... they are the same, but I just realized that the file that the user sent to me to try a test on a different server didn't actually have the colons! He must have downloaded it to Windows and it removed the colons!
22:19 nightowl313
i should have noticed that, or at least thought of it ... =| ... live and learn
22:20 nightowl313
however, I am currently a HUGE fan of DVUploader! such a help in this situation
22:23 pameyer
finding cases where things don't work right is pretty useful - that's the first step to fixing them
22:23 nightowl313
=D
22:23 pameyer
good that you found the difference - I wouldn't have thought that colons in filenames woule be a problem
22:24 pameyer
... of course, using colons in a filename seems odd to me, so I'm probably not the best test case for that
22:24 nightowl313
thanks to the really descriptive info in the payara log!
22:25 nightowl313
yea, the system generating all the data files writes them with the date and time (with colons), apparently
22:25 nightowl313
weird
22:29 pameyer
not the way I'd do it, but I can see how it makes sense
