Time
S
Nick
Message
08:21 jri joined #dataverse
09:47 MrK joined #dataverse
10:34 poikilotherm joined #dataverse
11:49 pdurbin
morning, all
11:49 poikilotherm
Good morning :-)
11:50 pdurbin
poikilotherm: I'm not using an IRC bouncer but I do run an IRC cliet (weechat) in a screen session in my home server.
11:50 poikilotherm
:-)
11:51 poikilotherm
I am heading for lunch...
11:51 poikilotherm
Cu later
11:51 pdurbin
bon appetit
11:53 pdurbin
MrK: did you see the "This looks promising overall" comment at https://github.com/IQSS/dataverse/pull/5349#pullrequestreview-180848867 ?
11:57 MrK
pdurbin: Ah yeah :P
11:59 pdurbin
He's asking if we need to rename our old scripts, etc.
12:04 MrK
Yeah I gotta answer, I'm just struggling with docker atm, but I think I'm gonna take a break.
12:04 pdurbin
thanks!
12:04 pdurbin
and sorry to hear about your docker struggles
12:06 MrK
Yeah I hoped it would be easy but replacing just a war is not that simple, I was doing workarounds like they use solr schema/config which is in the folder instead of unpacking from war (??) and using language bundle the same way instead of unpacking from war (??). But even fixes and using my war for some reason I can't upload files there is just a single <div> missing.
12:31 poikilotherm
Re :-)
12:33 poikilotherm
MrK while reading your answer to landreev: wouldn't it be an option to create a custom MigrationResolver that adds the necessary bits based on the knowledge what the prior running Dataverse version has been?
12:33 poikilotherm
That way you could reuse the SQLs made by landreev for updating
12:34 poikilotherm
But let Flyway takeover instead of using a script to get to a certain point
12:35 MrK
poikilotherm: Hmm you mean extracting the info about version and based on that execute the sql scripts? Yeah that's also an solution.
12:36 poikilotherm
YEs
12:37 poikilotherm
I dunno if this version is conserved somewhere except strings. In that case a upgrade script would be needed, but it is just about conserving the version number
12:37 poikilotherm
Maybe pdurbin or pameyer know sth about that
12:45 MrK
Ah right, recently the script was added so i would not need to execute sql scripts one by one.
12:52 pdurbin
MrK: ok, so "baseline" is whenever you start with flyway. Thanks.
12:53 MrK
pdurbin: Yeah if you dont have database schema(new project) you just don't use it.
12:53 MrK
the baseline
12:53 pdurbin
no baseline for a new project
12:53 pdurbin
ok
12:54 pdurbin
I don't know. It sounds like I should read some flyway docs. :)
12:55 pdurbin
Huh. Flyway creates a table called "flyway_schema_history"? I'm looking at https://flywaydb.org/getstarted/how
12:55 MrK
Yeah thats how it tracks which sql failed/ was executed before
12:55 pdurbin
"Immediately afterwards Flyway will begin scanning the filesystem or the classpath of the application for migrations. They can be written in either Sql or Java."
12:56 pdurbin
It sounds like we could make Flyway scan the file system for our old manual migration scripts.
12:57 MrK
Yeah but it would not be compatible since if 2 people upgrade from different version the sql will fail since there will be duplicates.
12:58 MrK
but poikilotherm suggested some good solution.
12:58 poikilotherm
;-)
12:59 poikilotherm
Thx for the flowers
12:59 MrK
we won't have to use flyway before all old sql scripts will be executed
12:59 MrK
xD
13:00 MrK
no problem
13:01 MrK
ahh yeah we will have to add old sql scripts to war but that's not a problem.
13:01 poikilotherm
Oh it would be really charming to use these SQL scripts from Flyway to support instant upgrading on deploying new version
13:01 pdurbin
Let's see what Leonid and others say in the issue or pull request. We should keep the conversation there. I mean, we can discuss here too but please don't expect them to read the IRC log.
13:01 andrewSC joined #dataverse
13:02 poikilotherm
MrK before I forget this: once we have Flyway in place, we should really try to get https://github.com/IQSS/dataverse/blob/develop/scripts/database/reference_data.sql replaced with a proper migration
13:02 pdurbin
4 hours until the community call: https://dataverse.org/community-calls
13:02 poikilotherm
At least the indexing stuff
13:03 poikilotherm
The inserts might be a good thing to do in bootstraping
13:03 poikilotherm
(See my issue https://github.com/IQSS/dataverse/issues/5361)
13:03 MrK
Yeah I see it that way, first check if the version is 4.9.5( with flyway support) if not use the script which will upgrade to 4.9.5 and then use flyway. That's during startup
13:04 poikilotherm
Oh IIRC that script from landreev compiles a specific SQL script
13:05 poikilotherm
Takes the changes between versions and mixes them together based on source version and target version
13:05 poikilotherm
I dunno if we should execute this script from within the code...
13:06 poikilotherm
IMHO it would be cleaner to use the SQL changes scripts and place those in migrations to run via custom resolver
13:19 pdurbin
poikilotherm: oh, the answer is yes, we have a different call in method today: https://groups.google.com/d/msg/dataverse-community/BxKl7Qt914I/OneAOeYIBgAJ
13:20 poikilotherm
I already saw that and tried it :-) Linux + Firefox = unsupported, also the docs state otherwise. Chrome + Linux seems to work ok
13:20 pdurbin
phew
13:20 poikilotherm
Thx for pointing out :-)
13:21 pdurbin
poikilotherm: do you want part 2 of the feedback on your glassfish/paraya post?
13:21 poikilotherm
Yeah sure :-) Go ahead
13:21 pdurbin
Ok, for context, here was part 1: http://irclog.iq.harvard.edu/dataverse/2018-11-30#i_80663
13:22 pdurbin
I guess we're on the "philosophy beyond container images" section.
13:23 pdurbin
You talk about what the container images for Dataverse should be like.
13:23 pdurbin
But we've never ever done this. Never made container images.
13:23 pdurbin
I'm the only one on the team who has pushed images to Docker Hub.
13:23 pdurbin
No one else even has access to push to "iqss" on Docker Hub.
13:24 pdurbin
I gave access to some Red Hat intern and their mentor.
13:24 pdurbin
They might still have access.
13:24 pdurbin
All this is to say that anything under "iqss" on Docker Hub should be regarded as highly experimental.
13:24 pdurbin
I don't know what I'm doing.
13:24 pdurbin
And I'm the one who created the "iqss" org on Docker Hub.
13:25 pdurbin
So I appreciate any tips you have. :)
13:26 pdurbin
When you say "usable directly from upstream releases" I assume you mean that we should be pushing images to "latest" on Docker Hub. Dataverse releases track the "master" branch, so I guess at release time, when we merge "develop" into "master" and tag a release, we should be pushing an image to Docker Hub.
13:26 pdurbin
I guess I want to back up.
13:26 pdurbin
I don't think IQSS should necessarily have to do this.
13:27 pdurbin
If poikilotherm or Slava or Craig or others want to do this under their own non-iqss namespace on Docker Hub, they should feel free.
13:27 pdurbin
I say this because I can't imaging developers at IQSS doing this any time soon. Most of us don't even have Docker installed.
13:28 pdurbin
But anyone, someone who wants all this Docker goodness will be pushing images to Docker Hub, I assume. They will be pushed at release time. They will be pushed to "latest". I assume.
13:29 pdurbin
Next you say "stateless"... you've already discovered that our EJB timers are not stateless. The Red Hat interns worked on this a bit within OpenShift. Only having one master timer server when you start up multiple Glassfish.
13:31 pdurbin
The other stateful thing is that when you upload a logo for a dataverse to a particular Glassfish server, the logo is saved only on the local file system of that Glassfish server. In production we point used to point to an NSF mount. Now on AWS, we have a script that rsyncs logos back and forth between the two Glassfish servers.
13:31 pdurbin
Pretty ugly. I don't remember if there's an issue tracking this or not.
13:32 pdurbin
With regard to secrets, the Red Hat interns implemented secrets in openshift.json in the code base. Done. There's an issue and pull request I'm sure I can dig up. Hopefully, we can use the same variables for "Docker Secrets", whatever that is.
13:32 pdurbin
I don't know what "Kubernetes ConfigMaps and Secrets" is.
13:34 pdurbin
I don't know how we'd achieve "rolling-update" ready in Dataverse. I would think we'd want this not just in Kubernetes but also in traditional installations. The Flyway stuff might help here. Updating the Solr schema is still a manual process. Solr does have an API for adding fields. I haven't used it.
13:35 pdurbin
For "multinode cluster" please see comments above about the EJB timer and the dataverse logos being specific to a Glassfish server. There's still some work to do here. I updated http://guides.dataverse.org/en/4.9.4/installation/advanced.html#multiple-glassfish-servers but a release hasn't been cut yet.
13:36 pdurbin
I am in support of configuration being easier.
13:36 pdurbin
"The script is to a large degree a derivative of the old installer from DVN 3.x. It is written in Perl. If someone in the community is eager to rewrite it, perhaps in a different language, please get in touch. :)"
13:36 pdurbin
http://guides.dataverse.org/en/4.9.4/installation/installation-main.html
13:37 pdurbin
We know, we know that our Perl installer is not great. Please see the quote above. If someone can help with a better approach, better technology, whatever, we are open to it.
13:37 MrK joined #dataverse
13:38 pdurbin
"container and classic support" is a must. We must continue to support classic installations. We don't want to scare people away with "kubernetes only" talk.
13:38 pdurbin
I don't think any images I've pushed to Docker Hub have test data.
13:39 pdurbin
I'm not sure where the "test data" comment comes from. docker-aio was explicitly developed for testing. It runs the API test suite with does leave cruft behind.
13:39 poikilotherm
Actually I think now this was a mistake
13:39 pdurbin
Ok, next up is the section on security. poikilotherm should I keep going or do you want to react to anything above?
13:39 poikilotherm
Sure
13:40 poikilotherm
err...
13:40 poikilotherm
Sure = feedback from me
13:40 poikilotherm
First, I don't think it might be necessary now that IQSS releases images
13:41 pdurbin
Good, because IQSS won't any time soon. :)
13:41 poikilotherm
There are options to use the simple maven call, once this is in place
13:42 poikilotherm
And I think this _might_ be something that the Consortium can take care of
13:42 pdurbin
yes, a great project for the consortium
13:43 poikilotherm
I don't think it would be a good idea to point people to images from others like me, slava, ...
13:44 poikilotherm
People around Docker often have little to no trust in user images
13:44 poikilotherm
Because often it is unknown how well maintained those are
13:49 poikilotherm
About being stateless
13:49 poikilotherm
Good that you have this on your radar
13:50 poikilotherm
The logo could be solved by storing it in S3...
13:51 pdurbin
IQSS should help provide the raw materials necessary for people in the community and/or consortium to create, push, and maintain awesome Docker images. IQSS doesn't have the capacity to do it.
13:51 poikilotherm
If someone uses a multinode installation, them needs a solution for this stuff anyway
13:51 poikilotherm
Good to know about secrets support - will look into this later
13:52 pdurbin
see "Separate secrets in openshift.json" at https://github.com/IQSS/dataverse/issues/4763
13:53 poikilotherm
Solr is indeed a tricky one. But one thing after the other
13:54 poikilotherm
About the installer: I think when some stuff is included in the codebase, this will result in a more lightweight version of it
13:55 poikilotherm
To a large degree I see a benefit from those changes for both worlds - Docker and classic
13:55 poikilotherm
And to repeat: I am absolutely with you on the "must" for Docker and "classic"
13:56 poikilotherm
Altough it might sound sometimes like I have only Docker and Kubernetes in mind
13:57 poikilotherm
Please rest assured that I am absolutely aware that my efforts will only have a chance to be merged, when this is respected
13:57 pdurbin
When what is respected? Kubernetes?
13:58 poikilotherm
No, the hybrid approach
13:58 poikilotherm
Both needs to continue to work
13:59 pdurbin
This is why I was suggesting that I could create a dataverse-kubernetes repo for you under IQSS on GitHub. You wouldn't be blocked. You could merge whatever you want. From my perspective, what you need are better raw materials.
13:59 poikilotherm
But I need code changes.
14:00 poikilotherm
And those have to be in small chunks
14:00 poikilotherm
When I work on a fork, this cannot be fullfilled
14:00 pdurbin
What we would merge on the main repo is a line or paragraph in the Installation Guide, saying "There's an experimental dataverse-kubernetes repo you can play with maintained by the community and/or consortium of here: <link>."
14:00 poikilotherm
And I don't like fork
14:00 poikilotherm
+s
14:01 pdurbin
Sure, code changes are part of the raw materials you need. You need a better war file. You need a better or different installer.
14:01 poikilotherm
Do you really think forking would be a good option? Any chances this might make its way back into mainline?
14:01 poikilotherm
I could just live on my own fork in github and use my feature branch
14:02 poikilotherm
Makes only little difference
14:02 pdurbin
I'm not saying you should fork. I'm saying you should continue to make pull requests for code changes you need. For any raw materials you need.
14:02 poikilotherm
I fear of the consequences
14:03 poikilotherm
Sorry, I don't get it. Could you give an example how this repo should look like and what I am supposed to do in there and what not?
14:03 pdurbin
I expect it to look a lot like https://github.com/IQSS/dataverse-docker
14:03 pdurbin
I only recently created that one for Slava.
14:04 poikilotherm
Right.
14:04 pdurbin
Back in June, it looks like.
14:04 pdurbin
I'd be happy to do the same for you.
14:04 pdurbin
Especially since you have a different vision than Slava.
14:04 poikilotherm
Sure. But what I need is going way deeper than what Slava did
14:05 pdurbin
IQSS would be happy to bless both community-supported approaches. Pick your poison. :)
14:05 poikilotherm
Slava wanted to get started quick and sticked to the surface. I am hacking inside, doing plumbing
14:05 pdurbin
Yeah, that's my impression too.
14:06 poikilotherm
And I don't see any option how to do that in a separated repo and not call it a fork?
14:06 pdurbin
You seem more focused on actually running in production. I don't think anyone is in production with Slava's stuff yet. It's great stuff, but I think it's proof of concept, great for demos.
14:06 poikilotherm
The layers on top are mainly additions to the pom.xml now.
14:06 poikilotherm
Everything else happens _in_ the codebase
14:06 pdurbin
You want the same thing as Red Hat, basically. But you want to run on Kubernetes instead of OpenShift.
14:07 poikilotherm
Yeah, that's just a distro of Kubernetes
14:07 pdurbin
yeah
14:07 pdurbin
You *could* volunteer to help Red Hat get Dataverse running well on OpenShift.
14:08 pdurbin
At one point they were even talking about having Dataverse in their catalog next to Jenkins when you click "Java". Just click a button and you have an installation.
14:09 poikilotherm
Sorry, I still don't get it. What I need to realize my vision, and what might help people using OpenShift/Kubernetes, is mostly about hacking on the codebase. not on the tools to get it deployed.
14:09 poikilotherm
I liked your analogy
14:10 poikilotherm
Its "plumbing", not the loader that brings the plumbs to the construction site
14:10 poikilotherm
err - language. plumbs = tubes
14:11 pdurbin
Ok, and you should continue to make pull requests for what you need. Please.
14:11 poikilotherm
Sure :-D
14:11 pdurbin
We don't know what you need. The 35 installations in production are fine with what we are already providing.
14:11 pdurbin
Except for the bugs, I guess. :)
14:12 pdurbin
And they want features. They always want features. :)
14:12 poikilotherm
Well, then just declare Docker / Kubernetes a feature ;-)
14:12 pdurbin
If we created a dataverse-kubernetes mailing list, how many people would sign up?
14:13 poikilotherm
1
14:13 poikilotherm
Me
14:13 pdurbin
heh
14:13 pdurbin
I'll sign up to and read whatever you write.
14:13 pdurbin
too*
14:13 poikilotherm
Looking at dataverse-i18n makes me fear that is no road leading somewhere...
14:14 pdurbin
not sure what you mean
14:14 pdurbin
we have a roadmap :)
14:14 poikilotherm
https://groups.google.com/forum/#!forum/dataverse-internationalization-wg
14:15 poikilotherm
Pretty dead
14:15 poikilotherm
I don't think a dataverse-kubernetes list will have better chances
14:15 poikilotherm
And actually this hole stuff is not only about Kubernetes
14:15 pdurbin
Oh. True. Lots of pull requests though. Jayanthy is working incredibily hard on internationalization.
14:16 poikilotherm
Kubernetes is just a tool
14:16 poikilotherm
The truly good stuff is the underlying Docker
14:16 poikilotherm
There is a good reason pameyer invented docker-aio
14:16 pdurbin
That was Pete's reaction. "This post is about a lot more than Glassfish vs. Payara." I'm paraphrasing.
14:17 poikilotherm
More or less, it is. This discussion is the base for a lot of stuff down the road
14:17 pdurbin
yeah, I was stuck doing Vagrant only, he showed me the way with Docker
14:17 pdurbin
Do you want part 3 feedback. "security needs" and the glassfish vs payara stuff?
14:18 poikilotherm
"Give to me baby, aha aha...."
14:19 pdurbin
Just because app servers haven't had a lot of CVEs lately doesn't mean much. We should always be vigilant. Who knows what will be discovered. This is true of any software, including Dataverse.
14:19 poikilotherm
True
14:20 pdurbin
I guess Java app servers don't fit into the normal "yum" or "apt" model that I'm so used to. I like it when my software is updateable via a package manager. I don't know if wildfly is in an rpm or not, coming from Red Hat. It would be nice.
14:21 pdurbin
But anyway, yes, quarterly releases, I guess. Different than Solr, which releases as soon as they discover a security vulnerability. Solr is also written in Java, of course. I haven't heard of Solr RPMs either.
14:21 pdurbin
I like the characterization as Eclipse Glassfish as the new kid on the block. I should have used that phrase during tech hours the other week.
14:22 poikilotherm
Well... it is no "new kid on the block".
14:22 poikilotherm
https://projects.eclipse.org/projects/ee4j.glassfish/reviews/5.1.0-release-review
14:22 pdurbin
I didn't know Payara had a non profit org as well as a for profit org. Interesting.
14:22 poikilotherm
This is first release from Eclipse. There are no API or implementation changes and has just new licensing and Maven coordinates.
14:22 poikilotherm
This is more or less Glassfish 5.0 renamed.
14:22 poikilotherm
Old wine in new pipes
14:23 pdurbin
Yeah, I think it'll be mostly the same. What's new is the team behind it. Hopefully the code works the same.
14:23 pdurbin
no regressions
14:23 pdurbin
fingers crossed
14:23 pdurbin
I hadn't heard of "variables or pwd aliases".
14:24 pdurbin
But you say it's essential stuff so ok.
14:24 poikilotherm
Do you want an showcase?
14:24 pdurbin
I don't know the difference between "application scoped" and "domain scoped" resource configuration.
14:25 pdurbin
I tried your small-container branch and Dataverse doesn't "just work" yet. I believe you'r still blocked in setup-all.sh territory. I'd be happy to try to help get you unblocked.
14:25 pdurbin
I think that's it. I could go on about Glassfish but whatever.
14:26 poikilotherm
I see. Maybe this is one of the reasons why nobody catched up... ;-)
14:26 poikilotherm
To much unknown territory
14:26 poikilotherm
Sorry about that
14:26 poikilotherm
+o
14:26 pdurbin
nobody replied to you?
14:27 poikilotherm
Only Gustavo in private email
14:28 poikilotherm
Asked for a timeslot this week, no reaction on my offered slots
14:28 pdurbin
Oh! Sounds like progress.
14:28 pdurbin
And Pete gave you a little feedback here in IRC .
14:28 poikilotherm
"Progress"
14:28 poikilotherm
Yes
14:28 poikilotherm
He seemed interested
14:28 poikilotherm
IIRC "lots of good stuff in there"
14:28 pdurbin
Well, Pete doesn't use Kubernetes and doesn't want to last I checked.
14:29 poikilotherm
(paraphrazed)
14:29 poikilotherm
But he wants Docker :-D
14:29 pdurbin
Well, he wants his stuff not to break.
14:29 pdurbin
His installation of Dataverse is different than the others. The only one with rsync.
14:29 poikilotherm
I can't emphasize it enough: Kubernetes is just something on top of Docker. My work is all about Docker first, the Kubernetes part is easy later on
14:30 poikilotherm
Obviously
14:30 pdurbin
Dev for https://github.com/IQSS/dataverse/pull/5359 is happening within three Docker images. Dataverse, DCM, and a client.
14:30 poikilotherm
IIRC he was with me on the integration testing part and docker-aio not so well suited for this, as it would make it even more bloated
14:31 pdurbin
DCM is Data Capture Module.
14:31 poikilotherm
:-)
14:31 pdurbin
The three of us are highly interested in running integration tests *before* pull requests are merged. If you can help with this using Docker or anything, please let us know.
14:32 poikilotherm
Yes I can
14:32 poikilotherm
(no, that wasn't Obama speaking)
14:32 poikilotherm
But it will need pretty Docker stuff
14:32 pdurbin
We have an AWS account we can use to spin up EC2 instances if necessary. I can een give you access. I already gave Don an account. We're using his dataverse-ansible code.
14:32 poikilotherm
And tooling like arquillian
14:33 pdurbin
For this EC2 stuff I'm talking about http://guides.dataverse.org/en/4.9.4/developers/deployment.html
14:33 poikilotherm
Actually I don't want to be on your bill.... :-D All that would be necessary is sth. like Travis CI which is for free
14:33 poikilotherm
Or Jenkins
14:33 poikilotherm
Or whatever :-D
14:34 pdurbin
The doc was delivered as part of this "spin up arbitrary branches: phase 1" issue: https://github.com/IQSS/dataverse/issues/4990
14:34 pdurbin
That's fine. I'm just saying that I don't mind if you're on our bill if you're helping us with this stuff.
14:34 poikilotherm
Ok, good to know :-)
14:35 poikilotherm
I hope you are with me on the track of "use existing commonly used tools" instead of "create your own toolset"?
14:35 pdurbin
It's barely used. The new script we wrote to spin up EC2 instances. Kind of sad.
14:35 pdurbin
Oh, sure. I'm the one who added Travis and Coveralls to the Dataverse mix.
14:35 poikilotherm
It's cool for trying out stuff I think
14:36 poikilotherm
We sth. in mind about using it or a similar script for deployment of testing branches to Kubernetes for RAD approaches to UI/UX dev
14:36 pdurbin
I'm the one who moved us from Sourceforge SVN to GitHub too. :)
14:37 poikilotherm
(I may not use AWS here, conflicts with company policy)
14:37 poikilotherm
:-)
14:37 poikilotherm
Cool!
14:37 pdurbin
A guy Raman moved us from Redmine to Github issues.
14:38 pdurbin
But who would host Kubernetes for us... oh, you're saying the script could be adjusted to used Kubernetes on AWS instead of raw EC2 instances?
14:38 poikilotherm
At least one could try that
14:38 pdurbin
to use*
14:38 poikilotherm
Haven't looked into it yet
14:38 poikilotherm
But why not
14:38 pdurbin
But you aren't allowed to try it on AWS? Company policy?
14:39 poikilotherm
I cannot make FZJ pay the bill for an account of our own
14:39 pdurbin
But I could add you to our account like how I added Don from UNC.
14:39 poikilotherm
Maybe...
14:40 poikilotherm
But let me do plumbing first :-D
14:40 MrK
I just read the first lines but how poikilotherm would you vision the docker containers would be, since right now replacing war is a lot of a trouble for me there :D
14:40 pdurbin
Then you could play Kubernetes in there.
14:40 poikilotherm
MrK I am using the fabric8io docker maven plugin
14:40 pdurbin
maybe MrK would be the second subscriber to the dataverse-kubernetes mailing list, if we make one :)
14:40 poikilotherm
It supports a watch target for auto rebuilding
14:41 poikilotherm
So this would be sufficient for dev purposes I think
14:41 poikilotherm
For production new images would be needed to get pushed to a registry
14:41 poikilotherm
Either into your custom registry, user account on Docker hub or somewhere else (currently using quay.io )
14:42 MrK
right now we had a pipline on jenkins where we just cloned a branch created war and replaced it but even dockerEU is not that easy.
14:42 poikilotherm
The production containers need to be replaced in a rolling-update fashin
14:42 poikilotherm
Yeah...
14:42 poikilotherm
I know...
14:42 MrK
yeah thats not a problem you would have a pipeline clone-maven-push image- and docker up
14:43 poikilotherm
for local dev that would shorten to clone+mvn
14:43 poikilotherm
As it is right now :-D
14:43 MrK
yep
14:43 poikilotherm
That doesn't need to change alot anymore ;-)
14:43 poikilotherm
The changes to the pom.xml are rather trivial
14:44 MrK
poikilotherm: poke me if you would have a working version where you can just do it so simply, my co-programmer will also be happy :D
14:44 poikilotherm
The pipeworks and plumbing inside the code base is the hard nut to crack
14:44 poikilotherm
Sure :-D
14:44 poikilotherm
You could watch issue #5292 ;-
14:44 poikilotherm
;-)
14:45 poikilotherm
(and related)
14:45 MrK
I also have a voice call with Slava on friday since I'm wondering how they do it
14:45 poikilotherm
Most certainly they will do a rebuild, too
14:46 MrK
the problem is that when i replace stuff some frontend stuff are missing like an icon on navbar or the upload function is completely missing and I don't even know why since the same war locally is working fine. but in docker, nope
14:48 poikilotherm
IMHO dataverse-docker is pretty bloated and tries to depict the normal setup into containers. IMHO this makes things worse than better.
14:48 poikilotherm
But this is just my opinion and if Slava is happy with their solution that is perfectly fine to me
14:48 poikilotherm
-to +for
14:49 MrK
Yeah I agree the project which in mind should've been easier and help is as much complicated as normal version
14:49 poikilotherm
I am an optimist and perfectionista
14:49 poikilotherm
Just like pdurbin...
14:50 poikilotherm
So I try to improve and contribute to upstream.
14:51 poikilotherm
Hey pdurbin: do you think it might be a good idea to add some labels to my issues?
14:51 poikilotherm
Like "Community dev blocked" and maybe "Docker"?
14:52 poikilotherm
Maybe I didn't a good job talking to the devs about what this is and how important it is
14:53 poikilotherm
I really hate forking, but I would really appreciate some feedback from the core devs
15:08 pdurbin_m joined #dataverse
15:09 pdurbin_m
poikilotherm: the easiest way to get attention is to make a pull request. We discuss issues in the code review column every morning at standup, in about an hour.
15:10 pdurbin_m
I don't think more labels will help.
15:10 poikilotherm
Ok
15:10 poikilotherm
THe problem with that approach: implementing a change might be a reasonable amount of work
15:11 poikilotherm
And not knowing if it is worth it or gets declined is... "suboptimal"
15:11 poikilotherm
I would REALLY appreciate discussing things first before starting implementation wokr
15:13 pdurbin_m
maybe just make a change to the dev guide then
15:14 pdurbin_m
documentation driven development
15:14 pdurbin_m
Does that make sense?
15:14 poikilotherm
Well... I can try.
15:14 poikilotherm
Altough I prefere TDD
15:14 poikilotherm
But ok
15:14 pdurbin_m
thanks
15:16 poikilotherm
I will create a PR for the necessary changes on the EJB timers
15:16 poikilotherm
This is rather triviail
15:16 poikilotherm
And if that starts moving things...
15:17 MrK
poikilotherm: I saw there is also a problem with some sequence sometimes it wont allow app to run.
15:34 pameyer joined #dataverse
15:47 poikilotherm
pdurbin most certainly I will be late for the call
15:48 poikilotherm
Arrival of government is delayed by traffic jams
15:49 poikilotherm
Hear you guys later :-)
16:11 Paul66 joined #dataverse
16:12 Paul66
Hi, what is the largest file size allowed and is there a limit on total file space?
16:15 pameyer
Paul66: for normal uploads, I think file size is configurable. not sure about total space
16:15 Paul66
ok, thanks!
16:15 Paul66 left #dataverse
17:04 pdurbin
There is no limit on total file space. Please see https://github.com/IQSS/dataverse/issues/4339
17:07 pdurbin
The community call is starting. Trying gotomeeting this time: https://dataverse.org/community-calls
17:16 poiki-at-home joined #dataverse
17:17 poiki-at-home
Hey guys, is the meeting still going?
17:19 poiki-at-home
Just joined, listening. Need to check micro later when appropriate
18:06 craig-willis joined #dataverse
18:31 pdurbin
craig-willis: thanks for calling in and participating!
18:39 craig-willis
pdurbin: you've suggested for years (?) that I join and now I have multiple good reasons. Very helpful to hear where y'all are headed.
18:40 pdurbin
It all seems like madness when we start talking about file hierarchy. Everyone is surprised when they discover how Dataverse stores files.
18:40 jri joined #dataverse
18:40 pdurbin
pameyer had to build a parallel way to store files where the filenames are not scrambled on disk.
18:41 pdurbin
I was surprised as anyone when I first looked at my file system with Dataverse running.
18:41 pdurbin
So it's a challenging situation.
18:42 pdurbin
As I was saying we can probably at least start by bringing back the file hiearchy feature we had in DVN 3: Zip file in, expand and put the path for each file in the database, zip file out. But less buggy. It would be a start.
18:43 craig-willis
Several systems I work with are similarr, I expect. I don't think DataONE supports hierarchy, so we've been publishing our own map.
18:43 pdurbin
interesting
18:43 pdurbin
What was the original thinking? You'll never need to upload more than a handful of files?
18:44 pdurbin
You'll only interact with the files via the GUI ?
18:44 pdurbin
Don't you want to save storage space when the same file is in multiple versions?
18:44 pdurbin
I don't know.
18:44 pdurbin
It's complicated. :)
18:46 craig-willis
I don't know the original thinking either.
18:49 pdurbin
The TRSA stuff might help. Jon said files in there will be in a hierarchy.
18:50 pdurbin
pameyer's rsync stuff helps. It'll be interesting to see if other Dataverse installations adopt it.
18:57 bricas
it's been a while since i've been here. oops.
18:58 bricas
pdurbin: you asked me a question, oh, a month ago .... :)
18:58 pdurbin
Did I? Don't worry about it. :)
18:58 bricas
"how about you? Glassfish or Payara?"
18:59 bricas
we use glassfish
18:59 pdurbin
Yeah, I think everyone does. Glassfish is what's in our installation guides.
19:00 pdurbin
bricas: but I'd be interested to hear what you think about https://groups.google.com/d/msg/dataverse-dev/L6rNXbbZG0o/UFVfI0_gAQAJ
19:05 bricas
we also have the goal of moving our dataverse install to our k8s infrastructure
19:05 pdurbin
interesting
19:06 pdurbin
I keep thinking that very few people want this.
19:06 bricas
making this as easy as possible would be great. we don't need anything super "high availability" since our demand is pretty low (at least right now)
19:07 pdurbin
bricas: sure. Most installations run Dataverse all on one box, I believe. Can you please take a look at https://github.com/IQSS/dataverse/issues/5292 ? That's the "epic" Oliver opened and just talked about on the community call. He'd love feedback.
19:07 bricas
using glassfish/payara has never really mattered to us since we're not java folks and just need it to work :)
19:08 pdurbin
heh, exactly
19:08 bricas
we're containerizing like crazy 'round here.
19:08 pdurbin
interesting, I haven't seen it at Harvard yet
19:08 pdurbin
but I'm in a different role now
19:08 pdurbin
I'm one of those java folks now, I guess. No longer a cluster person.
19:09 pdurbin
mostly the push seems to be to move to AWS
19:09 pdurbin
which we did for Harvard Dataverse
19:10 pdurbin
but we also have a nice new datacenter in western mass. Cheaper to run compute out there that AWS. I don't know. Again, I'm less involved in this stuff now.
19:13 poiki-mobile joined #dataverse
19:13 pameyer
it's interesting that the various containerization bits expose different areas of tech debt than the recommended installation path
19:13 bricas
we use a number of amazon services but haven't moved to their infrastrcuture
19:14 pdurbin
plenty of tech debt
19:14 pdurbin
bricas: you're running k8s on your own hardware?
19:15 bricas
we have a decent setup here and so far we're happy to stick with it but there may come a day when that cost is too much and it's easier just to throw money at aws
19:15 bricas
yep!
19:15 bricas
containerzing definitely shapes your mindset for deployment
19:17 poiki-mobile
Good evening boys n girls ;-)
19:17 poiki-mobile
thx for a very pleasing community call. although it was a bit confusing hearing the bus stations... :-D
19:18 pameyer
especially the bus station in a different city, with different announcements ;)
19:18 poiki-mobile
Kiddo needs his fairy tale now, shortly AFK
19:19 pdurbin
poiki-mobile: thanks for calling in. You have another customer: bricas :)
19:19 pdurbin
maybe :)
19:21 pdurbin
poiki-mobile: and I'm talking about your epic at https://javabot.evanchooly.com/logs/%23glassfish/2018-12-04 if you want to join in when you return. #glassfish on freenode
19:36 poiki-mobile2 joined #dataverse
19:38 poiki-mobile2
just joined #glassfish
19:39 craig-willis joined #dataverse
19:48 pdurbin
heh, nice
19:51 craig-willis joined #dataverse
19:56 poikilotherm joined #dataverse
20:01 jri joined #dataverse
20:22 jri joined #dataverse
20:42 jri joined #dataverse
21:03 jri joined #dataverse
21:21 pdurbin
poikilotherm: lots of chatter over there!
21:23 jri joined #dataverse
21:39 poikilotherm
pdurbin: yeah. Bad?
21:44 jri joined #dataverse
22:03 poikilotherm
pdurbin: any news from tech hours?
22:03 poikilotherm
Did anyone bring a pile of stones to throw in my direction?
22:07 pdurbin_m joined #dataverse
22:07 pdurbin_m
poikilotherm: good chatter
22:08 pdurbin_m
We didn't talk about your stuff at tech hours. Mostly it was Make Data Count and security.
22:20 poikilotherm
:-)
22:20 poikilotherm
Sounds good
22:20 poikilotherm
I really enjoyed the feedback from the community about Docker/k8s
22:20 pdurbin_m
me too
22:21 poikilotherm
Seems like this might strike a chord
22:25 jri joined #dataverse
22:26 poikilotherm
Alright, I am off for the day... 23:26 over here. Bedding time
22:26 poikilotherm
Read you tomorrow. And please pdurbin: - don't let me get between you and your wife at 6:30 am in the morning ;-) It's "just" Dataverse
