Time
S
Nick
Message
04:24 xarthisius joined #dataverse
04:24 xarthisius joined #dataverse
07:29 juancorr joined #dataverse
08:17 jri joined #dataverse
08:34 MrK joined #dataverse
10:40 poikilotherm joined #dataverse
11:47 pdurbin
MrK poikilotherm: I just left a comment on the flyway issue: https://github.com/IQSS/dataverse/issues/5344#issuecomment-442419253
11:51 pdurbin
and good morning :)
12:19 poikilotherm
Gooood morningggggg
12:29 MrK
Hiiii
12:35 poikilotherm
I'm in a good mood today :-)
13:15 pdurbin
poikilotherm: oh good. And I had a thought for you. An analogy. I wanted to ask you about software written in C with and without autotools and making packages (rpm and deb) out of it.
13:16 poikilotherm
Errr... not that experienced with combining those three together, but used all of 'em in some way...
13:17 poikilotherm
This stuff is pretty much standard for Linux, isn't it?
13:17 poikilotherm
I remember those days when you had to ./configure ./make ./make install
13:18 poikilotherm
But please: go ahead and ask :-)
13:18 poikilotherm
Will try my best
13:35 cyberdante joined #dataverse
13:35 pdurbin
poikilotherm: imagine you're me some years back. Users of your cluster want a certain piece of software. It's already in package repos. You run `yum install foo` across the cluster and you're done. Then they want "bar". No package. You're writing an RPM specfile.
13:36 pdurbin
Are you with me so far?
13:38 poikilotherm
Ok
13:44 pdurbin
The C code is the raw material. The author of the C code can make it easier to make an RPM. Like you said, they can arrange for `configure && make && make install` to "just work" by using autotools or other frameworks.
13:45 pdurbin
But even if the C author doesn't make the packager's job easy, the packager isn't blocked from writing an RPM spec file. It's just more work. Sometimes much more work, unfortunately.
13:46 poikilotherm
Indeed
13:46 pdurbin
So I'm wondering about blockers.
13:46 poikilotherm
Blockers on what layer / technology?
13:47 pdurbin
And I'm wondering about the raw materials.
13:47 pdurbin
Instead of C we're writing a load of Java.
13:48 pdurbin
Is this analogy helpful at all? Do you see what I'm getting at?
13:49 poikilotherm
Are going down the road "let us install Dataverse via RPM / DEB"?
13:49 pdurbin
No, no.
13:50 pdurbin
I feel like I'm on the other side right now. Instead of packaging software I'm working on the raw materials.
13:50 pdurbin
But I'm very sympathetic to anyone who is trying to package the software.
13:50 poikilotherm
Right
13:51 pdurbin
"Package" probably isn't the right word for a webapp. "Deploy" is probably a better word.
13:51 poikilotherm
You want to know if devs can make life easier for people that deploy stuff
13:52 pdurbin
Pretty much. And if people deploying are blocked. 35 production installations are not blocked, apparently. But I'm sure we can make deployment easier.
13:52 poikilotherm
Actually I think you need to take into account that since those days, the world has changed. Docker appeared and got impact by being a good workaround for the limitations of RPM. DEB and the like
13:53 MrK
Well I'm "We in the office" are also going towards that easier deploy
13:53 pdurbin
I will say that calls for us to produce a Dataverse RPM have disappeared. No one wants this anymore, I don't think.
13:53 MrK
Personally for starters the jar would be better not war since the server is embedded so you dont need to install any on your server, then getting rid of starting script
13:54 MrK
When I had previous job we just used maven to build the jar and then we just run it
13:55 pdurbin
MrK: sure, that's how Solr used to be. `java -jar start.jar` or something.
13:55 poikilotherm
MrK is heading for the UberJAR way ;-)
13:55 MrK
before you just needed to have the dependencies like solr which could be dealt with single script or just docker.
13:55 pdurbin
poikilotherm: you're right that Docker is hot but I don't think any of the 35 production installations of Dataverse are using Docker.
13:56 poikilotherm
Indeed. But there are more to come. Think about DataverseEU, FZJ, Göttingen unknown, MrK
13:56 pdurbin
Sure. Like Linux, we should strive for world domination. :)
13:57 MrK
Well docker is optional like you said, we just can make it easier for docker. Right now I'm trying out Jenkins for dataverse and docker-compose up is running for 16 minutes and its still not up :D so something is not right.
13:58 pdurbin
So I'm wondering about raw materials. We just merged a pull request to provide SQL schema files for all version of Dataverse from 4.0 to 4.9.4. And we are committed to provide these for every release. Hopefully this additional raw material will help. It's just an example.
13:58 pdurbin
What other raw materials would be helpful to people deploying Dataverse. I guess that's the question.
13:58 poikilotherm
MrK did you try my feature branch? https://github.com/poikilotherm/dataverse/tree/5292-small-container
13:59 MrK
I didn't could you do tl'dr :P?
13:59 poikilotherm
Since today I have successfull deployment, altough you cannot access Dataverse right now... Need to strip out old stuff
13:59 pdurbin
With the right raw materials, people deploy Dataverse can get creative. They won't be blocked by whatever IQSS is doing. I hope that makes sense.
13:59 pdurbin
people deploying* Dataverse
14:00 poikilotherm
Just do a git clone -b 5292-small-container and run mvn -Pcontainer clean package docker:build docker:run
14:00 poikilotherm
This is far from perfect, but I am getting nearer
14:01 poikilotherm
pdurbin: that totally makes sense!
14:01 pdurbin
good
14:01 pdurbin
I'm always trying to unblock people.
14:01 poikilotherm
When I started with Dataverse, it felt clumsy and heavyweight with this installer stuff and all
14:01 pdurbin
I don't want people to feel blocked in deploying Dataverse.
14:02 poikilotherm
It would be great to have a better experience for people
14:02 poikilotherm
Many is already getting addressed
14:02 pdurbin
Yeah. People struggle deploying Dataverse all the time. I've posted over a thousand times to our mailing list.
14:04 poikilotherm
Most certainly by making the initial deployment easier a lot of issues for upgrade-deployment gets addressed, too
14:04 poikilotherm
And right now I have the feeling that you guys should address people interested in an adaption first
14:04 pdurbin
It's pretty straightforward to tell a C developer how to make packaging easier. Webapps are a lot more complex. If there are small changes we can make to Dataverse over time (small pull requests) to make deployment easier, I'd like to know what they are.
14:05 MrK
poikilotherm: Have you looked how dataverse-docker project looks like?
14:06 poikilotherm
I have
14:06 poikilotherm
And I think its bloated
14:06 pdurbin
poikilotherm: adoption is our number one strategic goal according to https://dataverse.org/goals-roadmap-and-releases
14:06 poikilotherm
IMHO this is not a clean microservice approach
14:07 pdurbin
poikilotherm: do you think you'd be able to convince Slava to join us in here to talk about dataverse-docker?
14:07 poikilotherm
Dunno - just talked to him once.
14:07 poikilotherm
And I don't want to get them blocked
14:08 poikilotherm
They seem to be fine with their approach
14:08 poikilotherm
And need it right now
14:08 poikilotherm
This can be changed once proper images are available from "Upstream"
14:09 MrK
But we don't have microservices :P
14:09 poikilotherm
Depends on your definition of it ;-)
14:10 poikilotherm
At least this is SOA
14:10 MrK
Well citation from Spring dev "It does one thing and it does it well" and dataverse is monolithic :P
14:10 pdurbin
poikilotherm: you shouldn't expect Docker images from IQSS any time soon. And it sounds like you don't want Slava's images. Are you blocked from making your own images?
14:11 poikilotherm
I am working on it... :-D
14:11 poikilotherm
See #5292 ;-)
14:11 * pdurbin
looks at https://github.com/IQSS/dataverse/issues/5292
14:12 pdurbin
poikilotherm: right, your epic. Seems blocked. What's the next step to make?
14:13 poikilotherm
I am actively working on it
14:13 poikilotherm
Heading for Payara 5... ;-)
14:13 poikilotherm
People should start discussing technical stuff with me :-D
14:14 pdurbin
MrK: yes, Dataverse is a monolith. I mentioned this on my way out the door yesterday: https://github.com/IQSS/dataverse/issues/4821#issuecomment-442226108 . My attitude is to embrace what we are. But I'm open to change. :)
14:14 MrK
But the code would be needed properly splited, right now it's kinda chaos without any modularity.
14:15 poikilotherm
Wow, be carefull what you wish for... There be dragons. Changing the architecture here is a monster to fight
14:15 MrK
pdurbin: Oh there is nothing wrong in monolith.
14:15 MrK
Nah I'm not changing anything :D
14:15 pdurbin
yeah, we have dreams of modularity but aren't there yet. The most modular thing we have is our "external tool" framework
14:16 pdurbin
poikilotherm: is there anyone discussing technical stuff with you yet?
14:16 MrK
The problem is that in order to make the project pretty it would require a lot of work, right now the methods have 300 lines :P.
14:17 pdurbin
or more
14:17 poikilotherm
pdurbin: except you not really
14:17 pdurbin
bummer
14:18 poikilotherm
especially no core devs like matthew, scolapasta or landreev
14:18 MrK
The ideal world would be classes splitted into packages per functionally with package scope and function facade would be public which would communicate with other modules and classes/functions would be small.
14:19 pdurbin
poikilotherm: why do you need the Docker images "right now". Why not start with a boring old school deployment like the other 35 and work toward Docker stuff in the future? Are you worried your institution will consider the project "done" once you're in production?
14:19 pdurbin
MrK: sounds good. Which corner of the code would you start with?
14:19 poikilotherm
pdurbin: I don't need those - DataverseEU seems to use them right now
14:20 poikilotherm
I need them mid-term because we go for Kubernetes
14:20 poikilotherm
I would be happy with a classic approach, but my colleague wants to go for Kubernetes
14:20 poikilotherm
And I want to go for integration testing in containers ;-)
14:20 poikilotherm
That leads in the same direction
14:20 pdurbin
ah, so it's your colleague. ok
14:20 MrK
pdurbin: hard to say, I know that if me and tjanek will be getting buisness request we will be applying those principles.
14:21 pdurbin
MrK: makes sense. While you're hacking on a feature, you could also improve the architecture.
14:21 MrK
Yeah that would be ideal, small steps.
14:22 poikilotherm
Grml... Just hit http://showcase.omnifaces.org/#installation regression... Can't move that stuff to domaindir/lib
14:23 pdurbin
poikilotherm: do you want me to create you a "docker-kubernetes" repo under IQSS? You could start fresh with your own ideas of how to do this right.
14:24 poikilotherm
Oh currently I am using conf/docker/app and stuff in pom.xml
14:24 poikilotherm
I am not sure if I can use the dataverse project as a child and put my container stuff in a parent pim
14:24 poikilotherm
-i+o
14:25 poikilotherm
Not an expert in that... ;-)
14:25 pdurbin
Right, but a number of deployment solutions have their own life and own repo. dataverse-ansible, dataverse-puppet, dataverse-docker, dataverse-aws. They all use raw materials from the main repo.
14:25 poikilotherm
AFAIK all of those use a dvinstall.zip
14:26 poikilotherm
My target is currently just a single-line dev/demo experience
14:26 poikilotherm
Kubernetes is a next ste
14:26 poikilotherm
+p
14:26 pdurbin
Most do, probably. Recently Don made it so that dataverse-ansible can built a dvinstall.zip from any arbitrary branch in the main repo. Great feature.
14:28 pdurbin
Potential for integration testing before merging with develop.
14:34 poikilotherm
pdurbin: I really hate to admit it, but I think as long as Dataverse is monolithic, I can't help it: I need to package deps inside the WAR.
14:34 poikilotherm
Just read https://stackoverflow.com/questions/30535024/facescomponent-on-shared-library/
14:35 poikilotherm
Fiddeling around which packages might be left out and which don't seems no good option and would make the classic deployment going mad
14:36 poikilotherm
That means larger Docker images :-(
14:36 poikilotherm
Err... I mean... I cannot split the layer.
14:36 poikilotherm
Because it is one big fat WAR
15:00 pdurbin
poikilotherm: what if we stopped using JSF? Would that help?
15:00 poikilotherm
???
15:00 poikilotherm
You are kidding me, aren't you?
15:00 pdurbin
It's just a question. :)
15:01 pdurbin
Please don't read too much into it. :)
15:01 poikilotherm
There are good chances that there is more stuff that uses annotations, etc and need to live in WEB-INF/lib
15:02 poikilotherm
Maybe stuff like AWS SDK can be moved, but as I said before: this is a lot of manual work to get sorted out
15:02 poikilotherm
There are bigger problems to look at first :-D
15:08 donsizemore joined #dataverse
15:21 isullivan joined #dataverse
15:32 pdurbin
yeah
15:33 poikilotherm
pdurbin looking at the docs: what properties are really essential?
15:34 poikilotherm
I guess dataverse.timerServer, dataverse.files.directory and dataverse.fqdn?
15:34 dataverse-user joined #dataverse
15:39 pdurbin
poikilotherm: I guess a good starting point might be http://guides.dataverse.org/en/4.9.4/_downloads/default.config
15:40 pdurbin
I'm not sure if you've used "default.config" or not. It's documented at http://guides.dataverse.org/en/4.9.4/installation/installation-main.html
15:40 pameyer joined #dataverse
15:41 poikilotherm
Rserver seems not to be necessary as long as you don't use/have it, right?
15:41 pameyer
I've never used rserver
15:41 pameyer
and haven't noticed the lack yet
15:42 poikilotherm
OK Solr will be trickier... That's a database setting, not a property
15:43 poikilotherm
Maybe I add a code that checks for a sys prop and sets the database entry accordingly
15:45 poikilotherm
admin email is just the same, but even less important...
15:47 pdurbin
welcome back pameyer
15:47 pameyer
pdurbin: thanks
15:49 pdurbin
pameyer: when you get a chance, please read through " Future use of Glassfish and/or Payara " by poikilotherm at https://groups.google.com/d/msg/dataverse-dev/L6rNXbbZG0o/UFVfI0_gAQAJ
15:50 pdurbin
donsizemore jri juancorr bricas: you too, please
15:56 pameyer
that's a lot more than "future use of gf/payara"
15:56 pdurbin
yeah
16:08 pameyer
reminds me of the rock climbing principle that you want to have 3 points in contact at all times
16:09 pdurbin
sounds like good advice
16:10 pameyer
deploy artifacts, configs, state management, app servers, rolling updates, integration testing - there's lots of good stuff there
16:10 pdurbin
yeah
16:13 pdurbin
nice write up
16:14 pameyer
yup - although it does seem lacking in paranoia
16:14 pdurbin
heh
16:15 pdurbin
we have people for that
16:48 isullivan joined #dataverse
17:27 isullivan joined #dataverse
17:50 nanoz joined #dataverse
18:16 donsizemore joined #dataverse
18:31 donsizemore
@pdurbin thu-mai asks how about mapping?
18:35 pdurbin
donsizemore: mapping?
21:53 pameyer
pdurbin: since poikilotherm addressed https://groups.google.com/d/msg/dataverse-dev/L6rNXbbZG0o/UFVfI0_gAQAJ to the devs, I'll let other folks answer on that thread
22:01 pdurbin
pameyer: it's funny, there's a lot of devops in there too. But you're actually a dev, you know. :)
22:06 pameyer
pdurbin: kinda, but poikilotherm also knows where to find me ;) and probably has a general idea of my take on things. so it'd be good to have others in the conversation
22:26 jri joined #dataverse
22:33 pdurbin
pameyer: yeah, I figured we can talk it out with him here. I do have a fair amount of feedback to give.
22:45 pameyer
https://github.com/poikilotherm/dataverse/commit/d4ad175ecdfbdd934eb6233d4c3e09f482871993 :)
22:57 pdurbin
pameyer: heh, "disable phone home"? You've found a kindred spirit! :)
22:58 pdurbin
I just found this longish comment: https://github.com/nds-org/ndslabs-dataverse/issues/8#issuecomment-442637425j ... I'd love to get Craig and Oliver talking. :)
23:00 pameyer
tried to kick the tires on 5292-small-container - `mvn docker:build` fails :(
23:01 pdurbin
oh. bummer
23:02 pameyer
I'm probably jumping the gun looking at a work in progress
23:03 pdurbin
meh, he told MrK to try it out
23:04 pdurbin
see, if this were in it's own repo there would be a natural place to open an issue
23:04 pdurbin
its*
23:04 pameyer
you can't open issues in a fork?
23:04 pameyer
I didn't try
23:05 pdurbin
nope
23:05 pdurbin
I imagine you'll be able to let him know tomorrow. Or he'll catch up on the logs.
23:07 pdurbin
I guess you could leave the error as a comment on his latest commit. I don't know.
23:07 pdurbin
I'm kind of curious what it is.
23:07 pdurbin
I'd be less curious if my sausages on the grill were done cooking already.
23:08 pameyer
[ERROR] No plugin found for prefix 'docker' in the current project and in the plugin groups
23:08 pdurbin
huh
23:08 pdurbin
I got nothing. Sorry.
23:08 pameyer
well, I'm sure I've said I hate maven before ;)
23:08 pdurbin
I don't think Gradle is any better.
23:09 pdurbin
Ant was worse.
23:09 pameyer
and mentioning sausage grilling reminds me that it's probably getting to be time for me to head out
23:09 pdurbin
have a good night
23:09 pameyer
never tried gradle. ant never gave me problems, but I never used it on a big project
23:09 pameyer
have fun grilling
23:10 pdurbin
always
