Time
S
Nick
Message
02:28 jri joined #dataverse
06:20 jri joined #dataverse
07:56 poikilotherm joined #dataverse
08:14 otheus joined #dataverse
08:14 otheus
I want to contribute to the Admin/installation documentation, and possibly to making some parts deployable via RPMs
09:08 poikilotherm
Hi @otheus, Harvard is not awake yet... 05:08 over there ;-)
09:14 jri joined #dataverse
09:28 otheus
i'm patient.
10:17 pdurbin
IRC rewards patience.
10:21 poikilotherm
The dinosaur grandpa of all chat systems has some patience built in, hasn't it?
10:21 pdurbin
otheus: thanks for the pull request yesterday! Yes! Please help with our docs! More RPMs would be great too. We already build our own RPMs for rApache, downloadable from the TwoRavens section of the Installation Guide.
10:22 pdurbin
poikilotherm: I've been running this IRC channel since before Slack existed. :)
10:27 otheus
pdurbin: Are the Administration/installation docs in a git repo? I didn't see them in the sources repo.
10:28 otheus
Also, there's this: https://github.com/mentalblock/solr-rpm (but its 3 yrs old)
10:32 pdurbin
otheus: yes, we use Sphinx and .rst files which are in the main git repo. This is explained at http://guides.dataverse.org/en/4.9.4/developers/documentation.html but that page itself could use a refresh. :/
10:37 poikilotherm
pdurbin: and maybe a maven plugin to trigger the build instead of a make call? (plus maybe some CI...)
10:39 pdurbin
poikilotherm: for the make thing, it would be great for you to leave a comment at https://github.com/IQSS/dataverse/issues/4528
10:50 otheus
@pdurbin I'm looking at https://github.com/IQSS/dataverse/tree/develop/doc and I see no files containing documentation (except .md files).
10:50 otheus
oh wait, the documenation is under "sphinx-guides". Yeah, that wasn't expected
10:52 poikilotherm
pdurbin: I saw http://irclog.iq.harvard.edu/dataverse/2018-10-11#i_75440 earlier... Actually you have different options in Spring. You can embed the container runtime and build a runnable jar, that will start the container and the application or you can leave that aside and build deployable wars
10:53 poikilotherm
When you opt for runnable jars, spring boot is of great help.
10:53 poikilotherm
This stuff is really easy to dockerize, as it is just a java jar
10:54 poikilotherm
(thus the container basically needs java and maybe some other deps)
10:54 poikilotherm
But as you mentioned, Spring is not using EE stuff. I am not aware of anything equivalent like Spring Boot for Java EE applications
10:56 poikilotherm
The "fat jar" from Spring makes it also a breathe to run it on a laptop/dev env... Especially when you enable things like LiveReload, the development of frontend stuff is quite neat
10:59 poikilotherm
But IMHO all this nice stuff seems not be relevant to Dataverse... Porting all of this to Spring could be an effort to big for the project and let its evolution die...
11:08 pdurbin
Yeah. Making breakfast. Kids waking up soon.
11:09 poikilotherm
:-)
11:14 pdurbin
I have to admit that I don't have any real experience with Spring but I've heard a lot about it. Have you heard of MicroProfile?
11:15 poikilotherm
Read about it, but no exp. IIRC this was targeted to microservices. Unsure if Dataverse could be expressed as a set of microservices.
11:16 pdurbin
Well, maybe you'll turn the PID code into a microservice. :)
11:16 poikilotherm
:-D
11:17 poikilotherm
Actually I would think of a Spring Boot app for this... ;-)
11:17 poikilotherm
This stuff needs no EE, so that might be more lightweight
11:21 pdurbin
But people would be able to write their own PID service in whatever language, right?
11:23 donsizemore joined #dataverse
11:24 pdurbin
otheus: are you unblocked? Are you thinking of a big rewrite or just small tweaks? Either way, if you would create an issue before making a pull request, that would be great.
11:24 poikilotherm
Uh... I don't think so. Ok, I see your point about microservices as a possible approach for that
11:25 donsizemore
@pdurbin i still need to fix something in dataverse-ansible, but i'm going to go ahead and submit a pull request for #5162
11:25 pdurbin
poikilotherm: I'm confused. Someone can't come along and write a PID service in Python or whatever?
11:25 pdurbin
donsizemore: please! The minute someone from the community makes a pull request, I can help. :)
11:25 poikilotherm
Sure, as long as it has a REST api or whatever API that could be talked to by the "proxy service" written in Java
11:26 poikilotherm
But if you want to have the service _inside_ the proxy, you would have to use java
11:26 donsizemore
@pdurbin it's an asadmin command in dataverse-ansible i'm trying to club into submission. i think the ec2-create script is good once i proof-read to make sure i don't leave behind any debugging mess
11:26 poikilotherm
(at least DANS people started with java now, maybe stick with that...)
11:30 otheus
speaking of the asadmin command, when the install script runs it, it prompts for the admin username, even though $noninteractive is true. Is there any way to provide the java command (asadmin) with an option or variable so that this prompt isn't necessary?
11:36 pdurbin
poikilotherm: I'm not sure I understand the architecture. How do you feel about creating a new GitHub issue that explains the idea? Yesterday at standup I mentioned our chat about PID services but having the plan in one document (maybe a Google doc) would be great.
11:37 pdurbin
donsizemore: ok, if you need more eyes on in, please just let us know
11:39 pdurbin
otheus: I would love to remove that prompt. Have you tried using a "default.config" file with the installer like this? https://github.com/IQSS/dataverse/blob/v4.9.4/scripts/vagrant/install-dataverse.sh#L29
11:41 donsizemore
@otheus the install script does that at https://github.com/IQSS/dataverse/blob/develop/scripts/installer/install#L957
11:43 donsizemore
@otheus (and i'm checking now to see what you can do in asenv.conf)
11:44 donsizemore
@otheus not much on the latter... n/m
11:49 otheus
@pdurbin yes, I'm using the default.config file, but there is no asadmin user default.
11:50 otheus
@donsizemore but the asadmin login command prompts for a user name.
11:50 pdurbin
otheus: you're saying you'd like a way to configure the username and password for asadmin using the default.config file? I may be confused.
11:50 otheus
I'm saying if you want to do an automated install, without prompts, it should be automated without prompts.
11:50 pdurbin
heh, sure
11:50 otheus
OK, but more importantly I have a nother big issue.
11:51 pdurbin
In Vagrant it doesn't prompt. That's what I was trying to link you to.
11:51 otheus
It's a catch-22. The first install process failed because glassfish couldn't restart (timed out).
11:51 otheus
The second run complained that psql database already existed, so the install process aborted
11:51 otheus
The seeming work-around was to use --skip_db_setup
11:52 pdurbin
Hmm, you're using the installer in the 4.9.4 release?
11:52 donsizemore
@otheus i vaguely remember running into this, but the install script runs repeatedly without prompting. at a glance i don't see where in the script. leonid would be the guy to ask
11:52 otheus
But THAT skips "step 3" in the install script, which incidentally, sets up the "$pg_exec_path" required by "import_reference_data" (step 7)
11:53 pdurbin
Why did Glassfish fail to start? Not enough RAM or something?
11:53 otheus
No reason. It was stopped and wasnt started again.
11:53 otheus
I restarted it manually, no issue
11:54 pdurbin
Huh.
11:54 otheus
I downloaded the dvinstall zip file per the website instructions
11:54 pdurbin
Ok, great.
11:55 pdurbin
otheus: we just rewrote the installer so maybe a bug or two slipped in. If you would open a GitHub issue about the trouble you're having, it would be much appreciated.
11:55 otheus
Sure. :)
11:55 otheus
By the way, I know how much pain-in-the-ass-work it is to create an installer. Especially in unix.
11:56 otheus
Going to the dentist is by far a more pleasant experience
11:56 pdurbin
Yeah, that old Perl script is from 10 or 12 years ago. We just keep hacking on it.
11:56 * pdurbin
added "use strict"
11:58 pdurbin
donsizemore doesn't even use the Perl script. His dataverse-ansible stuff does the equivalent. That's my understanding anyway.
11:59 otheus
I can't hack ansible yet, so I stay away.
11:59 donsizemore
i kind of cobbled ansible around the installer. it does some stuff the install script would do, then calls the scripts it calls.
11:59 pdurbin
Sure, the same underlying scripts are called, which is good.
12:00 donsizemore
@pdurbin when you get time look over https://github.com/OdumInstitute/dataverse/blob/5162_cant_publish_using_EC2_script/scripts/installer/ec2-create-instance.sh ?
12:00 pdurbin
But yeah, I agree, writing installers isn't much fun. We are definitely open to ideas on how to do this better. "The script is to a large degree a derivative of the old installer from DVN 3.x. It is written in Perl. If someone in the community is eager to rewrite it, perhaps in a different language, please get in touch. :)" http://guides.dataverse.org/en/4.9.4/installation/installation-main.html
12:00 donsizemore
@pdurbin i spent too much time yesterday banging my head on the delete-jvm-option in https://github.com/IQSS/dataverse-ansible/blob/master/templates/configure_doi.sh.j2
12:01 * pdurbin
looks
12:01 pdurbin
I'm glad to see my friends -b and -r are back. :)
12:02 donsizemore
@pdurbin if your branch testing will only run against mds.test.datacite.org you're good to go. but ansible is putting the baseurl variable in template then firing that as a shell script, and i didn't hit on the right combination of escape syntax to make it remove the dataverse-provided mds.test.datacite.org jvm-option
12:02 pdurbin
For now I think we'll be using mds.test so that's fine.
12:02 pdurbin
\
12:03 pdurbin
sorry, cat on my keyboard
12:03 donsizemore
(and i wanted to get home because we lost a tree in the hurricane and i wanted to make sure the house was okay)
12:03 pdurbin
donsizemore: this isn't your technical debt, it's mine but any thoughts on "TODO: Add some error checking for this ssh command"? Should we run each command individually?
12:04 donsizemore
@pdurbin i can take that on if you want
12:04 pdurbin
for this pull request or the next one?
12:05 pdurbin
I don't want to keep piling it on.
12:05 donsizemore
(or we could simply set -e )
12:05 pdurbin
hmm? -e?
12:06 donsizemore
that may be shell only, but it's an easy test
12:06 poikilotherm
donsizemore I have some exp in Puppet and try to come up with Ansible. I saw you working on the ansible module and actually was beginning to invent my own because of granularity etc. But the thing that bothers me most about the installer script and the ansible module calling it is the lack of idempotency... Any chances we maybe could work together to improve that?
12:06 donsizemore
that makes a shell script exit on any error
12:06 donsizemore
@poikilotherm different people want the installer (and dataverse-ansible) for different reasons
12:07 pdurbin
donsizemore: let's defer until the next pull request. Ready to make one?
12:07 donsizemore
@poikilotherm so far the highest demand has been for development, which requires no idempotency
12:07 donsizemore
@poikilotherm the dataverse installer itself isn't idempotent, and dataverse-ansible is just a wrapper.
12:08 pdurbin
donsizemore: sure but you've got interest in production installs such as Jamie from UCLA
12:08 donsizemore
@poikilotherm but i'm all for idempotency and i'd love to work together on that
12:08 donsizemore
@pdurbin pull requests welcome =)
12:08 pdurbin
:)
12:08 pdurbin
I think I made one once. :)
12:09 cdsp-rmo joined #dataverse
12:09 pdurbin
Oh, all I did was add a Travis CI button. Lame.
12:10 poikilotherm
Actually I think most of the stuff in the installer is easy to get inside ansible and is idempotent. The biggest blocker for me seemed to be all of the `asadmin` calls.
12:10 poikilotherm
(err - would be idempotent when used properly in ansible)
12:11 pdurbin
How are the asadmin calls a blocker? I'm confused.
12:11 donsizemore
@poikilotherm dataverse-ansible itself doesn't do much by way of asadmin calls(?)
12:11 poikilotherm
Aye. But the installer scripts could be replaced by ansible playbooks that ideally are idempotent
12:12 poikilotherm
(and most certainly easier to maintain than a perl script)
12:12 pdurbin
cdsp-rmo: hi! Just a quick reminder about docs. :)
12:12 donsizemore
@poikilotherm then i'm tracking the dataverse installer... i intentionally tried to avoid that
12:12 donsizemore
@pdurbin bash supports '-e': ./ec2-create-instance.sh: line 3: bad: command not found
12:12 pdurbin
poikilotherm: that would force people like me and otheus to learn Ansible. Which may not be terrible. :)
12:12 cdsp-rmo
pdurbin: hi ! Yep, doing this till I go to my meeting
12:13 poikilotherm
asadmin does not check if e.g. a JVM option is already present, it will just add it again and again
12:13 otheus
*throws up in his mouth*
12:13 pdurbin
cdsp-rmo: thanks!
12:13 cdsp-rmo
and hi everyone
12:13 pdurbin
poikilotherm: checks could be added. Good idea.
12:13 pdurbin
otheus: you'd rather stick with Perl? :)
12:13 poikilotherm
Well, maybe its better to edit a config file if you can ;-)
12:14 donsizemore
@poikilotherm correct, i ran into that with the datacite provider url
12:14 poikilotherm
Basically the asadmin calls don't do other things than write into the domain config
12:14 donsizemore
@poikilotherm editing domain.xml was plan A but ansible's XML module wasn't cooperating
12:15 poikilotherm
pdurbin: adding checks with asadmin is pretty ugly stuff todo inside a perl script. clumbsy... better use dedicated software like puppet or ansible for such things
12:15 otheus
pdurbin: heh, i love perl. But I recognize it's becoming obsolete. I'd stick with shell scripting for an install, but my recent work on MacOS/BSD taught me just how backwards the pure POSIX/BSD world is w.r.t Gnu tools.
12:15 poikilotherm
Yeah, i read about that
12:15 pdurbin
poikilotherm: in DVN 3.x the installer made a lot of hacks to domain.xml but it was terrible and from my perspective the asadmin approach is much cleaner. I'm happy to link you to the old hacks if you'd like. :)
12:15 donsizemore
@otheus what made you throw up in your mouth?
12:15 otheus
Oh, the suggestion of having to learn Ansible. :)
12:16 otheus
I won't be happy with anything related to Python until 2105, when the last system running 2.7 is finally shut off
12:16 poikilotherm
otheus: I favor Puppet, but a lot of people tell me that the learning curve of ansbile is much lower...
12:16 donsizemore
well, i wrote dataverse-ansible because i was asked to, and am maintaining it on a volunteer basis
12:16 pdurbin
I don't mind Ansible but I'm not sure I want developers with Macs coming to me asking "why is step 1 of getting a dev environment set up to install Ansible and can you help me?"
12:17 poikilotherm
pdurbin: sure. but setting up a dev environment with a perl and shell script is not much better... IMHO your dev env should be maven, java, an IDE and docker.
12:17 pdurbin
donsizemore: we really appreciate all of your dataverse-ansible effort! We lean on it heavily in the new ec2 script, obviously! Thank you!
12:18 otheus
At least perl already exists on MacOS. *shrug*
12:18 donsizemore
@otheus as i often point out, i have 20 year-old perl which still runs on modern systems
12:18 pdurbin
poikilotherm: some sort of target like `mvn setup-dev-environment`?
12:18 poikilotherm
donsizemore: your ansbile playbook is good! I just miss the idempotency, so I can use it together with the rest of the modules
12:19 donsizemore
@poikilotherm i admire idempotency as well, and would happily work together to achieve it
12:19 otheus
Yes don, and that's why -- despite its lack of readability -- perl is still a sysadmin's choice
12:20 otheus
Where as ruby, python, and puppet make incompatible language choices 3 times a year it seems.
12:20 otheus
*changes
12:20 donsizemore
@pdurbin so, install question.
12:21 pdurbin
hit me
12:21 donsizemore
@pdurbin the install script creates a datacite baseurl jvm-option for mds.test.datacite.org, but users are left to add their own username/password options
12:21 pdurbin
yeah
12:22 donsizemore
@pdurbin should the installer do that? it's not really a convenience, as datacite doesn't (yet) allow publishing without credentials
12:22 pdurbin
Maybe not. What should it do instead?
12:24 donsizemore
@pdurbin config.html doesn't mention the baseurl, either. might be cleaner to pull the baseurl jvm-option from the install script and add it in the docs?
12:25 otheus
There's no reason the installer can't be designed to wrap idempotent directives which are invoked by ansible/puppet/whatever
12:25 donsizemore
@otheus on my part, that reason is available time =)
12:25 otheus
oh right. :)
12:25 otheus
there's no reason with inifinite resources...
12:26 otheus
ANYWAY, I'm retrying the install step and omg, this is not going well.
12:26 otheus
I dropped the database to get around the existing-database problem.
12:26 otheus
Once again, the install script stopped glassfish and failed to restart it.
12:27 poikilotherm
Oh there is a reason why the Gitlab guys bake rpm/debs and let Chef do the heavy lifting inside...
12:27 otheus
More importantly, "remote failure: Error occurred during deployment: Application with name dataverse is already registered. Either specify that redeployment must be forced, or redeply the application". Can someone please tlel me how to do that?
12:29 pdurbin
donsizemore: if you're saying we don't allow the doi baseurl to be configured by the Perl script, we should. If you're saying something else, I'm sorry that I'm confused. :) Also, yes docs. Always docs.
12:30 pdurbin
otheus: you may or may not get something out of https://github.com/IQSS/dataverse/blob/v4.9.4/scripts/deploy/phoenix.dataverse.org/rebuild for rebuilding
12:30 pdurbin
otheus: try `./asadmin deploy --force path/to/dataverse.war`
12:31 donsizemore
@pdurbin the test url is hard-coded at https://github.com/OdumInstitute/dataverse/blob/5162_cant_publish_using_EC2_script/scripts/installer/glassfish-setup.sh#L69 — would be great to make all three configurable
12:31 pdurbin
If it helps, I run stuff like this a lot: git pull && mvn clean && scripts/installer/custom-build-number && mvn package && /usr/local/glassfish4/glassfish/bin/asadmin deploy --force target/dataverse-4.9.4.war
12:32 pdurbin
donsizemore: ah, yes, that shouldn't be hard coded
12:33 donsizemore
@pdurbin want me to open an issue to make all three config_variables like the others?
12:34 pdurbin
donsizemore: please!
12:34 otheus
Thanks people. I added '--force' to my copy of the install script.
12:34 pdurbin
otheus: awesome
12:34 otheus
I believe the error traces back to >>>
12:34 otheus
remote failure: Application dataverse-4.0 is not deployed on this target [server]
12:34 otheus
Command undeploy failed.
12:35 otheus
Ugh, sigh.
12:35 otheus
remote failure: Error occurred during deployment: Exception while loading the app : Error in linking security policy for dataverse -- Inconsistent Module State. Please see server.log for more details.
12:38 otheus
@pdurbin the problem with restarting glassfish seems to be related to a conflict with systemd and asadmin
12:38 otheus
so if systemd started glassfish, asadmin can stop it and start it, but it doesnt behave well enough for a proper restart
12:38 donsizemore
@otheus for inconsistent module state, stop glassfish, rm -rf $GLASSFISH_HOME/glassfish/domains/domain1/generated and start
12:40 otheus
donsizemore: thanks. on the next run I got "EJB Timer Service is not available. Please see server.log for more details."
12:40 donsizemore
@otheus that typically means glassfish didn't clean its entries from the EJB postgres table when it shutdown and/or crashed
12:42 otheus
ok, so "rm -rf generated" and dropping database *should* fix that?
12:43 poikilotherm
pdurbin: watching a talk by thilo frotscher about using the microservices approach similar to spring boot for Java EE
12:43 poikilotherm
https://jaxenter.de/java-ee-microservices-frotscher-60969 (sorry, it's in german)
12:44 otheus
oje scheiße!
12:44 poikilotherm
I think you dipped your nose in the right bucket - this could definitly be interesting...
12:46 otheus
@donsizemore and @pdurbin success! (?)
12:46 otheus
installer then tries "Executing curl -X PUT -d ADMIN_EMAIL http://localhost:8080/api/admin/settings/:SystemEmail
12:46 otheus
" which outputs an HTML page with an error "ERR_CANNOT_FORWARD"
12:48 otheus
<li id="permission-denied">The administrator may not allow this cache to make direct connections to origin servers.</li>
12:49 otheus
But I do see "Your Dataverse has been configured to use DataCite, to register DOI global identifiers in the
12:49 otheus
test name space "10.5072" with the "shoulder" "FK2"
12:49 otheus
"
12:49 otheus
in the same output.
12:55 pdurbin
otheus: that's a new one for me... that cache stuff
12:56 otheus
*snap* No dice. I visit the service on port 8080 (from another host) and I get 500 error, with tons of stack traces in server.log.
12:56 pdurbin
poikilotherm: isn't DropWizard the Java EE equivalent of Spring Boot? Not sure.
12:56 otheus
A lot of complaints about EJB service bean.
12:56 poikilotherm
pdurbin: better look at WildFly Swarm and Payara Micro, etc
12:57 pdurbin
otheus: so have you had a successful install of Dataverse at all? :/
12:57 poikilotherm
pdurbin: this leads to a java -jar ability
12:57 otheus
I get no login, so... no. :(
12:57 pdurbin
bah
12:58 pdurbin
otheus: you *could* try `vagrant up` and see if that works. It should.
12:58 otheus
Part of my task is to make this process reproduicable for other universities.
12:58 otheus
So vagrant solution only gets me to say 'it works'
12:59 pdurbin
otheus: ah, well we'd love to help you with that
12:59 otheus
but my employer wants something more significant than that
12:59 otheus
And I'm working at a uni, and they see it as their mission to contribute back to this thing, so I will do that
12:59 pdurbin
great!
12:59 otheus
Provided it's selected among the 3 alternatives
12:59 * pdurbin
crosses fingers
13:00 pdurbin
Please save us from ourselves. :)
13:00 otheus
lol
13:00 otheus
direct chat isn't working. dont know why.
13:00 otheus
I'm not using a proxy to connect and maybe our firewall is blocking it
13:00 pdurbin
Lots of spam on freenode lately. They added +R on everyone's account.
13:01 otheus
https://gist.github.com/otheus/c31e67e13f8f47954623afd2a001684f
13:02 otheus
^ that's the first stack trace I see. Does it tell you anything?
13:02 pdurbin
otheus: you might be suffering from https://github.com/IQSS/dataverse/issues/3148
13:03 donsizemore joined #dataverse
13:03 otheus
I think you hit it on the nose
13:03 pdurbin
otheus: you could give our new EC2 script a try if you want: http://guides.dataverse.org/en/4.9.4/developers/deployment.html
13:05 pdurbin
The nice thing about Vagrant is that you get a clean environment after a `vagrant destroy`. Maybe you could tell us some more about the server you're using. CentOS 7?
13:08 pdurbin
The guy who started the EC2 script said a "small" is too small. Weird errors. Had to bump up to a "medium".
13:12 otheus
yes CentOS7
13:13 otheus
wait, so does this issue 3148 actually have a general resolution?
13:19 jri joined #dataverse
13:22 otheus
@pdurbin: re issue 3148 refers to https://groups.google.com/forum/#!msg/dataverse-community/Esuj7HxZiF4/s3MUjtnzBgAJ . I'm getting the exact (AFAIK) same error
13:26 poikilotherm
pdurbin: as we really want to dockerize our setup, this microprofile stuff is very interesting for us... using this for dev could be interesting too, as the installation of glassfish could be skipped. maybe I can prioritize this before working on the external pid and draft doi stuff to get this going for us (test deployments etc)
13:26 poikilotherm
what do you guys think is more important?
13:28 poikilotherm
shall I open an issue first and maybe discuss there?
13:29 poikilotherm
I think this is related to the "update to glassfish 5" stuff, too
13:32 poikilotherm
Thorntail (successor to Wildfly Swarm) sound pretty promising
13:32 donsizemore joined #dataverse
13:39 poikilotherm
Payara Micro seems to be a good alternative approach more close of course to the existing Glassfish usage. Although the maven plugins of Thorntail seem to make life easier...
13:40 poikilotherm
And it has some neat other stuff onboard, like clustering etc
13:42 cdsp-rmo joined #dataverse
13:43 otheus
My $0.02 I'm not sure dockerization is a good fit for this software based on deployment use-case. But, I am generally critical of puppet deployments of software for a number of reasons, and find them suitable for environments where hard resources are outsourced. Again, my $0.02.
13:44 poikilotherm
otheus: using a fatjar/uberjar has nothing to do with puppet/ansible/... It is entirely possible to instead just use another maven profile to receive the oldschool WAR file.
13:45 poikilotherm
For everyone going to docker, kubernetes, automated testing, etc. fatjars make life much easier
13:46 otheus
sorry, it's "Unadulterated opinion Friday"
13:47 poikilotherm
*G*
13:47 pdurbin
poikilotherm: how do you feel about the time and bandwidth spent pushing fat jars to Docker Hub?
13:48 poikilotherm
Actually you can provide hollow jars
13:48 otheus
@pdurbin I think I see something which may be the source of my problem, and why setup-all.sh fails to setup the root dataverse...
13:48 poikilotherm
By that deps and the surrounding server jars are in one layer and on top of that is the application war layer
13:49 otheus
The glassfish sever complains about not being able to handle requests to 'localhost.uibk.ac.at:8080'
13:49 poikilotherm
Thus you only send the small changed application layer to the hub
13:49 poikilotherm
At least this is possible with Thorntail, haven't looked that up for Payara Micro
13:49 pdurbin
otheus: that's... weird... I think
13:49 otheus
which has a 'permission denied' << >The administrator may not allow this cache to make direct connections to origin servers.< >>
13:50 otheus
And it's trying to go through our proxy server to get to the cache server. i think.
13:51 poikilotherm
Ah, the same seems to apply for Payara Micro, you don't need to generate a Uber jar with it.
13:51 poikilotherm
Sry guys, read you all on monday. Gotta go, pick up kids
13:51 pdurbin
poikilotherm: bye!
13:51 poikilotherm
Will open an issue - maybe today or on Monday
13:52 pdurbin
otheus: the only proxy issue I remember is this one: https://github.com/IQSS/dataverse/issues/4707
13:52 otheus
i think i have the opposite problem.
13:53 pdurbin
otheus: so the creation of the root dataverse is the first thing that fails? I thought the installer hit some Dataverse APIs before that.
13:53 otheus
well, AFAIK, setup-all fails on its first url call
13:54 pdurbin
ah, ok, that makes more sense
13:55 pdurbin
maybe the "Setup the metadata blocks" step
13:55 pdurbin
otheus: you can see how the output of "setup all" is supposed to look at https://build.hmdc.harvard.edu:8443/job/phoenix.dataverse.org-deploy-develop/279/console
13:56 otheus
right, it fails on the first step
13:56 pdurbin
ok, and it's something with your proxy maybe?
13:57 otheus
it does "deleting all data from Solr" and a URL
13:57 otheus
+ curl 'http://localhost:8983/solr/collection1/update/json?commit=true' -H 'Content-type: application/json' -X POST -d '{"delete": { "query":"*:*"}}'
13:57 pdurbin
Does deleting data from Solr work?
13:57 otheus
And then I get HTML output
13:58 otheus
i think from my proxy server
13:58 otheus
omg.
13:58 pdurbin
Huh. When I delete data from Solr I see this: { "responseHeader":{ "status":0, "QTime":1}}
13:59 pdurbin
(on multiple lines)
13:59 otheus
no_proxy is not set
13:59 otheus
*shaking my head*
14:00 otheus
I'm the guy who wrote the deployment script for setting up the proxy env variables. So damn, it's my fault
14:00 pdurbin
What's the proxy for anyway?
14:03 otheus
running inside a non-routable ip address
14:04 otheus
NOW IT WORKS
14:05 otheus
*fist pump* *hangs head in shame* *kicks self in disgust* *time for a beer*
14:05 pdurbin
heh, good job
14:06 pdurbin
otheus: oh, before you grab your beer, since you're comparing Dataverse to two other platforms, you may be interested in our blog post on this. One sec.
14:06 pdurbin
https://dataverse.org/blog/comparative-review-various-data-repositories
14:06 pdurbin
It's a bit oriented toward *services* rather than *software* but I hope it's helpful.
14:18 cdsp-rmo
pdurbin: I found how to fix my french logging problem, so I can work without changing the dataverse log system
14:18 pdurbin
cdsp-rmo: great! What should we do with that issue?
14:19 cdsp-rmo
the fix is: do a full reinstall of the dataverse (undeploy, stop, delete generated, start, deploy, stop, start, stop solr, download schema.xml, start solr, curl -X DELETE http://localhost:8080/api/admin/index/timestamps , curl http://localhost:8080/api/admin/index/continue)
14:19 cdsp-rmo
so, it's not a REAL fix
14:19 cdsp-rmo
and, if I ever stop and start glassfish
14:19 cdsp-rmo
the problem reappears
14:20 cdsp-rmo
I don't understand what's happening
14:20 cdsp-rmo
if I don't do all the steps of the reinstall
14:20 cdsp-rmo
the problem keeps going
14:21 cdsp-rmo
there is really something strange here ...
14:21 pdurbin
ok, I agree... not a real fix... bummer
14:22 pdurbin
I wonder if anyone else is suffering from this.
14:23 otheus
@pdurbin I'm tasked also with evaluating Dspace and Invenio
14:24 pdurbin
Huh, I don't think I've heard of Invenio.
14:25 pdurbin
Ah. Yes I have. A friend at CERN uses it.
14:26 pameyer joined #dataverse
14:28 cdsp-rmo
pdurbin: I add the full procedure to "fix" the problem to the ticket ? In case someone has the mysterious problem too, and waiting for the real fix
14:29 otheus
FYI: https://github.com/IQSS/dataverse/issues/3148#issuecomment-429344986
14:30 pdurbin
cdsp-rmo: yes please!
14:31 pdurbin
otheus: ah, thanks. How do you feel about opening an issue to document this or tweak the shell scripts?
14:35 pameyer
otheus: last I checked, non-interactive installation with default.config was non-interactive (aka - no `asadmin login` required. but it's been a week or 2 since I checked; can't rule out the possibility that it's broken since then
14:36 otheus
Should I put the issues on the https://github.com/IQSS/dataverse/issues project or elsewhere?
14:36 pdurbin
pameyer: 4.9.4 has only been out 9 days. :) That's what otheus is using.
14:36 pameyer
and for what it's worth, my experience with the dcm lead me to think that using ansible as a user-facing (vs developer facing) installation process is a bad idea
14:37 pdurbin
otheus: yes, please create the issue there. Thanks!
14:37 pdurbin
otheus: oh, pameyer has been building RPMs too
14:37 pameyer
pdurbin: people aren't using develop? ;)
14:37 pdurbin
A tagged release should be safer. Should be. :)
14:39 pameyer
yeah - as much as I'd like to see dataverse and it's dependencies RPM-ized; I think there's a big chunk of tech debt that would need to be paid down before that becomes feasible
14:41 pameyer
... and with more lagged unsolicited opinions about stuff from irc logs: docker's been a big help for development, but I'm not sold on it for production dataverse yet
14:41 pameyer
... and fortran > perl ;)
14:42 pdurbin
no installers in fortran. sorry
14:43 pameyer
good point. they're good at different things (although I was impressed to find what was effectively a scripting language with the interpreter written in fortran)
14:47 pameyer
otheus: non-interactive installation just worked for me with no prompts (on c95e53d805579b1f6718c7b8f49b6d4126db52ba; so I'd assume 4.9.4 should be ok)
14:48 otheus
Is there some special variable used for the admin account?
14:49 pameyer
not in conf/docker-aio/install.bash
14:49 pdurbin
I don't really understand the Glassfish admin account stuff, to be honest. I run Glassfish on my Mac and as far as I know I don't need a password to start and stop Glassfish. Maybe the password is put in a file somewhere by the installer? I don't know.
14:50 pameyer
pdurbin: if you want glassfish "secure remote admin", you need to do some asadmin login magic to enable a password
14:50 pdurbin
Yeah, do not want. Not on my laptop anyway.
14:51 pameyer
I'm not sure when I'd want it - if there are advantages over connecting to the server and using asadmin commands, I'm not sure what they are
14:51 otheus
./install runs "asadmin login" and THAT requests a password (but empty is accepted)
14:51 pdurbin
yeah, I'm not sure why it runs "asadmin login"
14:51 otheus
or?
14:53 otheus
or it happens at some point in glassfish-setup.sh, but again, via adadmin
14:57 otheus
FYI: other issue: https://github.com/IQSS/dataverse/issues/5175
15:05 pdurbin
otheus: thanks! I don't suppose that https://github.com/IQSS/dataverse/pull/5167 magically fixes this
15:06 otheus
probably not since I was the one who made those changes :P
15:06 pdurbin
bummer :)
15:06 otheus
But the fact that it never worked in the first palce indicates that this install process was never well-tested with it
15:06 otheus
It may be necessary to simply say in installation docs that the systemd service must be stopped before running install
15:07 pdurbin
otheus: I think the problem is that our test servers are still on CentOS 6.
15:07 pameyer
I'm still using init for glassfish on cent7. may be suboptimal, but has been working for me so far
15:08 otheus
Anyway, 2 other issues created which we already discussed. I would love to work on improving the install process once my other fires are put out. Feel free to contact me: otheus.shelling
15:08 * pdurbin
puts out a couple fires
15:14 pdurbin
cdsp-rmo: thanks for leaving a comment on that login issue
15:15 cdsp-rmo
yup
15:15 cdsp-rmo
I answer to the multiple depositors issue aswell
15:16 pdurbin
Ah, great. Yes. Very helpful. Thanks.
15:17 pdurbin
cdsp-rmo: time for your weekend, right?
15:18 cdsp-rmo
I still has some time to finish my doc and push it
15:18 cdsp-rmo
;)
15:18 pdurbin
nice!
15:18 pdurbin
donsizemore: "Switched to a new branch '5162_cant_publish_using_EC2_script'" :) Thanks!
15:19 donsizemore
@pdurbin i'll tackle the installer changes once i knock a couple of other things out of the way
15:19 donsizemore
@pameyer thanks for testing that. i was going to grab the 4.9 or so release and test
15:20 pameyer
@donsizemore it's usually a quick and easy thing for me to test
15:20 pameyer
hasn't yet been worth the trade off of 100% automating it
15:21 cdsp-rmo
pdurbin : about merging the json import and the xml import as one api, a fast answer may be useful, till I still has an idea of what I've done :D
15:21 cdsp-rmo
*ave
15:21 cdsp-rmo
**have
15:24 pdurbin
cdsp-rmo: unfortunately, I don't think you'll get an answer until next week. Sorry.
15:24 cdsp-rmo
okay
15:47 isullivan joined #dataverse
15:54 cdsp-rmo
doc updated
15:54 cdsp-rmo
now I disappear
15:54 cdsp-rmo
have a nice weekend everyone ;)
15:56 pdurbin
Thanks! You too!
15:59 pdurbin
donsizemore: this worked just fine: ec2-create-instance.sh -b 5115-move-dataset
15:59 pdurbin
gonna try -r next
16:01 pdurbin
donsizemore: how do you feel about me tweaking the dev guide to mention the new "env" file?
16:30 pdurbin
donsizemore: -r seems to work fine too. I think I'm gonna tweak the docs.
17:11 CaseyLF joined #dataverse
17:12 CaseyLF left #dataverse
17:18 donsizemore
@pdurbin you do boss! (sry i forgot to document it)
17:19 donsizemore
^da
17:31 pameyer joined #dataverse
17:48 pdurbin
donsizemore: no worries! I hope I did ok.
17:49 isullivan joined #dataverse
17:56 isullivan joined #dataverse
18:13 andrewSC joined #dataverse
18:50 donsizemore
@pdurbin ok? you're great to work with.
18:51 pdurbin
Ah, well I meant what I wrote in that doc but thanks. :)
18:51 pdurbin
Did you ever figure out if there's another mailing list? I'm not sure what Jon meant.
18:56 donsizemore
i think there may have been an e-mail chain in addition to the github issues
18:59 pdurbin
Ah. Ok. Plenty of into in the GitHub issue. Looking forward to hearing about CoRe2.
20:11 donsizemore joined #dataverse
20:12 donsizemore joined #dataverse
20:13 donsizemore
@pdurbin my turn to work from a bar! (they have blood orange bold rock on tap and i'll be working on #5173 until my friends show up)
20:15 pdurbin
way to be
20:33 pdurbin
have a good weekend, all!
20:33 pdurbin left #dataverse
21:47 pameyer left #dataverse
23:52 isullivan joined #dataverse
