Time 
            S 
            Nick 
            Message 
         
        
07:22 poikilotherm joined #dataverse 
 
        
08:37 jri joined #dataverse 
 
        
11:52 pdurbin 
poikilotherm: morning. I was thinking... if you have other people on your branch, they'll be able to just run "git pull" if you avoid using "git --force push". 
 
        
11:53 poikilotherm 
Yeah 
 
        
11:54 poikilotherm 
Morning 
 
        
11:54 poikilotherm 
From now on this should be possible 
 
        
11:55 pdurbin 
Ok, good. It also helps in the QA scenario we were talking about earlier. To help answer "What changed in the branch since I last tested it?" 
 
        
11:57 poikilotherm 
It was quite hard to get this deployed to the app server and needed some very ugly changes 
 
        
11:57 poikilotherm 
But those should be all set now 
 
        
11:58 pdurbin 
Ok, yesterday the war was deploying but setup-all and friend were failing. 
 
        
11:58 pdurbin 
friends* 
 
        
11:59 poikilotherm 
Oh did you try to apply the script to the docker image based server? 
 
        
12:01 pdurbin 
No. I'm at "Your branch and 'poikilotherm/5292-small-container' have diverged" ... conflicts because I did a pull. 
 
        
12:01 poikilotherm 
Aye ;-) 
 
        
12:01 poikilotherm 
Please use git reset --hard <remote name here>/5292-small-container instead 
 
        
12:02 poikilotherm 
Oh and maybe a git fetch before 
 
        
12:02 pdurbin 
done, thanks 
 
        
12:02 poikilotherm 
What is the git hash of the commit? 
 
        
12:03 poikilotherm 
Should be 20c25ee5 
 
        
12:05 pdurbin 
yeah. 20c25ee Support Dataverse system properties as ENV vars. 
 
        
12:05 poikilotherm 
Great :-) 
 
        
12:07 pdurbin 
javax.persistence.NoResultException: getSingleResult() did not retrieve any entities. 
 
        
12:07 poikilotherm 
Yeah - the setup-all.sh stuff is missing 
 
        
12:07 pdurbin 
ok, so expected behavior 
 
        
12:07 poikilotherm 
Or in other words: we need a fixture 
 
        
12:08 pdurbin 
that's what they're called in Django 
 
        
12:08 poikilotherm 
Yeah, but it is a more general term, too 
 
        
12:09 pdurbin 
I wouldn't be surprised. 
 
        
12:10 pdurbin 
scripts/database/reference_data.sql  could be considered a fixture, right? 
 
        
12:10 poikilotherm 
Totally 
 
        
12:11 pdurbin 
Ok. Without it, SWORD won't work. 
 
        
12:11 poikilotherm 
Altough it mixed 
 
        
12:11 poikilotherm 
It has fixtures inside, but also some migrations (indexes being created) 
 
        
12:12 pdurbin 
yeah, there are some indexes that can't be expressed in JPA 
 
        
12:20 poikilotherm 
Maybe this splited and any DDL plus sequence.sql  can be added to Flyway migrations 
 
        
12:21 poikilotherm 
Your current SQL  scripts made for migrations could be reused with Flyway, too IIRC 
 
        
12:21 poikilotherm 
But still - I need a fixture for demo/dev purposes 
 
        
12:21 poikilotherm 
That was what I was talking about with pameyer yesterday 
 
        
12:24 pdurbin 
Have you considered jumping on the flyway branch and playing around with it? 
 
        
12:24 poikilotherm 
Not yet 
 
        
12:25 poikilotherm 
First looking at the setup-all.sh stuff - this cannot be fixed with Flyway 
 
        
13:17 poikilotherm 
Oi this is driving me nuts 
 
        
13:18 poikilotherm 
pdurbin do you think it would be OK to have some startup bean that will load initial data from JSON  file? 
 
        
13:18 poikilotherm 
(of course based on a env var and possibly more) 
 
        
13:19 poikilotherm 
Building a database snapshot everytime you need to rebuild the image (e. g. because you are developing) is pretty slow 
 
        
13:19 poikilotherm 
And cumbersome 
 
        
13:19 poikilotherm 
Using the existing JSON  files seems to be pretty cool 
 
        
13:20 poikilotherm 
Those are a definite reference 
 
        
13:20 poikilotherm 
I am thinking about a bean at startup time that will load those and call the internal Java API  without REST to store this initial config in the database 
 
        
13:28 poikilotherm 
I would really like to avoid hacking a script that waits in background for the app to be deployed and then call the REST APIs for initial config 
 
        
13:48 pdurbin 
poikilotherm: have you ever installed Wordpress? 
 
        
13:48 poikilotherm 
Sure 
 
        
13:48 poikilotherm 
Multiple times 
 
        
13:49 pdurbin 
The user experience is pretty nice. Web based. 
 
        
13:49 pdurbin 
"5 minute install" I think they say 
 
        
13:49 pdurbin 
You get a GUI  that asks for a database server (could be localhost) and a db username and db password. 
 
        
13:49 pdurbin 
And that's it. 
 
        
13:49 pdurbin 
You're up and running. 
 
        
13:50 pdurbin 
Could what you're talking about tie into a user experience like this? 
 
        
13:50 poikilotherm 
Well perhaps not with a GUI  asking for DB  credentials 
 
        
13:50 pdurbin 
Oh. Why not? 
 
        
13:50 poikilotherm 
Because it is not PHP  :-D 
 
        
13:51 poikilotherm 
You need a persistance framework and JDBC resources 
 
        
13:51 pdurbin 
Even with the magic microprofile config api? 
 
        
13:51 poikilotherm 
I don't think so. I am a bit nearer to this with my JDBC resources living in glassfish-resources.xml now 
 
        
13:52 poikilotherm 
You just need to provide credentials as an ENV var 
 
        
13:52 pdurbin 
Bummer. Weird that this isn't possible with Java. Kudos to PHP  I guess. 
 
        
13:53 pdurbin 
Why is it that PHP  can do this but Java can't? 
 
        
13:53 poikilotherm 
I dunno if this is impossible, I just don't know if there is an option for this. And from what I know about Java EE and app servers, this is at least not easy... 
 
        
13:53 poikilotherm 
Well you could do that by having an installer based on Java that will write some sort of config 
 
        
13:53 poikilotherm 
That's how PHP  does it 
 
        
13:54 poikilotherm 
There could be workaround for this 
 
        
13:54 poikilotherm 
That config could be usable via MicroProfile API  
 
        
13:54 poikilotherm 
But you will need an installer app 
 
        
13:54 poikilotherm 
Java is a compiled language in contrast to PHP ... 
 
        
13:55 poikilotherm 
That definitly makes a difference 
 
        
13:55 poikilotherm 
You can't "deploy" a PHP  app like you do with Java EE apps 
 
        
13:56 pdurbin 
Sure. I'm just wondering if we could have various "modes" or whatever when installing Dataverse: 
 
        
13:56 pdurbin 
- default.config for fully unattended installations 
 
        
13:57 pdurbin 
- run the installer at the command line and get prompted for answers, as now 
 
        
13:57 pdurbin 
- the first visitor to the Dataverse web GUI  gets a Wordpress-like experience 
 
        
13:58 pdurbin 
- other modes I can't even imagine without more coffee 
 
        
13:58 poikilotherm 
Well, having a default config deployed from within the app would already help, wouldn't it? 
 
        
13:59 pdurbin 
Absolutely. 
 
        
13:59 poikilotherm 
A good example might be JIRA and the like 
 
        
13:59 poikilotherm 
They definitly have a wordpress like installer 
 
        
13:59 pdurbin 
I've never installed JIRA. Or Jenkins. 
 
        
13:59 poikilotherm 
You can even configure your database connection from the browser 
 
        
13:59 poikilotherm 
I don't know how they did that 
 
        
14:00 pdurbin 
So it is possible. 
 
        
14:00 pdurbin 
We don't need PHP  for this. 
 
        
14:00 poikilotherm 
Most certainly yes - it might be difficult to achieve, but there seem to be options 
 
        
14:01 pdurbin 
We may never know. JIRA is not open source. 
 
        
14:03 pdurbin 
Is there anything written in Java that's open source that has a Wordpress-like installation experience? Where you put in your database server, username, and password and go? 
 
        
14:04 poikilotherm 
I am not aware of sth. 
 
        
14:05 poikilotherm 
Most certainly the atlassian guys setup a web app that kicks in when no config is present 
 
        
14:05 poikilotherm 
That one writes the config to disk 
 
        
14:05 poikilotherm 
And then they restart the app 
 
        
14:05 poikilotherm 
That last part is a real puzzle for me, dunno how to achieve that 
 
        
14:06 poikilotherm 
At least for all database-stored configuration stuff this is relatively easy to achieve for Dataverse, too 
 
        
14:07 poikilotherm 
One would need to change the startup, if no root dataverse is present 
 
        
14:08 poikilotherm 
The initial config for database resources etc could stay elsewhere 
 
        
14:08 pdurbin 
Right. If no root dataverse is present, it would mean you didn't use default.config or the installer that prompts you with questions. 
 
        
14:08 poikilotherm 
ACtually you could change the default.config to something inside the app 
 
        
14:08 poikilotherm 
At least for everything database related 
 
        
14:09 poikilotherm 
errr - stored 
 
        
14:09 pdurbin 
No objections. 
 
        
14:09 poikilotherm 
Most of the system properties in use right now could be changed to database IMHO, so the external app stuff would be reduced to setting up things like DB  
 
        
14:10 pdurbin 
Sure. 
 
        
14:10 poikilotherm 
Or be replaced by ConfigAPI and use a JDBC source to add these options 
 
        
14:11 poikilotherm 
However... 
 
        
14:11 poikilotherm 
Do you see a chance that people might like the idea of a Startup job or separate module deployed that will pipe in the setup-all.sh absolute basics? 
 
        
14:12 poikilotherm 
(Of course with a switch to turn it on/off) 
 
        
14:12 poikilotherm 
This could either enhance or replace the current setup-all.sh... 
 
        
14:14 pdurbin 
Would it deliver a Wordpress-like experience of being able to complete the installation of Dataverse in a browser? And not break the currently supported installation modes? 
 
        
14:14 poikilotherm 
It could offer one step towards that 
 
        
14:15 poikilotherm 
About breaking: that's what the switch is used for... It defaults to off 
 
        
14:18 pdurbin 
You know I like small steps. 
 
        
14:18 poikilotherm 
Yes 
 
        
14:18 pdurbin 
Three points of contact while rock climbing, right? Not sure if you saw that. 
 
        
14:19 poikilotherm 
Errr - pardon? 
 
        
14:20 pdurbin 
"reminds me of the rock climbing principle that you want to have 3 points in contact at all times" http://irclog.iq.harvard.edu/dataverse/2018-11-28#i_80227  
 
        
14:21 pdurbin 
Hopefully you see what he's getting at. 
 
        
14:22 poikilotherm 
I'm not sure 
 
        
14:22 poikilotherm 
pameyer seems to like some of it and sees potential, but I am totally unsure if he is with me 
 
        
14:23 poikilotherm 
Or if this is just overwhellming 
 
        
14:23 pdurbin 
We should talk it out with him sometime. I have feedback for you too. 
 
        
14:23 poikilotherm 
Name place, time and topic :-D 
 
        
14:24 pdurbin 
The calendar doesn't say he's out today. How long until your weekend starts? 
 
        
14:25 poikilotherm 
~25 minutes 
 
        
14:26 pdurbin 
! 
 
        
14:26 jri joined #dataverse 
 
        
14:26 pdurbin 
Do you want my feedback now? 
 
        
14:26 poikilotherm 
As I don't know if this is good or bad, I am not sure if this should be a public recorded channel... 
 
        
14:27 poikilotherm 
+on 
 
        
14:27 pdurbin 
It's mostly good! But I can correct some misconceptions. 
 
        
14:28 poikilotherm 
Misconceptions sounds technical, so at least I haven't upset anyway, I hope? 
 
        
14:28 poikilotherm 
-anyway + anyone 
 
        
14:28 pdurbin 
I doubt anyone is upset. 
 
        
14:29 pdurbin 
The first challenge is to get people to read what you wrote. :) 
 
        
14:29 poikilotherm 
Sorry... 
 
        
14:30 poikilotherm 
I just had to braindump 
 
        
14:30 poikilotherm 
This whole stuff is moving around in my head and there are so many things to keep in mind... 
 
        
14:30 pdurbin 
No need to apologize. Pleas see my post about "documentation driven development" at https://groups.google.com/d/msg/dataverse-community/rQWNllAyTu0/RMD0GEFzAgAJ  :) 
 
        
14:32 poikilotherm 
Ok, go ahead. 
 
        
14:33 pdurbin 
Ok, for context, I'm giving feedback on your "Future use of Glassfish and/or Payara" post at https://groups.google.com/d/msg/dataverse-dev/L6rNXbbZG0o/UFVfI0_gAQAJ  
 
        
14:33 pdurbin 
poikilotherm: first, thanks for making this post in the first place and calling Dataverse a wonderful open source software project! 
 
        
14:34 pdurbin 
Thanks also to you and your institution for being willing to put effort into improving Dataverse! Time is money! 
 
        
14:36 pdurbin 
I'm with you on up-to-date software. Especially as Dataverse starts to host sensitive data (see https://dataverse.org/goals-roadmap-and-releases  ), having patched software will be more important than ever. I feel terrrible that we are holding the 35 production installtion back on an ancient version of Glassfish. 
 
        
14:37 pdurbin 
Yes, a stable platform is very important. In a recent GitHub survey, stability and security are the most important things to users by far. I mentioned this in a talk at JavaOne: http://wiki.greptilian.com/talks/2017/javaone-how-to-run-an-open-source-project/  
 
        
14:39 pdurbin 
I don't object to a microservice architecture but I'm also comfortable calling Dataverse a monolith. Our monolith's first ability to support plugins/add-ons/components in our "external tool" framework. We could and should do much more in terms of modularity. I don't think Java makes this particularly easy but it's possible. 
 
        
14:41 pdurbin 
"more people are opting for running Dataverse in Docker / Kubernetes / OpenShift" is not true for production installations. Of the 35 installations on the map, none of these are using these technologies. 
 
        
14:42 pdurbin 
"others might still prefer the classic installation for good reasons and that should continue to work." 
 
        
14:43 pdurbin 
that should say "must" instead of "should" because, again, everyone in production is using the classic installation. new installations ask how established installations do things 
 
        
14:44 pdurbin 
Yes, Glassfish 4.x is EOL. All "Oracle Glassfish" is EOL. The future for Glassifhs is under the name "Eclipse Glassfish". 
 
        
14:44 pdurbin 
Yes, devs are more interested in a small chunk of moving to a non EOL Glassfish-compatible server before even thinking about open liberty or wildfly. 
 
        
14:45 pdurbin 
Yes, we want the app server to be secure and not EOL. 
 
        
14:47 pdurbin 
It's too strong to say that some people at IQSS don't like the options of MicroProfile ConfigAPI. It sounds interesting to all, I'd say, but some seem willing to wait for it to land in Java EE 9 or whatever the next version is called. This would probably be Eclipse Glassfish 6. 
 
        
14:47 pdurbin 
Yes, devs mostly want to think about code, not app servers. 
 
        
14:47 pdurbin 
features, I'd say. bugs 
 
        
14:47 pdurbin 
grant requirements 
 
        
14:48 pdurbin 
"building an image" is not something that devs think about. not on this floor anyway. 
 
        
14:48 pdurbin 
Yes, it absolutely destroying me that it would take so long to upload changes to DockerHub when I was hacking on openshift stuff. Please deliver me from this agony. 
 
        
14:49 pdurbin 
I don't know what a builder is. "Dockerfiles are for builders". 
 
        
14:50 poikilotherm 
pdurbin: please carry on - I will read up the logs later. I'm sorry, but I have to pickup kids now. Will try to come back later and give some feedback on your feedback :-) 
 
        
14:50 pdurbin 
poikilotherm: this seems like a natural place to pause before I launch into "philosophy behind container images" 
 
        
14:50 pdurbin 
more to come, next week 
 
        
15:14 drew-jhu joined #dataverse 
 
        
15:31 * pdurbin 
waves at drew-jhu 
 
        
15:34 drew-jhu 
waves back 
 
        
15:35 pdurbin 
drew-jhu: I wanted to point you to a discussion about Google Analytics at https://groups.google.com/d/msg/dataverse-community/gCFpuhShecY/RQINkYYBBAAJ  
 
        
15:36 pdurbin 
The question was "how do I put more than one Google Analytics code in Dataverse?" 
 
        
15:36 pdurbin 
And the workaround is to use a custom footer instead of the ":GoogleAnalyticsCode" database setting. 
 
        
15:37 pdurbin 
You can read about custom footers at http://guides.dataverse.org/en/4.9.4/installation/config.html#custom-footer  
 
        
15:37 pdurbin 
The idea is that you can put whatever Javascript you want in a custom footer. 
 
        
15:37 drew-jhu 
good to know. thanks 
 
        
15:39 pdurbin 
Sure, https://github.com/IQSS/dataverse/issues/4660  is related. Basically, the Google Analytics javascript is frozen in time and we've been talking about maybe having users grab the latest from Google and put it into the footer or a new dedicated file for this purpose. 
 
        
15:40 drew-jhu 
we haven't been using our GA data at all. personally, I'm interested in exploring Matomo as an alternative, rather than committing further to GA 
 
        
15:40 drew-jhu 
but it is good to know about this option 
 
        
15:40 pdurbin 
Ah, we have support for Matomo too but in our guides we still call it by the old name, which is Piwik. 
 
        
15:43 pdurbin 
Please see http://guides.dataverse.org/en/4.9.4/installation/config.html#piwikanalyticsid  
 
        
15:43 drew-jhu 
i did note that with appreciation. at the moment, i am working hard to get our dataverse upgraded and our DOIs migrated to DataCite. hoping to move on to analytics sometime in the new year 
 
        
15:44 pdurbin 
Yeah, I'm not sure when EZID will stop working. 
 
        
15:45 drew-jhu 
my understanding is that they'd like everyone off by the new year. idk whether that is viable or not, but we are striving to meet that goal 
 
        
15:46 pdurbin 
Ah. ok. Do you know there's a new consortium that save you some money on DataCite DOIs? Please see http://dataversecommunity.global/blog/introducing-global-dataverse-community-consortium  
 
        
15:48 drew-jhu 
yes. the manager of our data folks is plugged in there. thx for pointing it out 
 
        
15:49 pdurbin 
Sure. And on the technical site, you might get something out of "Harvard Dataverse Move from EZID to DataCite" at https://groups.google.com/d/msg/dataverse-community/s-6_7cjCs7w/S0gkpmzhAQAJ  
 
        
15:50 pdurbin 
side* 
 
        
15:50 drew-jhu 
that is useful to me. thx! 
 
        
15:50 pdurbin 
sure, I hear it was pretty smooth 
 
        
15:50 pdurbin 
but it did take some coordination with DataCite on the timing 
 
        
15:51 pdurbin 
from what I understand 
 
        
15:52 drew-jhu 
that's what Mara (said manager) was saying. so far, my focus has been on our upgrade, but this info will be very useful for our next step 
 
        
15:52 pdurbin 
cool 
 
        
15:53 pdurbin 
drew-jhu: speaking of upgrades, we just merged a pull request for https://github.com/IQSS/dataverse/issues/4980  
 
        
16:03 drew-jhu 
glad to hear you all are working on this. i have ansibilized the process of upgrading each version one-by-one, but it's not ideal. once DSpace incorporated Flyway, it made upgrading *much* easier, so i'd echo the comments in support of that approach. i'm very close to going live with what we have, so i may not have a good oppotunity to try out your script for a bit. 
 
        
16:06 drew-jhu 
my repo doesn't have my latest changes yet, and may be drifting out-of-sync with IQSS's ansible, but it can be found here: https://github.com/dheles/dataverse-upgrade-ansible-role  
 
        
16:06 pdurbin 
drew-jhu: can you please leave that comment on https://github.com/IQSS/dataverse/issues/5344  ? That flyway issue is in code review right now so there's a good chance your comment will have an impact. 
 
        
16:11 drew-jhu 
https://github.com/IQSS/dataverse/issues/5344#issuecomment-443253269  
        
16:12 pdurbin 
Thanks! And your Ansible comment reminds me to ask if you plan to make a pull request for https://github.com/jhu-sheridan-libraries/dataverse-ansible-role/blob/v1.1/roles/dataverse/files/dv-mdblocks/jhuSoftware.tsv  
 
        
16:17 pdurbin 
No pressure but we've been discussing that custom metadata block of yours in the context of "Code Deposit - Github Integration": https://github.com/IQSS/dataverse/issues/2739  :) 
 
        
16:17 drew-jhu 
well, that was one of the last things Tim worked on before he moved on from supporting our Dataverse. We are keeping those metadata entries on our instance, but I'm afraid I don't know much about them, how frequently they are used, and how strongly we recommend them for wider adoption. I'll ask about and see what my better-informed colleagues think about the matter. 
 
        
16:20 pdurbin 
drew-jhu: sounds fine. We are also wondering how strongly you recommend what you've come up with. Thanks. :) 
 
        
16:23 drew-jhu 
i will def get back to you about that, then 
 
        
16:25 pdurbin 
drew-jhu: thanks, no rush. We have plenty to work on. :) 
 
        
16:28 drew-jhu 
heh. word. 
 
        
16:34 pdurbin 
drew-jhu: 15 of your 54 published datasets use "swTitle": https://archive.data.jhu.edu/dataverse/root?q=swTitle:*  
 
        
16:40 donsizemore joined #dataverse 
 
        
17:00 drew-jhu joined #dataverse 
 
        
17:34 pdurbin 
donsizemore: another Ansible user. Meet drew-jhu :) 
 
        
17:40 pdurbin 
donsizemore: oh, and I'm still not sure what you meant by "mapping" the other day. 
 
        
17:52 drew-jhu 
waves at donsizemore 
 
        
17:54 pdurbin 
drew-jhu: I added a comment to https://github.com/IQSS/dataverse/issues/5357  in case you'd like to make a pull request. :) 
 
        
18:15 pameyer joined #dataverse 
 
        
18:16 pameyer 
looks like I missed poikilotherm - which is good; because I didn't want him waiting to start his weekend on me getting caught up in other stuff 
 
        
18:19 pdurbin 
He doesn't sound blocked. "This whole stuff is moving around in my head." 
 
        
18:19 pdurbin 
He may not be a rock climber. :) 
 
        
18:30 poiki-mobile joined #dataverse 
 
        
18:31 poiki-mobile 
Ding dong. just a short message: thx pdurbin for your elaborated feedback :-) 
 
        
18:31 poiki-mobile 
Really appreciate that. Hope there is more from others coming, too. 
 
        
18:31 pdurbin 
poiki-mobile: sure, but I'm only halfway through your email. :) 
 
        
18:32 pdurbin 
Any feedback on my feedback? What's a builder? 
 
        
18:32 poiki-mobile 
a builder is someone that builds an image 
 
        
18:32 poiki-mobile 
just my wording, dunno if there is some official stuff for this. 
 
        
18:33 poiki-mobile 
maybe use devop instead of builder 
 
        
18:34 poiki-mobile 
anyway: someone that knows the knobs and turns of Dataverse well and likes Docker 
 
        
18:35 poiki-mobile 
admins and users often prefer to be on the "just use upstream image, don't rebuild" side of Docker life 
 
        
18:36 pdurbin 
Ok. I don't think there are any builders on this floor. 
 
        
18:37 pdurbin 
But I definitely appreciate builders. :) 
 
        
18:37 poiki-mobile joined #dataverse 
 
        
18:37 poiki-mobile 
Maybe pameyer? 
 
        
18:38 pdurbin 
He's only on our floor two days a week. :) 
 
        
18:39 poiki-mobile 
LOL 
 
        
18:39 poiki-mobile 
That should be sufficient... ;-) 
 
        
18:39 poiki-mobile 
Alright guys gotta go... more on Monday. Have a good night! 
 
        
18:40 pdurbin 
pameyer: being into Docker and being into Kubernetes are two very different things. 
 
        
19:08 donsizemore joined #dataverse 
 
        
19:22 pameyer 
pdurbin: yup 
 
        
19:24 pdurbin 
Like being into floating pool chair vs. being into sailing. 
 
        
19:25 pameyer 
feels more like "do you need a canoe or a submersable aircraft carrier?" 
 
        
19:26 pdurbin 
either way you could drown 
 
        
19:29 pdurbin 
and go places you can't on foot 
 
        
19:31 Sherry joined #dataverse 
 
        
19:32 Sherry 
Hey, looks like dataverse.harvard is down? 
 
        
19:32 Sherry 
FYI UVa dataverse has gone down three times in three days. Something up with attacks on dataverse sites? 
 
        
19:32 pdurbin 
bloops! really? 
 
        
19:32 pameyer 
harvard dataverse looks up to me 
 
        
19:33 pdurbin 
phew 
 
        
19:33 pameyer 
at least, I get the expected response to https://dataverse.harvard.edu/api/info/version  
 
        
19:33 Sherry 
Ok, it is up now, but not 5 min. ago. 
 
        
19:33 pameyer 
:( 
 
        
19:33 pameyer 
@Sherry - could you remind us which version UVa is on? 
 
        
19:34 Sherry 
I have a screen shot, but I guess can't send via chat? 
 
        
19:34 Sherry 
UVa is 4.8.6 
 
        
19:34 pdurbin 
Sherry: you could upload it to https://imgur.com  
 
        
19:34 Sherry 
@pdurbin show off ;) 
 
        
19:34 pdurbin 
Sherry: if we're all under attack, what's our strategy? Who is our general? 
 
        
19:35 Sherry 
Not me!!! 
 
        
19:35 * pdurbin 
takes a step back 
 
        
19:35 pdurbin 
Three outages in as many days doesn't sound good. Any pattern? 
 
        
19:38 pameyer 
I'd want to rule out gf instability before assuming there's an ongoing pattern of attacks 
 
        
19:38 sherry joined #dataverse 
 
        
19:38 sherry 
Must have exited my window. 
 
        
19:38 pdurbin 
pameyer: we might be doing this to ourselves :) 
 
        
19:38 sherry 
Here's a couple of screen shots from around 2:30: https://imgur.com/a/B91Vjze  
 
        
19:39 pdurbin 
sherry: what do you see at https://dataverse.harvard.edu/host.txt  ? 
 
        
19:39 sherry 
dvn-cloud-app-2.lib.harvard.edu 
 
        
19:39 pdurbin 
ok, I see dvn-cloud-app-1 
 
        
19:39 pdurbin 
maybe one of the two glassfish servers was down 
 
        
19:39 pameyer 
sherry: if you missed it when you were disconnecting, my first suspicion would be glassfish stability 
 
        
19:39 sherry 
dataverse harvard is up now. 
 
        
19:39 sherry 
broken glass? 
 
        
19:40 sherry 
Friday afternoon punchy-ness 
 
        
19:40 pameyer 
:) application server 
 
        
19:40 pdurbin 
heh, rimshot 
 
        
19:40 pameyer 
but it does feel more glass-like than I'd like sometimes 
 
        
19:41 pdurbin 
we tried ironfish but it sank to the bottom of the sea 
 
        
19:41 sherry 
Oh, my! 
 
        
19:45 pameyer 
sherry: do you know if the system administratorss at UVa are monitoring memory usage, or file descriptor counts? 
 
        
19:47 sherry 
Probably not. What should they be looking for? Will restarting the server help? I think memory usage problem was fixed in one of the 4.9.X releases. We are upgrading next month. 
 
        
19:49 pameyer 
restarting the application server may help 
 
        
19:50 pameyer 
there were some stability related improvements between 4.8.6 and 4.9.something 
 
        
19:50 pameyer 
but if those fixes were the only stability issues, my suspicion is that app servers at dataverse.harvard.edu wouldn't be freezing every so often 
 
        
19:51 pameyer 
for what to look for - possibly memory or swap usage over time, and output of `lsof | wc -l` 
 
        
19:51 pameyer 
pdurbin or donsizemore may have some additional suggestions 
 
        
19:53 pdurbin 
I put my monitoring suggestions in http://guides.dataverse.org/en/4.9.4/admin/monitoring.html  
 
        
19:54 donsizemore joined #dataverse 
 
        
19:54 pdurbin 
sherry: this is my home server but your sysadmins should be able to retrieve data like this on the health of your server over time: http://munin.greptilian.com/greptilian.com/server3.greptilian.com/index.html  
 
        
19:56 sherry 
Thanks for all the tips. I'll take a look and give to my sysadmin. Now on to put out a Grad Student Fire in submitting to one of our other repositories! 
 
        
19:56 pameyer 
good luck with both 
 
        
19:57 pdurbin 
phew, the heat is off us, for now :) 
 
        
20:00 pameyer 
at some point, the glassfish falling over problem probably will need to get sorted 
 
        
20:01 pdurbin 
yeah, it would be nice if we didn't need a restart script 
 
        
20:06 pameyer 
did the restart script ever make it into the guides? 
 
        
20:06 pdurbin 
nope 
 
        
20:08 pdurbin 
I have a copy of it in my email somewhere, I think. 
 
        
20:09 pameyer 
if enough folks start seeing things like this, it might be worth adding it. 
 
        
20:09 pdurbin 
looks like it kicked off around 2:33 which lines up pretty well with sherry saying that she saw an outage at 2:30 
 
        
20:09 pameyer 
making it unnecessary would probably be better, but would also be more effort 
 
        
20:10 pameyer 
yeah - 5m interval on HU DV always has me thinking about the bounce script 
 
        
20:10 pameyer 
even though I have no idea if it's actually using a 5m interval 
 
        
20:13 pdurbin 
well, the interval comes from nagios, I beleive. Or at least it did in the version of the script that I have from 2015. 
 
        
20:16 pdurbin 
if [ $2 == "HARD" ]; then 
 
        
20:30 donsizemore 
@pdurbin does the restart script simply call a stop-domain / start-domain? 
 
        
20:30 donsizemore 
(if it's file descriptors for example, the stop-domain command should fail) 
 
        
20:31 donsizemore 
ultimately, i'm asking about stderr from the script 
 
        
20:33 pdurbin 
donsizemore: well, it uses the init script. calls a "stop" and does a "sleep 20" followed by a "start". 
 
        
20:34 donsizemore 
so likely not a process limit then. coolies 
 
        
20:34 pdurbin 
Oh, I should add that if there's no life after that "sleep 20" it goes into "kill -9" mode like any good script. :) 
 
        
20:41 pameyer 
I've gotten into the habit of using jps to fingure out if glassfish has actually stopped when told to 
 
        
20:46 pdurbin 
that fish won't behave 
 
        
20:48 pameyer 
see - this is the real reason for glassfish5 vs payara5. I don't know what kind of animal a payara is :) 
 
        
20:51 pdurbin 
heh 
 
        
20:52 pdurbin 
donsizemore: heads up that "Import a Dataset into a Dataverse with a DDI file" just got merged: https://github.com/IQSS/dataverse/pull/5170  
 
        
21:08 donsizemore 
i saw that =) i'm background-writing some python to translate IPUMS' particular structure 
 
        
21:10 pdurbin 
nice 
 
        
21:29 pameyer 
pdurbin: more messages crossing in the interwebs ;) https://github.com/IQSS/dataverse/issues/4714  
 
        
21:31 pdurbin 
pameyer: Craig just gave you a "hooray" emoji 
 
        
21:32 pdurbin 
Seems like a good way to start the weekend. Take care, all. :) 
 
        
21:34 pdurbin left #dataverse