Time
S
Nick
Message
01:24 yoh joined #dataverse
09:14 sivoais joined #dataverse
10:28 michbars1 joined #dataverse
11:46 pdurbin joined #dataverse
12:44 michbars1 joined #dataverse
13:08 donsizemore joined #dataverse
13:35 michbars1 joined #dataverse
13:49 djbrooke joined #dataverse
14:33 dataverse-user joined #dataverse
14:34 dataverse-user
UVa did a yum upgrade of our system (where DV was installed - V4.5.1). Now dataverse will not deploy. Any advice or logs to check to see why?
14:37 pdurbin
dataverse-user: anything in server.log? That's the main Glassfish log.
14:38 dataverse-user
let me check
14:39 pdurbin
dataverse-user: please feel free to email the entire server.log to support
14:39 dataverse-user
appears it is not responding on 8009
14:40 dataverse-user12 joined #dataverse
14:40 dataverse-user12
SHerry again.
14:40 dataverse-user12
glassfish is not listening - connecting on 8009.
14:44 pdurbin
Hi Sherry. As you may know, port 8009 is used to proxy traffic from Apache to Glassfish.
14:45 pdurbin
Please go ahead and email support
14:45 dataverse-user12
OK, zipping server log now and then sending.
14:45 pdurbin
It sounds like you have two problems. 1. The Dataverse war file will not deploy to Glassfish. 2. Nothing is listening on port 8009.
14:48 pdurbin
Do you have a preference on which problem you want to solve first?
14:51 dataverse-user12
we would like to get DV to deploy. Is glassfish expecting DV to be there to pick up the connection? Or does glassfish pick it up and then DV gets it. Chicken or egg problem?
14:52 pdurbin
Yeah, let's focus on getting the war file deployed. Are you familiar with `asadmin list-jvm-options`?
14:52 pdurbin
er, scratch that... I meant to write `asadmin list-applications`
14:53 dataverse-user12
yes, executing now......
14:55 dataverse-user12
looking up password 1st...
14:56 dataverse-user12
results of "asadmin list-applications"===== dataverse-4.5.1 <web>
14:57 pdurbin
Oh! So the war file has been deployed.
14:57 dataverse-user12
Ok, cool, now what?? why are connectors talking glassfish/apache?
14:58 dataverse-user12
that was NOT talking
14:58 pdurbin
If you do `curl localhost:8080` do you see Dataverse stuff? For me it ends with "Google Analytics Snippet".
15:00 dataverse-user12
nothing comes back, just prompt
15:00 pameyer joined #dataverse
15:01 pdurbin
Hmm. Can you please stop and start Glassfish? We want Dataverse stuff to come out of Glassfish.
15:01 donsizemore
@dataverse-user12 a yum update, within OS version, or an OS upgrade?
15:01 michbars1 joined #dataverse
15:01 dataverse-user12
Here's what is in the apache SSL error log: [Sun Mar 05 12:03:38 2017] [error] proxy: AJP: disabled connection for (localhost) [Sun Mar 05 12:03:39 2017] [error] proxy: AJP: disabled connection for (localhost) [Sun Mar 05 12:03:39 2017] [error] proxy: AJP: disabled connection for (localhost) [Sun Mar 05 12:03:42 2017] [error] proxy: AJP: disabled connection for (localhost) [Sun Mar 05 12:03:45 2017] [error] proxy: AJP: disabled co
15:02 donsizemore
@dataverse-user12 on the machine can you issue: "netstat -na |grep LIST" and see which ports are listening?
15:02 dataverse-user12
a yum update within OS version, part of quarterly patching
15:03 donsizemore
@dateverse-user12 RHEL6, RHEL7 or Debian/Ubuntu? i've been patching ours (a mix of RHEL6/7 and CentOS6/7 all along)
15:03 donsizemore
@dataverse-user12 scratch debian/ubuntu, i'm awake =)
15:04 dataverse-user12
We are CentOS6.8
15:04 pameyer
AJP errors make me thing that something in the apache configs was changed during the update
15:04 dataverse-user12
Sending via email the "netstat" results.
15:05 dataverse-user12
Puppet manages our apache config files (has been checked).
15:05 donsizemore
@dataverse-user12 my centos 6.8 box picked up a java patch on jan 12th. have you stopped glassfish, removed the generated/ and osgi-felix/ directories, then started glassfish?
15:06 pdurbin
donsizemore: I think you're on the right track to make sure Glassfish is actually serving the Dataverse application. From a quick curl test above, it seems like it isn't.
15:06 donsizemore
@dataverse-user12 /usr/local/glassfish4/glassfish/domains/domain1/generated and /usr/local/glassfish4/glassfish/domains/domain1/osgi-cache specifically
15:06 dataverse-user12
Yes, we removed the generated directory, but not osgi-felix
15:08 dataverse-user12
"generated" cleaned, but not "osgi-cach" - should we try? up for anything.
15:08 donsizemore joined #dataverse
15:08 pdurbin
dataverse-user12: you are serving Dataverse on port 8080, right? For "admin" API calls or whatever?
15:09 pameyer
removing ogi-cache too (and restarting glassfish) is definately worth a try
15:09 donsizemore
@dataverse-user12 you can nuke them both while glassfish is stopped, it'll rule out old kruft lying around. i've run into that after java patches
15:10 donsizemore
@dataverse-user12 though my osgi-cache/ only contains felix/ so it might be surgical to remove only felix/
15:10 dataverse-user12
remove directory or contents?
15:11 donsizemore
@dataverse-user12 the felix directory itself. glassfish will recreate it
15:11 dataverse-user12
Ok stopping glassfish and removing felix.
15:12 dataverse-user12
also deleted the generated directory.
15:12 dataverse-user12
Restart glassfish?
15:12 donsizemore
@dataverse-user12 yes
15:13 donsizemore
@dataverse-user12 note that sometimes once apache is unforgiving once it thinks proxy target isn't listening, so you may have to subsequently restart apache as well
15:14 dataverse-user12
We noticed that and put retry=0 in proxy pass directive to tell apache to keep trying
15:17 pdurbin
Before stopping Glassfish, I would suggest running `asadmin undeploy dataverse-4.5.1`. Then stop Glassfish. Then blow away the "generated" and "osgi-cache" directories. Then start Glassfish. Then deploy the war file. Then check port 8080 to see if Glassfish is serving Dataverse.
15:21 dataverse-user12
Is this a good war file to use (not just for upgrade?) https://github.com/IQSS/dataverse/releases/download/v4.5.1/dataverse-4.5.1.war
15:22 pdurbin
Yes. That's the one from https://github.com/IQSS/dataverse/releases/tag/v4.5.1 .
15:24 donsizemore
@pdurbin what's your take on those 500 array index errors? script kiddies? we were going to move to 4.5.1 tomorrow, then 4.6 on wednesday
15:25 dataverse-user12
OK, undeploying...stopping glassfish... blowing away directories.... restarting fish
15:25 pdurbin
donsizemore: oh, you're talking about https://github.com/IQSS/dataverse/issues/3665 . Dunno! Like I commented there, I'm curious which line number is throwing that exception.
15:26 pdurbin
dataverse-user12: good. It seems cleaner to get ride of any trace of the old Dataverse war file via that "undeploy" step.
15:26 pdurbin
rid*
15:28 dataverse-user12
deploying war file....
15:28 * pdurbin
crosses fingers
15:32 dataverse-user12
deploy failed
15:32 donsizemore
@dataverse-user12 are there SEVEREs in server.log?
15:32 pdurbin
Any details about the deployment failure in server.log?
15:33 dataverse-user12
sending log via email - support@
15:33 pameyer
can we exclude database schema changes? aka - 4.5.1 was the working dataverse version before upgrades?
15:33 dataverse-user12
Yes, 4.5.1 was the working dv version before upgrade.
15:33 pameyer
thanks - one less thing to worry about
15:34 dataverse-user12
just sent output from deploy.... checking server.log now
15:36 dataverse-user12
EJB timer service is SEVERE - log on its way to support@ from Tim Stevens
15:38 donsizemore
@dataverse-user12 i'm dls
15:42 michbars1 joined #dataverse
15:42 pdurbin
dataverse-user12: got the log in https://help.hmdc.harvard.edu/Ticket/Display.html?id=247303 . Thanks. It says "java.lang.RuntimeException: EJB Timer Service is not available".
15:43 pdurbin
This reminds me of the issue donsizemore opened: https://github.com/IQSS/dataverse/issues/3336
15:44 pdurbin
dataverse-user12: as I commented in that issue, for me the fix was to stop Glassfish and move the "generated" directory aside.
15:49 dataverse-user12
OK, read #3336, now stopping Glassfish and removing generated and felix (as @donsizemore suggested)
15:52 djbrooke joined #dataverse
15:55 dataverse-user12
now DV is not deployed,
15:56 dataverse-user12
Any idea of apache settings that could be a problem?
15:56 donsizemore
@dataverse-user12 now that the memories trickle back in, the osgi module problem would've prevented glassfish from starting. but dumping the cache is never a bad idea
15:57 donsizemore
@dataverse-user12 quarterly yum patches... when was the last update? there were two java-1.8 patches for centos6 in january
15:58 donsizemore
@dataverse-user12 and selinux remains permissive/disabled?
15:59 pameyer
re: apache - either allowed hosts, or the ajp module
16:00 dataverse-user12
selinux disabled
16:01 dataverse-user12
ajp localhost:8009
16:02 pdurbin
dataverse-user12: worst case, if you really can't get the Dataverse war file to deploy to your installation of Glassfish... I guess you could reinstall Glassfish. A fresh copy of Glassfish. But then you'd have to patch it, configure it, etc.
16:04 dataverse-user12
we are taking a break.... re-assess. If any other ideas, we are open (before we go re-installing Glassfish).
16:14 pdurbin
dataverse-user12: Can you please try this? select * from "EJB__TIMER__TBL";
16:16 dataverse-user12
ok...
16:18 djbrooke joined #dataverse
16:18 pdurbin
On my little "dev1" server I get "0 rows". I'm curious what you see on yours.
16:18 dataverse-user12
We have 2 rows.
16:19 pdurbin
Ah ha. I'm being told that you should remove those rows.
16:19 pdurbin
We've only ever seen this problem on production servers. Not on developers' machines.
16:20 dataverse-user12
Checking our test server now (which is/was at 4.6 during the weekend update and did not problem coming back) that server also has 2 rows
16:22 pdurbin
Ok. Well, we're hoping the fix is to remove all the rows in that "EJB__TIMER__TBL" and then do the whole undeploy, stop glassfish, remove "generated" song and dance again.
16:28 dataverse-user12
Ok, we will try this song ;-) We will save the dancing until we see what happens.
16:29 pameyer joined #dataverse
16:30 pdurbin
deal
16:32 dataverse-user12
How do we deploy the war file without the database upgrade scripts? Our DB is already at 4.5.1.
16:32 pameyer
asadmin deploy /path/to/dataverse.war
16:38 pdurbin
what pameyer said
16:39 dataverse-user12
Still no go:
16:39 dataverse-user12
stopped glassfish removed “generated” “osgi-cache” Restarted glassfish exploded war file - tried to reply from the exploded directory received the same error that we got when deploy the war file. getting errors for database updates.
16:39 dataverse-user12
Sending snipets (via email) of reply standard error - EJB__TIMER service is OK!!! now!!
16:42 pameyer
dataverse-user12: does `asadmin list-applications` show dataverse as deployed?
16:43 pameyer
I'm used to seeing numerous database warnings/logs , even for successful deployment
16:43 pdurbin
dataverse-user12: your production site sure looks better. Can we dance yet? :)
16:44 pameyer
if I'm remembering correctly, there are cases where "failure to create a table/index/relation that already exists" gets reported as an error
16:45 dataverse-user12
no dataverse deployed
16:46 pdurbin
dataverse-user12: huh? I just went to your production URL and your site seems to be back up.
16:50 dataverse-user12
Strange.... we assumed that the "Command Deploy Failed" meant it didn't work....
16:50 dataverse-user12
Now we have dataverse-4.5.1 <ejb, web> running
16:50 dataverse-user12
Yes, now back up!!!
16:51 pdurbin
phew :)
16:52 pdurbin
dataverse-user12: that "Command deploy failed" bug was reported by pameyer in https://github.com/IQSS/dataverse/issues/3515 and fixed in Dataverse 4.6, by the way.
16:52 pdurbin
I agree that it's *very* confusing.
16:54 dataverse-user12
We are dancing!!!
16:55 pdurbin
heh
16:55 dataverse-user12
Thank you Phil, Donnie, and all other IQSS helpers!!!
16:56 pdurbin
dataverse-user12: You're quite welcome. Thank to to Leonid for telling me about the EJB__TIMER__TBL table. After you're done dancing can you please open an issue at https://github.com/IQSS/dataverse/issues to document the bit about the EJB__TIMER__TBL table?
18:01 dataverse-user joined #dataverse
18:10 donsizemore joined #dataverse
18:20 METRANS joined #dataverse
18:23 METRANS
Hi there, I am an administrative assistant at METRANS research center. We are looking for data repositories to deposit research data generated by our PIs. We are looking into dataverse right now to see how the institution dataverse option works. May I ask do we need to rent any server or pay money for data sets that exceeds the size limitation? Thank you!
18:24 donsizemore joined #dataverse
18:44 djbrooke joined #dataverse
18:44 pdurbin
djbrooke: did you see that? I guess it's this place: https://www.metrans.org
18:45 djbrooke
missed it, thanks - I'll follow up
18:46 pdurbin
cool
18:49 djbrooke
I dropped a note to them on their contact form
18:50 djbrooke
Crossing fingers!
18:52 pdurbin
heh, it's worth a shot
19:04 djbrooke joined #dataverse
19:07 pameyer joined #dataverse
19:21 bjonnh
hey
19:22 bjonnh
pdurbin: do you have a presentation comparing dataverse to other solutions? I'd like to present that to our funders and explain why we think dataverse should be preferred
19:23 pdurbin
bjonnh: buh. Dunno! :)
19:23 pdurbin
djbrooke: do you know if we have this?
19:24 pdurbin
reminds me of http://whygitisbetterthanx.com ;)
19:25 pdurbin
bjonnh: oh, someone... TDL?... may have published a comparison. Memory fading... fading...
19:26 djbrooke joined #dataverse
19:28 djbrooke joined #dataverse
19:31 pdurbin
bjonnh: you should write something up. :)
19:31 donsizemore joined #dataverse
19:33 djbrooke joined #dataverse
19:52 bjonnh
pdurbin: well I'll have to then
19:53 pdurbin
now you're talkin' :)
19:54 bjonnh
especially compared to dryad
19:56 bjonnh
I'd say first factor is that harvard instance is free and not space limited as much as dryad is
19:56 pameyer joined #dataverse
19:57 pdurbin
bjonnh: oh, well of course you have to keep in mind the *software* vs any particular *installation*
19:57 bjonnh
sure
19:58 bjonnh
I have to admit I have really little experience with the others
19:58 bjonnh
I picked dataverse for the ease and the community :p
20:02 pdurbin
bjonnh: nice. :)
20:02 pdurbin
bjonnh: I'm hearing that Emory may have done a comparison.
20:03 pdurbin
donsizemore: does that ring a bell?
20:05 pdurbin
bjonnh: I'm also hearing there may have been a presentation at the last Dataverse Community Meeting.
20:06 djbrooke joined #dataverse
20:11 donsizemore
@pdurbin i haven't seen one but i can ask thu-mai
20:30 djbrooke joined #dataverse
20:35 pdurbin
bjonnh: see slides 7 and 8 of http://projects.iq.harvard.edu/files/dcm2016/files/steansdataverse_comm_meet2016.pptx such as "Why Dataverse?" in that "A Consortial Model for Research Data Services Using Dataverse" talk.
20:35 pdurbin
That's from "Ryan Steans, Assistant Director, Texas Digital Library (Presentation Slides)" at http://projects.iq.harvard.edu/dcm2016/meeting-agenda
21:14 pdurbin
bjonnh: still there?
21:24 pdurbin
Oh good, Sherry make an issue to document the EJB__TIMER__TBL fix: https://github.com/IQSS/dataverse/issues/3669
21:48 pameyer joined #dataverse
21:52 axfelix joined #dataverse
21:56 axfelix joined #dataverse
22:10 pameyer joined #dataverse
22:22 pameyer joined #dataverse
22:53 axfelix joined #dataverse
22:54 bjonnh
pdurbin: yes I'm here
22:54 bjonnh
thanks!
22:56 bjonnh
I'll need probably more than that
22:56 bjonnh
I'll dig and come back to you with what I get
23:08 donsizemore joined #dataverse
23:44 bshambaugh joined #dataverse
23:51 djbrooke joined #dataverse
23:56 pdurbin
bjonnh: awesome. thanks
