Time
S
Nick
Message
03:09 axfelix joined #dataverse
06:01 dataverse-user joined #dataverse
06:25 dataverse-user
good morning
06:26 dataverse-user
coming from https://github.com/nds-org/ndslabs-dataverse/issues/8 by pdurbin
06:27 dataverse-user
regarding handle registration @ dataverse results in error after using the jvm options provided with database settings through curl
06:27 dataverse-user
Error – This dataset may not be published because the Handle Service is currently inaccessible. Please try again. Does the issue continue to persist? If you believe this is an error, please contact Root Support for assistance.
07:02 dataverse-user
i found in the logs :
07:02 dataverse-user
[2017-07-11T07:02:23.723+0000] [glassfish 4.1] [SEVERE] [] [edu.harvard.iq.dataverse.HandlenetServiceBean] [tid: _ThreadID=28 _ThreadName=http-listener-1(2)] [timeMillis: 1499756543723] [levelValue: 1000] [[ Can't load private key in null: java.lang.NullPointerException]]
10:26 pdurbin
dataverse-user: hi!
10:27 pdurbin
That error seems to be coming from here: https://github.com/IQSS/dataverse/blob/v4.7/src/main/java/edu/harvard/iq/dataverse/HandlenetServiceBean.java#L303
11:21 donsizemore joined #dataverse
11:52 andrewSC joined #dataverse
12:13 dataverse-user
oh
12:13 dataverse-user
so i need to include the private phrase as well with the private key ? because i don't think we encrypted one
12:14 pdurbin
well, I'm wondering if your JVM options are configured correctly
12:16 pdurbin
See http://guides.dataverse.org/en/4.7/installation/config.html#jvm-options
12:18 pdurbin
dataverse-user: can you please email the output of `asadmin list-jvm-options` to support
12:52 dataverse-user joined #dataverse
13:12 pdurbin
dataverse-user: the output should look something like this: https://paste.fedoraproject.org/paste/M80dUPUeVa0apzWScl4eaQ/raw
13:12 pdurbin
Please notice how it has Dataverse-specific lines such as this one: -Ddataverse.fqdn=phoenix.dataverse.org
13:13 dataverse-user
hmm
13:13 dataverse-user
what about the handles config like this one : -Ddataverse.handlenet.admcredfile=/pathtokey.bin
13:14 dataverse-user
or do i need another options for this?
13:15 pdurbin
dataverse-user: I don't know very much about how to configure Handles (I use DOI instead) but if you email the output to support
13:17 dataverse-user
Thank you
13:17 dataverse-user
done
13:18 pdurbin
dataverse-user: ah, at https://help.hmdc.harvard.edu/Ticket/Display.html?id=251167 . Perfect. Thank you!
13:19 pdurbin
dataverse-user: Is there a file at /siteinfo.bin ?
13:19 pdurbin
Right in the root filesystem?
13:21 donsizemore joined #dataverse
13:22 dataverse-user
pdurbin : yeah
13:25 pdurbin
hmm, ok. well, someone will reply. sorry you're having trouble
13:26 dataverse-user
no problem , thank you for your help !
13:27 pdurbin
Oh sure. So you use Docker in production?
13:28 pdurbin
That's what I was reading at https://github.com/IQSS/dataverse/issues/3938#issuecomment-313366127
13:28 dataverse-user
Yes
13:28 pdurbin
Do you like it? Any pain points? I'm pretty new to Docker.
13:29 dataverse-user
No pains :D
13:29 pdurbin
heh, good!
13:29 pdurbin
bjonnh has been trying to help me learn Docker
13:29 dataverse-user
we're not that good at java configs or java in general so we use the image to do the hardwork
13:30 pdurbin
That makes sense.
13:30 dataverse-user
and just use the REST api to generate data sets
13:31 dataverse-user
but i think updating the docker image will be hard if there is no official support since we have to rely on someone to update it
13:31 dataverse-user
anyway i have to go , thanks again for the help much appreciated! good bye
14:02 donsizemore joined #dataverse
14:03 pdurbin
donsizemore: thanks for https://github.com/IQSS/dataverse/pull/3991 !! I moved it to Code Review for you.
14:04 donsizemore
@pdurbin it may not be purty, but the installation works on Ubuntu and in Vagrant, and I've tested the init script on CentOS6 and the systemd script on RHEL7
14:07 pdurbin
Nice. systemd is still new and fancy for me. :/
14:12 donsizemore
@pdurbin it ain't no thang. i'm more concerned with whether my soy sauce is low sodium =)
14:12 pdurbin
heh
14:13 pdurbin
donsizemore: are you aware of how many Dataverse installations are on AWS? I'm not but I hear it's a thing.
14:15 donsizemore
@pdurbin i'm not but we've priced out putting ours there. once our VMware cluster comes up for hardware refresh, AWS may look pretty attractive
14:15 donsizemore
@pdurbin i can share my silly little map of how Odum might implement it, if you're interested?
14:16 pdurbin
TDL is on AWS according to some notes I took at https://groups.google.com/d/msg/dataverse-community/6Twh8KLhmQg/rR9TP-cPAQAJ
14:16 pdurbin
donsizemore: map? Like a network diagram? Sure, please share it up.
14:17 donsizemore
@pdurbin oh, i meant pricing particular VMs by service level
14:20 pdurbin
oh, sure, I'll take that too
14:22 donsizemore
@pdurbin sorry, i thought we could share with external folks. i blame Bill Gates personally.
14:22 pdurbin
heh, no worries
14:32 pameyer joined #dataverse
14:36 pameyer
systemd still seems somewhat counter-intiutive to me
14:38 pdurbin
pameyer: lemme know when you upgrade from RHEL /CentOS 6 to something newer :)
14:38 pameyer
we've got cent7 boxes in production ...
14:39 bjonnh
donsizemore: how many users?
14:40 donsizemore
@bjonnh we have 1118 in our authenticateduser table... @pdurbin would the builtinusers be copied over as they sign in after our migration to 4?
14:41 bjonnh
what is the size of the storage?
14:41 bjonnh
and are all the users located in a single geographical region?
14:41 pdurbin
donsizemore: the dashboard for users just got merged into develop yesterday. You should try it out on you 1118 users. :)
14:41 pdurbin
your*
14:42 donsizemore
@bjonnh right now our Dataverse VM has a 2TB thin-provisioned VMDK (the maximum supported by fault tolerance) and consuming <200GB
14:43 pdurbin
donsizemore: I
14:43 pameyer
huh - I'd assumed you'd had VMs talking to fileservers over NFS/IRODS
14:43 bjonnh
oh wow so your storage cost will be like nothing on AWS
14:43 pdurbin
donsizemore: I'm confused. You already migrated to Dataverse 4.
14:43 donsizemore
@bjonnh we accept users from anywhere. a select distinct on affiliation from authenticateduser is all over the place - 491 results
14:43 bjonnh
do you have an idea of how much upload you are doing?
14:44 bjonnh
(how much your server outputs to the net)
14:44 donsizemore
@pdurbin i wasn't sure of the relationship between builtinuser and authenticateduser pre- and post-migration
14:44 donsizemore
@bjonnh downloads are bursty but our average over a month is i think... hold on, i measured that, gimme a sec
14:44 bjonnh
well just a range, 1G, 1TB, 1PB…
14:45 pdurbin
Ever user has a row in the authenticateduser table. Only builtin users have a row in the builtinuser table.
14:46 * bjonnh
noticed this is not really a range
14:46 donsizemore
@pameyer our Dataverse VM irsyncs its files.dir and PG backups into iRODS. we didn't get the iRODS/WebDAV dealy to scale prior to our migration to 4
14:47 pdurbin
The builtinuser table is where we store a salted and hashed password for builtinusers. We do not store a password for non-builtin users because there password is stored by a shibboleth server or ORCID's server or GitHub's server or Google's server.
14:47 donsizemore
@pdurbin cool, then i was looking in the right place =)
14:48 donsizemore
@bjonnh we're primarily social science with historical data, so we have a bunch of small files. i'm fishing a network summary out of VMware if i can find the right screen
14:50 pdurbin
pameyer: but you're running Dataverse on el6 not el7, right?
14:50 donsizemore
@bjonnh we averaged 11kb/s (incoming and outgoing) over the past month. each VMware host has dual 10GB NICs, tho =)
14:50 pameyer
pdurbin: dv.sbgrid.org is c7; my dev VMs are currently c6
14:51 pdurbin
hmm, interesting. thanks, pameyer
14:51 donsizemore
@pameyer i'm waiting impatiently for rhel8 before it's time to rebuild our production machines
14:51 bjonnh
donsizemore: how redundant are your machines/VMs?
14:51 pdurbin
donsizemore: I got your PDF about AWS stuff. Is this issue on your radar? https://github.com/IQSS/dataverse/issues/3921
14:53 donsizemore
@pdurbin we've got a grant proposal in to extend that: s3 and irods in addition to swift. that's just a proposal at this time.
14:54 donsizemore
@bjonnh we've currently got 7 vmhosts of varying vintage over a NetApp. our Dataverse fit neatly in one VM within fault tolerance's maximum specs, so we're sitting there until we outgrow it.
14:55 bjonnh
donsizemore: well 4 machines on AWS with 2 cores 8 gigs of ram are 2200/y (with upfront reservation)
14:56 bjonnh
donsizemore: 2TB of S3 storage with 1GB in/out is arount $50/mo
14:56 bjonnh
around
14:57 bjonnh
anyway I'm not sure how dataverse is clusterisable, so you may end up with only one machine for the app itself and eventually the db on 3
14:57 bjonnh
that would be the paranoid option ;)
14:57 donsizemore
@bjonnh i chose different service levels depending on VM (GF, PG, TwoRavens) and came up with $5500/yr not including data
14:57 bjonnh
wow
14:58 donsizemore
@bjonnh but my boss told me to spec it the way *I* would want it done, without regard to price
14:58 bjonnh
also
14:58 bjonnh
remember that amazon is offering a grant system
14:58 bjonnh
so you can get stuff for free
14:58 donsizemore
@bjonnh tell me more
14:58 bjonnh
hah
14:58 bjonnh
you're going to cut me the grass under the feet ;)
14:59 bjonnh
https://aws.amazon.com/grants/
14:59 bjonnh
"Build cloud-hosted publicly available science-as-a-service applications, software, or tools to facilitate their future research and the research of their community."
14:59 bjonnh
that's totally you ^
15:00 donsizemore
@bjonnh i really appreciate your mentioning this =)
15:01 bjonnh
well if you file one, I will be interested in knowing how it went
15:01 bjonnh
because I will have to do it too
15:16 donsizemore
@bjonnh jonathan_crabtree
15:26 donsizemore
@bjonnh sorry, i just re-read my cost estimate, and that $5500 would include a generous estimate of outgoing data (~$720/yr)
15:40 bjonnh
great thx
15:43 pameyer joined #dataverse
16:10 pameyer joined #dataverse
16:12 jhand joined #dataverse
17:26 pameyer joined #dataverse
17:27 donsizemore joined #dataverse
17:30 axfelix joined #dataverse
18:24 pameyer
donsizemore: I just learned your glassfish user has a home directory
19:05 axfelix joined #dataverse
19:08 pameyer joined #dataverse
19:31 donsizemore joined #dataverse
19:32 donsizemore
@pameyer should it not(?)
19:33 donsizemore
@pameyer or are you telling me i have some remote vulnerability?
19:34 pameyer
@donsizemore: not a problem, just something that I hadn't been expecting
19:35 pameyer
I'd overlooked creating $HOME for glassfish and just noticed while looking at 1934
19:35 donsizemore
@pameyer on the production box, everything is done from my home directory on the assumption that i'll be around for a while ;) i thought a glassfish home might be more central
19:35 pameyer
:)
19:36 donsizemore
@pameyer #1934 was only me taking a stab at it. improvements welcome =)
19:37 pameyer
it looks good to me so far
19:37 donsizemore
@pameyer the vagrant installer still runs as root due to vagrant mount permissions, but then it launches as $glassfish_user
19:37 pameyer
most of the glitches I've hit have been problems on my end
19:37 pameyer
ah - I hadn't though to try the vagrant installer
19:38 pameyer
but it appears to play nicely with the non-interactive / changed defaults installer
19:38 donsizemore
@pameyer the "bare metal" installation process may be run unprivileged; the vagrant machination was simpler as root (and then it's in a temporal vagrant box)
19:40 pameyer
yup - non-vagrant unprivileged install looks good :)
19:41 pameyer
and once I'd put my default.config someplace that wasn't /root ; wipe and re-provision installation could send mail (that's what I'd meant by non-interactive)
19:41 donsizemore
@pameyer i was starting to look at your mvn clean issue out of curiousity
19:42 pameyer
anything that reduces the number of steps I need to remember is something I try to do
19:42 pameyer
does it make sense?
19:42 donsizemore
absolutely
19:57 pameyer joined #dataverse
20:08 pameyer joined #dataverse
20:24 pameyer joined #dataverse
20:55 axfelix joined #dataverse
21:22 pameyer joined #dataverse
22:05 donsizemore joined #dataverse
