Time
S
Nick
Message
03:02 sivoais joined #dataverse
03:02 sivoais joined #dataverse
12:48 donsizemore joined #dataverse
13:44 donsizemore joined #dataverse
13:57 pameyer joined #dataverse
14:07 bikramj joined #dataverse
14:21 bikramj joined #dataverse
14:39 pameyer left #dataverse
14:53 bikramj joined #dataverse
15:29 bikramj joined #dataverse
18:09 jamie-ucla joined #dataverse
18:13 jamie-ucla
Hi, would this be a good place to ask a question about AWS/S3 setup?
18:25 bikramj joined #dataverse
18:30 pameyer joined #dataverse
18:31 pameyer
hi jamie-ucla - this might be a good place; but I don't have direct experience with s3 config
18:31 pameyer
pdurbin might be more conversant about it, since they're using s3 in production
18:33 jamie-ucla
I also posted in ghe google groups. pdurbin has posted there and been nice about my beginner questions
18:35 pameyer
I just read that post. this might be a silly question, but did you delete the `-Ddataverse.files.storage-driver-id=file"` glassfish config entry?
18:38 jamie-ucla
Checked for that and it is gone.
18:41 pameyer joined #dataverse
18:46 pameyer
if you haven't checked that the aws credentials dataverse is using can write to the bucket, that might be worth doing
18:47 pameyer
I'm vaguely puzzled why that exception trace has what looks like a space in the url (`Skipping outbound rewriting of invalid URL : http://54.67.118.35 :8080/api/datasets/export?ex`); but I'm also not sure how that would be related
18:49 jamie-ucla
Thank you - will take a look at that.
18:56 jamie-ucla
Found it. Thank you! I had a space in the jvm option. Going to restart and see what happens.
19:11 donsizemore joined #dataverse
19:13 jamie-ucla
Fixed the 54.67.118.35:8080 typo. I should update the groups posting, files still now showing, seem to be trying to write to the local storat /usr/local/dvn/temp/...
19:14 pameyer
assuming that's "files still not showing"
19:15 jamie-ucla
Yes, sorry - the files still aren't showing. Appear to upload and immediately disappear.
19:16 pameyer
I don't know about how the s3 storage drivers work - I could see it being possible that files land on local filesystem before being moved to s3. but the `dvn` part of `/usr/local/dvn/temp` puzzels me
19:20 jamie-ucla
I use the ansible script to install. There is a defaults/main.yml file that has the default settings. One is the filesdir: /usr/local/dvn/data option. That's the one I'm wondering if it should be set diffeerently for s3 buckets.
19:21 pameyer
that makes sense then :)
19:21 pameyer
does that directory exist on the system?
19:25 jamie-ucla
It's definately on the server where glassfish, dataverse, etc are but that directory structure isn't on the s3 bucket.
19:25 pameyer
ok - so that means it could write to temp files to that directory
19:26 jamie-ucla
Also I may not have set the s3 permissions correctly, this is my first time setting anything up on AWS so there is much frantic reading and googling as I do things.
19:26 pameyer
aws docs are non-straightforward
19:26 pameyer
but things are suggesting that those are the things to check
19:27 jamie-ucla
Its an interesting learning curve. And powers-that-be have decided they would like a working dataverse in 2 weeks. Never a dull moment.
19:34 pdurbin
jamie-ucla: a working dataverse in 2 weeks! Great. :)
19:34 * pdurbin
waves at bikramj
19:34 jamie-ucla
Hah! Only great if it gets done.
19:36 pdurbin
I'm tempted to ask you the same questions about "dataverse.files.storage-driver-id" that pameyer was asking you.
19:37 jamie-ucla
Deleted the original driver-id and replaced with: Ddataverse.files.storage-driver-id=s3
19:38 bikramj
hi phillip, I ended up on IRC google search result today after our R version automatically upgraded from 3.4 -> 3.5
19:39 bikramj
sorry for an extra l in name :)
19:39 pdurbin
I've been called worse. :)
19:39 pdurbin
jamie-ucla: then did you restart glassfish?
19:40 jamie-ucla
No, I did a server restart. WIll go back and try that now.
19:40 pdurbin
Thanks.
19:45 pameyer
jumie-ucla: working production in 2 weeks, or working staging in 2 weeks?
19:49 jamie-ucla
They seem to think Tim and I can make working production but he and I don't think that is realistic. Will most certainly slide a bit.
19:49 jamie-ucla
Tried glassfish restart, still no files. I'll go and update the google group.
19:50 pameyer
pdurbin: do you know if temp files get writting to posix storage when using the s3 storage driver?
19:51 donsizemore joined #dataverse
19:51 pdurbin
jamie-ucla: sorry, when you say "still no files" do you mean that after restarting glassfish, you created a new dataset and tried to add files to that new dataset?
19:52 jamie-ucla
I was adding to an existing dataset. I'll try making a new dataset and adding to that.
19:52 pdurbin
jamie-ucla: please. thank you!
19:54 pdurbin
pameyer: when you upload files via the GUI , they go a temp directory. Documenting that directory is what https://github.com/IQSS/dataverse/issues/2848 is about. However, I believe this directory is different than the "dataverse.files.directory". My understanding is that once you switch the driver from "file" to "s3" nothing should be written to the "dataverse.files.directory" directory anymore.
19:56 pameyer
let's assume your understanding is better than mine
19:57 pdurbin
if you insist
19:57 pdurbin
jamie-ucla: for fun you could change the storage driver to "swift" and restart glassfish and try again to see if you get a different error in server.log
19:58 jamie-ucla
ok, tying that now
20:00 pdurbin
jamie-ucla: the behavior you're seeing so far feels like the "file" driver was never removed but I know you said it's gone. It's weird.
20:03 jamie-ucla
I know, it does but (I'm using vi to edit) doing a search on 'files.storage-driver-id' I can only find one line that has that.
20:07 pdurbin
Is the driver line near a bunch of other dataverse lines?
20:08 pdurbin
lines like "dataverse.rserve.host" or whatever?
20:27 jamie-ucla
Those lines are at the end of the Ddataverse line. The lines were added usuaing asadmin. It puts the lines at the end of the Ddataverse jvm options.
20:29 pdurbin
jamie-ucla: ok. Can you try changing any other dataverse-related JVM setting to make sure it works?
20:30 pdurbin
Or did you do that already with the extra space thing?
20:33 jamie-ucla
I already fixed that typo
20:33 pdurbin
was it in a JVM option?
20:33 jamie-ucla
Yes, the fqdn option
20:34 pdurbin
ok, so making changes works fine. hmm
20:34 pdurbin
I know you emailed the google group already. Have you tried support
20:35 jamie-ucla
I'll give that a try next.
20:35 pameyer
have you had a chance to check the aws credentials
20:35 pameyer
?
20:35 pdurbin
Cool. Please attach your domain.xml file when you email support.
20:35 pameyer
you should already have the aws cli on the box; that should be enough to test copy/list/delete
20:36 pdurbin
Oh, and did you try changing it to "swift" to see if you get a different error in server.log?
20:37 jamie-ucla
ok, will try that
20:52 donsizemore joined #dataverse
20:52 pameyer
@donsizemore - are you folks doing ok with the weather so far?
21:04 pameyer left #dataverse
21:19 bikramj joined #dataverse
21:21 bikramj_ joined #dataverse
21:22 matthew-dunlap joined #dataverse
21:25 matthew-dunlap
I was reading the logs on the aws setup issues. I'm fairly sure we do not store the files locally before writing them to s3. jamie-ucla did you make sure to restart glassfish after changing the storage driver to s3?
21:26 matthew-dunlap
If there is an s3 credentials issue I'd be surprised if you get to the point where files are being written anywhere
21:27 dataversasdfe-us joined #dataverse
22:26 jamie-ucla left #dataverse
22:33 bikramj joined #dataverse
23:37 bikramj joined #dataverse
