Time
S
Nick
Message
06:35 Virgile joined #dataverse
08:50 poikilotherm joined #dataverse
14:01 juancorr joined #dataverse
14:27 pdurbin joined #dataverse
18:03 pdurbin
poikilotherm: check out dataverse-react: https://github.com/IQSS/dataverse/issues/7827#issuecomment-828664404
18:06 poikilotherm
Nice
18:14 nightowl313 joined #dataverse
18:15 nightowl313
hi all ... how are you all doing? still trying to get a store set up using dell isilon s3 ... the endpoint url is using port 9021 ... does anyone know if there is a way to specify a port in the jvm setup?
18:16 pdurbin
Specify the port that S3 is running on?
18:17 nightowl313
the endpoint to access the s3 on isilon ... they have it configured to use a specific port
18:18 nightowl313
i'm assuming that would be what I would use in the config for dataverse.files.<id>.custom-endpoint-url to point to the store
18:19 pdurbin
Ah. Yes. I would think so.
18:19 nightowl313
i did put a question out to the community on this, but no response ... don't think anyone else has tried this particular storage option
18:20 pdurbin
Yes, but I hope we can add Isilon to "Reported Working S3-Compatible Storage" :)
18:20 nightowl313
blazin' a trail!
18:20 pdurbin
that's right
18:21 nightowl313
the settings are a little different, they have an access ID and access key, which I hope still maps to the aws access key/secret key .. and "access zone" to aws region
18:22 nightowl313
but so far hasn't worked ... maybe because of the port issue
18:22 nightowl313
or some other thing i have configured wrong!
18:22 pdurbin
Sounds like the port would definitely be an issue.
18:23 pdurbin
Hopefully the only issue. :)
18:24 nightowl313
i will ask if they can make it 443
18:25 pdurbin
Oh, you tried the port in the custom-endpoint-url already?
18:25 nightowl313
yea, it doesn't allow it ... just gives an error message when I try to run the api to add it
18:26 pdurbin
What's the error?
18:27 nightowl313
remote failure: JVM option 9021 is invalid because it does not start with a '-'
18:30 pdurbin
Weird.
18:31 pdurbin
You're putting the full URL in the JVM option? Something like https://foo.bar.com:9021 ?
18:32 nightowl313
it's fine if I don't include the port (so, using "/asadmin create-jvm-options "-Ddataverse.files.iso.custom-endpoint-url=<custome endpoint>:9021"
18:32 nightowl313
oh i didn't include the https:// .. will try
18:33 nightowl313
no didn't work still
18:33 nightowl313
could try just adding it to the xml file
18:33 pdurbin
Not a bad idea. Or at least look at it in the xml file to make sure it looks ok.
18:34 pdurbin
There's a surprising amount of `substring` and `replace` going on with that JVM value.
18:37 pdurbin
poikilotherm: you use a non-AWS S3 set up, right?
18:38 poikilotherm
pdurbin: yes. Currently using Minio, soonish using Ceph S3 RADOS Gateway
18:38 pdurbin
And you use this custom url thing?
18:40 poikilotherm
Yes of course
18:40 poikilotherm
It's a matter of escaping
18:40 poikilotherm
Targeting "http://minio:9000 " here
18:40 poikilotherm
Kinda similar ;-)
18:41 poikilotherm
Remember to escape the colons for JVM props
18:41 poikilotherm
Maybe @donsizemore can add this to the Playbook and let ansible do that for y'all like I do in dataverse k8s
18:42 nightowl313
ah if I would only look at the documentation! I see that there are ports for minio listed in there ... i hadn't looked at that
18:43 poikilotherm
sed -e 's/:/\:/g'
18:43 poikilotherm
GRML
18:43 poikilotherm
That got escaped in escaping
18:43 poikilotherm
Lemme try with MD style: `sed -e 's/:/\\\:/g'`
18:44 poikilotherm
There ya go %)
18:44 nightowl313
lol or edit the xml file and add the port!
18:45 nightowl313
with minio, is there an actual access key and secret key, and region? for this there is just a secret key, and the region, I assume is the "zone"
18:45 poikilotherm
That should work, too
18:45 poikilotherm
(Without escaping)
18:46 poikilotherm
Minio does not bother about zones/regions
18:46 nightowl313
oh so did you just leave that out of the config for aws?
18:46 poikilotherm
So it's basically just access key and secret key
18:46 nightowl313
this doesn't have an access key, but an "access ID" which is basically a userid
18:46 nightowl313
i'll try that
18:47 poikilotherm
No need to set a zone
18:47 poikilotherm
(for minio)
18:48 poikilotherm
If that DELL monster wants it, you can provide a custom region with dataverse.files.<id>.custom-endpoint-region
18:49 poikilotherm
It will default to dataverse, which is simply ignored by Minio
18:49 nightowl313
oh i was thinking it was needed in the .aws config file ...
18:50 poikilotherm
IIRC that one is ignored when using the custom endpoint URL
18:50 poikilotherm
Lemme check S3AccessIO class
18:53 poikilotherm
OK the region is set when using the custom endpoint url
18:53 poikilotherm
And it will default to "dataverse"
18:54 poikilotherm
So I guess the setting in ~/.aws/config will be ignored, as there is more specific config available
18:55 poikilotherm
Man I tend to forget how f***d up that code is. 1200 lines and pretty complex. Some day we will do refactoring of the storage subsystem.
18:55 nightowl313
okay, they indicate an "access zone", so maybe that is region
18:56 nightowl313
lol
18:56 nightowl313
i have everything else set, and it shows in dataverse as a store, but if i select it the files won't upload ... have confirmed I can telnet to it
18:57 poikilotherm
Please try with a real client
18:57 poikilotherm
Either s3cmd or mc. I prefer mc, as the Minio Client is much easier to use with custom S3 implementations. https://docs.min.io/docs/minio-client-quickstart-guide.html
18:58 nightowl313
oh man there is a "path to bucket" as well
19:00 poikilotherm
Buckets with style use either fresh DNS subdomain lipgloss or oldschool path based access bling bling rings
19:00 poikilotherm
Minio is of "path based flavor", too
19:01 poikilotherm
Actually, most S3 compatible are. No wonder Amazon wants to drop support for that :-D
19:01 poikilotherm
(Some support both, like Ceph et al)
19:01 nightowl313
the parameters we were given are: https port for s3, "access zone", local account, bucket name, path to bucket, dns
19:02 poikilotherm
Can you scramble the secrets and post the full stuff here?
19:03 poikilotherm
That might reveal some of the hidden goodies ;-)
19:05 nightowl313
https port 9021, access zone "awords3", local account "abcd", bucket "test-test
19:05 poikilotherm
Sorry, but that does not make much sense...
19:05 nightowl313
path bucket /tst/test/tests3, dns "s3.test.test.coM
19:05 poikilotherm
ah
19:05 nightowl313
yea, sorry, trying to post without posting the actual stuff! lol
19:06 poikilotherm
Uhm that might hide some necessary stuff to help ya
19:06 nightowl313
access ID = "3_test_test", access key = a normal looking aws access key
19:06 poikilotherm
Like I can't tell if the dns part is including some of the path/url or not
19:07 poikilotherm
Dell Isilon/OneFS docs say "OneFS support two types of requests when resolving buckets and objects. "
19:08 poikilotherm
So I have no idea what flavor your admins are using
19:08 nightowl313
would you mind if i email you?
19:08 poikilotherm
Sure, go ahead
19:08 nightowl313
thank you! =)
19:09 poikilotherm
You can also reach me on GDCC Community Slack
19:19 poikilotherm
BTW using https://privatebin.info/ for such info is also a nice option.
19:19 poikilotherm
Can be used on https://privatebin.net/
19:22 poikilotherm
(There's a burn-after-reading function :-) )
19:22 poikilotherm
nightowl313: email received
19:22 poikilotherm
OK looks like you people are using the path based access flavor
21:08 pdurbin left #dataverse
21:44 nightowl313
thanks to @poikilotherm for his help on the above .. still working on this, but very close!
21:51 nightowl313
another question for the day ... anyone know why I would not be able to preview text files on a specific dataset? i can preview other file types, such as .png and tab files
21:51 nightowl313
and i can preview .txt files on other datasets
21:52 nightowl313
but if I try to preview .txt files I get "Unable to retrieve file.Please try again. The most common issue is that your login has timed out. If the problem persists, please contact the support team of this data repository. Please include any status code included at the end of this message: error"
21:52 nightowl313
i tried uploading a very simple new txt file and same thing
21:53 nightowl313
the dataset is not published yet, and it is the only dataset so far that we have used the direct upload (this user uploaded 17TB of 155G files!)
21:53 nightowl313
some 365G!
21:57 nightowl313
there is a guestbook on the dataset, but I am logged in as superadmin and preview other files
23:22 nightowl313 joined #dataverse
23:22 nightowl313
i actually think the problem is bigger .. i have a ticket in from before related to issues we were having with v5.3 and will follow up on that ... thanks
