Time
S
Nick
Message
06:42 juancorr joined #dataverse
07:58 xarthisius joined #dataverse
07:58 xarthisius joined #dataverse
08:20 jri joined #dataverse
08:37 jri_ joined #dataverse
10:08 andrewSC joined #dataverse
10:20 andrewSC joined #dataverse
12:01 donsizemore joined #dataverse
14:18 pdurbin joined #dataverse
14:25 donsizemore
@pdurbin morning! you haven't done much with DCT, have you?
14:41 pdurbin
donsizemore: buh. Do I know what that is?
14:42 pdurbin
a proprietary audio file format for digital dictation?
14:42 donsizemore
@pdurbin Scholars' Portal's Data Curation Tool. you have closed issues in the repo, so didn't know if you had tinkered with it
14:43 dataverse-user joined #dataverse
14:44 dataverse-user
Hello everyone,I hope you guys are doing well
14:44 dataverse-user
I have a few problems with some files format inside datasets
14:45 dataverse-user
1. I can upload .zip and .7z files, but cant download them
14:45 dataverse-user
dataverse throws the following error:{"status":"ERROR","code":403,"message":"'/api/v1/access/datafile/120' you are not authorized to access this object via this api endpoint. Please check your code for typos, or consult our API guide at http://guides.dataverse.org ."}
14:45 donsizemore
@dataverse-user is the dataset published? (and if not, are you passing your API token?)
14:46 dataverse-user
and 2. I can't download multiple files at the same time... this is the error that I get: {"status":"ERROR","code":404,"message":"'/api/v1/access/datafiles/113,114' Datafile null: no such object available"}
14:47 dataverse-user
the dataset is not published
14:47 donsizemore
then you'll need to pass your API token along with the download request
14:47 dataverse-user
I'm sorry
14:47 dataverse-user
How do i pass the api token?
14:48 donsizemore
it's the X-Dataverse-Key part of http://guides.dataverse.org/en/latest/api/dataaccess.html
14:48 donsizemore
(the docs alternating call it a key or a token, I forget which we're supposed to use these days)
14:49 dataverse-user
so I have to run the curl command in order to make this work?
14:51 pdurbin
dataverse-user: hi! I'm confused. Are you trying to use the API ?
14:51 dataverse-user
No sir
14:51 pdurbin
Heh. Ok.
14:51 dataverse-user
I'm trying to download the files from the platform
14:51 pdurbin
Is this your own installation?
14:51 dataverse-user
Yes it is
14:52 pdurbin
Are the files restricted?
14:52 dataverse-user
They don't have the lock that identifies them as restricted
14:53 pdurbin
Is your installation public? Do you mind if we look at it?
14:53 dataverse-user
Is private because of the university policy
14:54 pdurbin
Ok, no problem. You said the dataset hasn't been published yet. And you must be logged in or else you wouldn't see the files. Hmm.
14:54 dataverse-user
Correct
14:54 pdurbin
Is this Dataverse 4.20?
14:55 dataverse-user
It's v4.9.4
14:55 pdurbin
Ok, so an older version.
14:55 pdurbin
Are you logged in as a superuser?
14:55 dataverse-user
Yes
14:56 pdurbin
Would it violate university policy if you take screenshots and open a GitHub issue?
14:56 dataverse-user
No sir
14:56 dataverse-user
I can do that
14:56 pdurbin
Fantastic! Please open an issue at https://github.com/IQSS/dataverse/issues
14:56 dataverse-user
Thank you!
14:56 dataverse-user
will do
14:57 pdurbin
dataverse-user: anything else we can do for you?
14:57 dataverse-user
Not for the moment... Thanks for your time and have a great day!
14:57 pdurbin
donsizemore: yes, I'm a huge fan of the Data Curation Tool and uploaded a ton of screenshots to https://github.com/IQSS/dataverse.harvard.edu/issues/32
14:57 pdurbin
dataverse-user: great, I'll look for that issue. Thanks again.
14:58 dataverse-user
Thanks to you, bye
14:59 donsizemore
@pdurbin at the risk of being non-SLOPI, may I slack you a screenshot of what I get when I click on the "view" eyeball in DCT?
15:00 pdurbin
please slack away
15:00 donsizemore
@pdurbin I get the same behavior from dev2.dataverse.org that I'm getting on my local test
15:01 pdurbin
I got your screenshot. Let's look at dev2
15:02 donsizemore
@pdurbin we're expecting to see... something more than that.
15:04 pdurbin
some numbers
15:05 pdurbin
I'm asking Kaitlin in PKP Slack if she has time to help.
15:05 donsizemore
i've tried a number of files and file formats. squashed a few errors i was getting in the javascript console, but... that would be wonderful.
15:08 pdurbin
I guess I've used edit more than view. I've been using Data Explorer to view, to see the numbers. BRB, going on a call.
15:12 donsizemore
@poikilotherm knock knock?
15:17 poikilotherm
Hmm? On my way to construction site, but hit me. Will read up later.
15:26 donsizemore
@poikilotherm eh, I can't look away from #5274. such a huge difference in warfile size. I ran that change in pom.xml against Payara5 with Gustavo's patched PrimeFaces
15:27 donsizemore
@poikilotherm every automated test succeeded except uningest, which is a separate issue. if I'm reading your comments correctly it should be a safe switch under payara5?
15:41 kaitlin joined #dataverse
15:45 pdurbin
donsizemore: psst. Guess who's here.
15:46 kaitlin
Just taking a look at the chat history, but I think I'm missing some info about the issue
15:47 pdurbin
kaitlin: please try this link (and remind me to reset the API token): https://scholarsportal.github.io/Dataverse-Data-Curation-Tool/?dfId=83&siteUrl=https://dev2.dataverse.org&key=65dcee41-151e-447f-884f-8aa9d8cb0bb1&fileMetadataId=77
15:47 kaitlin
I see, so an issue with the view functionality
15:49 kaitlin
do you have the original file as well?
15:50 pdurbin
Yeah, the original file is here: https://dev2.dataverse.org/file.xhtml?fileId=83&version=1.0
15:50 kaitlin
curious about how it's formatter
15:50 kaitlin
ah, I see, thanks
15:51 pdurbin
donsizemore noticed this with a different file
15:52 kaitlin
I'm going to test the file on our demo site just to verify that it works on our install
15:52 pdurbin
ok, thanks
15:53 poikilotherm
donsizemore enlighten me - 5274 is about removing the bloody AWS SDK, right?
15:53 pdurbin
Yep. Stripping AWS Mail component with TrueZIP blocks packaging for docker images #5274
15:59 poikilotherm
Well, we should take a look at the AWS SDK, too. It is a pretty dated version, so it should be updated. And there is that Jackson dependency, which could be switched to provided, but we need to make sure that the shipped version is working for all things depending on it. I can't remember all deps relying on it, but there is also #6810. Then we don't have to care for our code, but still should take a look to avoid
15:59 poikilotherm
unnecessary includes that bloat the WAR.
15:59 poikilotherm
Apart from that, should be safe to do :-)
15:59 pdurbin
I'd love to make the war smaller. And reduce deployment times.
16:01 poikilotherm
Crossing fingers that will help with it...
16:05 poikilotherm
donsizemore pdurbin are you asking me to take a look at this tomorrow?
16:05 pdurbin
I'm not.
16:05 pdurbin
poikilotherm: what's in focus? Drywall?
16:06 poikilotherm
Right now? Yeah...
16:11 kaitlin
I think I'll need to investigate further into how the ingest process handles certain filetypes, e.g. specifically when frequency calculations are being done
16:11 kaitlin
I see the same issue in our instance
16:12 kaitlin
but it works for my test spss and sav files
16:14 kaitlin
I suspect currently frequencies are only generated for these file types, and not for tsv or other formats
16:14 kaitlin
these file types meaning spss and sav
16:20 kaitlin
unfortunately our lead dev is out of the office currently, so it may take some time for me to get back to you
16:20 kaitlin
If you could file a ticket in the DCT github, that would be helpful!
16:23 pdurbin
kaitlin: thanks! I'll see if I can get donsizemore or one of the folks he's working with to create it. :)
16:23 donsizemore
@poikilotherm i'll hold off on tinkering with it then
16:24 donsizemore17 joined #dataverse
16:24 donsizemore17
@pdurbin was in the process of replying to kaitlin that I uploaded an .sav file and got the same behavior. was scrounging around for similar experiences before opening an issue
16:28 jri joined #dataverse
16:28 kaitlin joined #dataverse
16:29 kaitlin
is the sav file somewhere I can download to test it on my end as well?
16:29 kaitlin
sorry, dropped off the chat for a moment there
16:30 pdurbin
If it helps, there are some sav files at https://github.com/IQSS/dataverse-sample-data
16:34 donsizemore17
@kaitlin hi =) i replied earlier but my connection died
16:35 donsizemore17
@kaitin I did try a native .sav and got the same behavior. I'm happy to open an issue but wanted to rule out ID10T(me) problems first
16:36 kaitlin
usually sav files work for me, at least the sample files I have, but it could depend on the formatting/data of the file
16:37 kaitlin
This is one of my test files: https://demodv.scholarsportal.info/file.xhtml?fileId=10159&version=1.0
16:38 donsizemore17
@kaitlin thank you! uploading to my test instance now
16:40 donsizemore17
@kaitlin the files we _want_ to use are going to be a long shot to start with... they're SAS7bdat from the 100MB-1GB range
16:41 donsizemore17
@kaitlin yup, your test file looks good
16:46 kaitlin
I don't think sas7bdat files go through the tabular ingest process
16:46 kaitlin
at least not in 4.17 where I'm testing
16:47 donsizemore17
I imported it into SAS and exported as .sav
16:47 kaitlin
ah, got it
16:47 donsizemore17
You're right, Dataverse doesn't ingest tab-delimited (I think I remember an issue on this) but it is taking CSV
16:49 donsizemore17
and if i take your w1130 file in tab-delimited, convert the tabs to csv I can't view anything in DCT
16:50 donsizemore17
@kaitlin what would be the most helpful way to present this issue? file format at time of ingest, tabs v. commas, or just a general whine? so far my troubleshooting has been limited to different file formats
16:51 kaitlin
I think specifically highlighting that tsv and xlsx files don't show anything in the "view" panel
16:51 kaitlin
(I tested with xlsx as well)
16:53 kaitlin
You mentioned a sav file where it didn't work - do you have that file available? I'm curious about that one.
16:53 donsizemore17
It's a 90MB file (I chose one of the smaller ones) and I don't think I can make it public.
16:53 donsizemore17
I can send you a "head -2" of the surrogate copy maybe
16:57 kaitlin
no problem, was just wondering if it points to a possible bug
16:58 donsizemore17
or better yet let me ask my boss for permission to e-mail you either the SAS original or the SPSS file I'm attempting to view?
16:59 kaitlin
sure, that works. I'll message you my email.
17:36 pdurbin
donsizemore17: Dataverse should ingest tab delimited. And CSV. But not semicolon delimited.
17:37 donsizemore17
4.20 didn't ingest tab-delimited (.tab or .tsv extension)
17:37 donsizemore17
I can try again
17:46 pdurbin
Huh. Weird. I swear I worked for a few releases at least. :)
17:46 pdurbin
it* worked
17:52 pdurbin
merged back in 4.9.2: https://github.com/IQSS/dataverse/pull/4854
18:00 donsizemore17
you're right, it's ingesting now (4.20)
18:04 pdurbin
\o/
18:21 pameyer joined #dataverse
18:23 pameyer
low urgency question - if I was going to do a PR for docker-aio/docker-dcm payara update, should that be against develop or a different branch?
18:32 donsizemore17
my understanding is breaking changes are allowed
18:35 pameyer
which suggests "not develop" - or at least a blinking notice "do not merge into develop yet"
18:38 pdurbin
pameyer: hi! We're kind of at a strange time right now where we have multiple branches up in the air. Yours could be the 4th or 5th. But once one of them gets merged, we want them all to get merged. To answer your question, please just make a pull request against develop, like usual. :)
18:39 pameyer
pdurbin: thanks
18:40 pameyer
probably wouldn't be that much harder to set it up to be able to do both glassfish4 and payara5 at the same time (in different containers, obviously)
18:41 pameyer
not sure how useful it'd be
18:41 pdurbin
meh, overkill, I'd say
18:41 pdurbin
Thanks so much for jumping in on this.
18:41 pameyer
don't thank me yet - haven't done it yet ;)
18:42 pdurbin
pameyer: oh, you can do draft pull requests now. It's new.
18:42 pameyer
I was going to comment on how much things have changed in a few months .... then realized it's been a year
18:44 pdurbin
It's nice to see you again!
18:45 pdurbin
donsizemore17: word on street is that you're a man in search of a bucket.
18:46 donsizemore17
preferably with ice and some vintage wild turkey
18:46 pdurbin
a wild turkey pooped on my patio
18:46 pdurbin
but I already told you that
18:47 donsizemore17
kevin caught some s3 problems on primefaces8 but the test VMs only have local storage
18:48 pdurbin
Right. I forget how much access you have on our AWS account. We should both log in and take a look.
18:49 donsizemore17
i have an access key and secret key (to spin up ec2 instances) but nothing more
18:51 pdurbin
There's a pameyer user who has access to ec2 and s3 and a donsizemore user that only has access to ec2.
18:51 pdurbin
poikilotherm is full like me
18:54 poikilotherm
I'm honored
18:54 donsizemore17
i don't want keys i don't need. denial of plausibility and such
18:54 * poikilotherm
picks up screws and driver again
18:58 pameyer
weird - I'm suprised I've got access to your aws
18:59 pdurbin
pameyer: we've been wondering why our amazon bill is so high. Please try to tone it down a bit. Thanks!
18:59 pdurbin
poikilotherm has two accounts actually. And there's a guy who hasn't worked here for a while.
19:10 pdurbin
Anyway it sounds like donsizemore17 found the buckets. Good.
19:11 donsizemore17
It's Bouquet! B-U-C-K-E-T!
19:12 pdurbin
chowdah
19:34 donsizemore17
@pdurbin knock knock?
19:37 pdurbin
donsizemore17: hit me
19:37 donsizemore17
does dataversebot have permission to comment on pull requests?
19:37 pdurbin
Thanks more of a poikilotherm question. I don't have the password.
19:38 donsizemore17
ah, i thought it was an IQSS setting
19:38 donsizemore17
or would you rather create an IQSSbot user within IQSS
19:38 pdurbin
We can't let poikilotherm have all the fun with dataversebot.
19:41 pdurbin
donsizemore17: do you need a bot right now, right now, right now, as my kids used to say?
19:41 donsizemore17
i don't but the PR job wasn't able to report back. i suspect mr. dataverse bot may not have permissions. will pester poikilotherm
19:43 pdurbin
That bot used to leave comments on pull requests all the time. Something like, "Will one of the admins verify this patch?"
19:47 donsizemore17
and then it got quiet
19:47 pdurbin
:)
19:48 pdurbin
I'm looking at https://github.com/IQSS/dataverse/issues/6803
19:50 poikilotherm
I'm not sure what permissions dataversebot has in IQSS/dataverse
19:50 poikilotherm
Someone with admin rights might check thta
19:50 poikilotherm
I'm happy to share the password with IQSS
19:51 poikilotherm
There hasn't been any progress on that fronzier so far because no one asked ;-)
19:51 poikilotherm
donsizemore17 you con just use the bot from Jenkins. The credentials are stored and ready to use.
19:52 donsizemore17
that's the one i had selected
19:54 poikilotherm
In doubt, please look at the k8s jobs. Mr bot is very active for me :-)
19:55 poikilotherm
Something in the back of my head is about there had been two credentials...
19:55 poikilotherm
But I don't remember exactly and looking it up on my mobile is not so easy ;-)
19:56 poikilotherm
And as said: pdurbin or someone else with admin permissions needs to check the permissions on the repo...
19:56 pdurbin
Which repo?
19:59 donsizemore17
there is a dataversebot dockerhub user, a dataversebot github user, and a me.
20:01 pdurbin
https://github.com/orgs/IQSS/teams?query=%40dataversebot says dataversebot is a member of one team: dataverse-sample-data-write
20:05 poikilotherm
donsizemore17 seems like pdurbin is your man for granting mr dataversebot everything you need...
20:05 pdurbin
lemme know how I can help
20:25 dataverse-user joined #dataverse
20:27 dataverse-user joined #dataverse
