IQSS logo

IRC log for #dataverse, 2020-04-16

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

All times shown according to UTC.

Time S Nick Message
06:42 juancorr joined #dataverse
07:58 xarthisius joined #dataverse
07:58 xarthisius joined #dataverse
08:20 jri joined #dataverse
08:37 jri_ joined #dataverse
10:08 andrewSC joined #dataverse
10:20 andrewSC joined #dataverse
12:01 donsizemore joined #dataverse
14:18 pdurbin joined #dataverse
14:25 donsizemore @pdurbin morning! you haven't done much with DCT, have you?
14:41 pdurbin donsizemore: buh. Do I know what that is?
14:42 pdurbin a proprietary audio file format for digital dictation?
14:42 donsizemore @pdurbin Scholars' Portal's Data Curation Tool. you have closed issues in the repo, so didn't know if you had tinkered with it
14:43 dataverse-user joined #dataverse
14:44 dataverse-user Hello everyone,I hope you guys are doing well
14:44 dataverse-user I have a few problems with some files format inside datasets
14:45 dataverse-user 1. I can upload .zip and .7z files, but cant download them
14:45 dataverse-user dataverse throws the following error:{"status":"ERROR","code":403,"me​ssage":"'/api/v1/access/datafile/120' you are not authorized to access this object via this api endpoint. Please check your code for typos, or consult our API guide at http://guides.dataverse.org."}
14:45 donsizemore @dataverse-user is the dataset published? (and if not, are you passing your API token?)
14:46 dataverse-user and 2. I can't download multiple files at the same time... this is the error that I get: {"status":"ERROR","code":404,"message​":"'/api/v1/access/datafiles/113,114' Datafile null: no such object available"}
14:47 dataverse-user the dataset is not published
14:47 donsizemore then you'll need to pass your API token along with the download request
14:47 dataverse-user I'm sorry
14:47 dataverse-user How do i pass the api token?
14:48 donsizemore it's the X-Dataverse-Key part of http://guides.dataverse.org/en/latest/api/dataaccess.html
14:48 donsizemore (the docs alternating call it a key or a token, I forget which we're supposed to use these days)
14:49 dataverse-user so I have to run the curl command in order to make this work?
14:51 pdurbin dataverse-user: hi! I'm confused. Are you trying to use the API?
14:51 dataverse-user No sir
14:51 pdurbin Heh. Ok.
14:51 dataverse-user I'm trying to download the files from the platform
14:51 pdurbin Is this your own installation?
14:51 dataverse-user Yes it is
14:52 pdurbin Are the files restricted?
14:52 dataverse-user They don't have the lock that identifies them as restricted
14:53 pdurbin Is your installation public? Do you mind if we look at it?
14:53 dataverse-user Is private because of the university policy
14:54 pdurbin Ok, no problem. You said the dataset hasn't been published yet. And you must be logged in or else you wouldn't see the files. Hmm.
14:54 dataverse-user Correct
14:54 pdurbin Is this Dataverse 4.20?
14:55 dataverse-user It's v4.9.4
14:55 pdurbin Ok, so an older version.
14:55 pdurbin Are you logged in as a superuser?
14:55 dataverse-user Yes
14:56 pdurbin Would it violate university policy if you take screenshots and open a GitHub issue?
14:56 dataverse-user No sir
14:56 dataverse-user I can do that
14:56 pdurbin Fantastic! Please open an issue at https://github.com/IQSS/dataverse/issues
14:56 dataverse-user Thank you!
14:56 dataverse-user will do
14:57 pdurbin dataverse-user: anything else we can do for you?
14:57 dataverse-user Not for the moment... Thanks for your time and have a great day!
14:57 pdurbin donsizemore: yes, I'm a huge fan of the Data Curation Tool and uploaded a ton of screenshots to https://github.com/IQSS/dataverse.harvard.edu/issues/32
14:57 pdurbin dataverse-user: great, I'll look for that issue. Thanks again.
14:58 dataverse-user Thanks to you, bye
14:59 donsizemore @pdurbin at the risk of being non-SLOPI, may I slack you a screenshot of what I get when I click on the "view" eyeball in DCT?
15:00 pdurbin please slack away
15:00 donsizemore @pdurbin I get the same behavior from dev2.dataverse.org that I'm getting on my local test
15:01 pdurbin I got your screenshot. Let's look at dev2
15:02 donsizemore @pdurbin we're expecting to see... something more than that.
15:04 pdurbin some numbers
15:05 pdurbin I'm asking Kaitlin in PKP Slack if she has time to help.
15:05 donsizemore i've tried a number of files and file formats. squashed a few errors i was getting in the javascript console, but... that would be wonderful.
15:08 pdurbin I guess I've used edit more than view. I've been using Data Explorer to view, to see the numbers. BRB, going on a call.
15:12 donsizemore @poikilotherm knock knock?
15:17 poikilotherm Hmm? On my way to construction site, but hit me. Will read up later.
15:26 donsizemore @poikilotherm eh, I can't look away from #5274. such a huge difference in warfile size. I ran that change in pom.xml against Payara5 with Gustavo's patched PrimeFaces
15:27 donsizemore @poikilotherm every automated test succeeded except uningest, which is a separate issue. if I'm reading your comments correctly it should be a safe switch under payara5?
15:41 kaitlin joined #dataverse
15:45 pdurbin donsizemore: psst. Guess who's here.
15:46 kaitlin Just taking a look at the chat history, but I think I'm missing some info about the issue
15:47 pdurbin kaitlin: please try this link (and remind me to reset the API token): https://scholarsportal.github.io/Dataverse-Data-Curation-Tool/?dfId=83&siteUrl=https://dev2.dataverse.org&key=65dcee41-151e-447f-884f-8aa9d8cb0bb1&fileMetadataId=77
15:47 kaitlin I see, so an issue with the view functionality
15:49 kaitlin do you have the original file as well?
15:50 pdurbin Yeah, the original file is here: https://dev2.dataverse.org/file.xhtml?fileId=83&version=1.0
15:50 kaitlin curious about how it's formatter
15:50 kaitlin ah, I see, thanks
15:51 pdurbin donsizemore noticed this with a different file
15:52 kaitlin I'm going to test the file on our demo site just to verify that it works on our install
15:52 pdurbin ok, thanks
15:53 poikilotherm donsizemore enlighten me - 5274 is about removing the bloody AWS SDK, right?
15:53 pdurbin Yep. Stripping AWS Mail component with TrueZIP blocks packaging for docker images #5274
15:59 poikilotherm Well, we should take a look at the AWS SDK, too. It is a pretty dated version, so it should be updated. And there is that Jackson dependency, which could be switched to provided, but we need to make sure that the shipped version is working for all things depending on it. I can't remember all deps relying on it, but there is also #6810. Then we don't have to care for our code, but still should take a look to avoid
15:59 poikilotherm unnecessary includes that bloat the WAR.
15:59 poikilotherm Apart from that, should be safe to do :-)
15:59 pdurbin I'd love to make the war smaller. And reduce deployment times.
16:01 poikilotherm Crossing fingers that will help with it...
16:05 poikilotherm donsizemore pdurbin are you asking me to take a look at this tomorrow?
16:05 pdurbin I'm not.
16:05 pdurbin poikilotherm: what's in focus? Drywall?
16:06 poikilotherm Right now? Yeah...
16:11 kaitlin I think I'll need to investigate further into how the ingest process handles certain filetypes, e.g. specifically when frequency calculations are being done
16:11 kaitlin I see the same issue in our instance
16:12 kaitlin but it works for my test spss and sav files
16:14 kaitlin I suspect currently frequencies are only generated for these file types, and not for tsv or other formats
16:14 kaitlin these file types meaning spss and sav
16:20 kaitlin unfortunately our lead dev is out of the office currently, so it may take some time for me to get back to you
16:20 kaitlin If you could file a ticket in the DCT github, that would be helpful!
16:23 pdurbin kaitlin: thanks! I'll see if I can get donsizemore or one of the folks he's working with to create it. :)
16:23 donsizemore @poikilotherm i'll hold off on tinkering with it then
16:24 donsizemore17 joined #dataverse
16:24 donsizemore17 @pdurbin was in the process of replying to kaitlin that I uploaded an .sav file and got the same behavior. was scrounging around for similar experiences before opening an issue
16:28 jri joined #dataverse
16:28 kaitlin joined #dataverse
16:29 kaitlin is the sav file somewhere I can download to test it on my end as well?
16:29 kaitlin sorry, dropped off the chat for a moment there
16:30 pdurbin If it helps, there are some sav files at https://github.com/IQSS/dataverse-sample-data
16:34 donsizemore17 @kaitlin hi =) i replied earlier but my connection died
16:35 donsizemore17 @kaitin I did try a native .sav and got the same behavior. I'm happy to open an issue but wanted to rule out ID10T(me) problems first
16:36 kaitlin usually sav files work for me, at least the sample files I have, but it could depend on the formatting/data of the file
16:37 kaitlin This is one of my test files: https://demodv.scholarsportal.info/file.xhtml?fileId=10159&version=1.0
16:38 donsizemore17 @kaitlin thank you! uploading to my test instance now
16:40 donsizemore17 @kaitlin the files we _want_ to use are going to be a long shot to start with... they're SAS7bdat from the 100MB-1GB range
16:41 donsizemore17 @kaitlin yup, your test file looks good
16:46 kaitlin I don't think sas7bdat files go through the tabular ingest process
16:46 kaitlin at least not in 4.17 where I'm testing
16:47 donsizemore17 I imported it into SAS and exported as .sav
16:47 kaitlin ah, got it
16:47 donsizemore17 You're right, Dataverse doesn't ingest tab-delimited (I think I remember an issue on this) but it is taking CSV
16:49 donsizemore17 and if i take your w1130 file in tab-delimited, convert the tabs to csv I can't view anything in DCT
16:50 donsizemore17 @kaitlin what would be the most helpful way to present this issue? file format at time of ingest, tabs v. commas, or just a general whine? so far my troubleshooting has been limited to different file formats
16:51 kaitlin I think specifically highlighting that tsv and xlsx files don't show anything in the "view" panel
16:51 kaitlin (I tested with xlsx as well)
16:53 kaitlin You mentioned a sav file where it didn't work - do you have that file available? I'm curious about that one.
16:53 donsizemore17 It's a 90MB file (I chose one of the smaller ones) and I don't think I can make it public.
16:53 donsizemore17 I can send you a "head -2" of the surrogate copy maybe
16:57 kaitlin no problem, was just wondering if it points to a possible bug
16:58 donsizemore17 or better yet let me ask my boss for permission to e-mail you either the SAS original or the SPSS file I'm attempting to view?
16:59 kaitlin sure, that works. I'll message you my email.
17:36 pdurbin donsizemore17: Dataverse should ingest tab delimited. And CSV. But not semicolon delimited.
17:37 donsizemore17 4.20 didn't ingest tab-delimited (.tab or .tsv extension)
17:37 donsizemore17 I can try again
17:46 pdurbin Huh. Weird. I swear I worked for a few releases at least. :)
17:46 pdurbin it* worked
17:52 pdurbin merged back in 4.9.2: https://github.com/IQSS/dataverse/pull/4854
18:00 donsizemore17 you're right, it's ingesting now (4.20)
18:04 pdurbin \o/
18:21 pameyer joined #dataverse
18:23 pameyer low urgency question - if I was going to do a PR for docker-aio/docker-dcm payara update, should that be against develop or a different branch?
18:32 donsizemore17 my understanding is breaking changes are allowed
18:35 pameyer which suggests "not develop" - or at least a blinking notice "do not merge into develop yet"
18:38 pdurbin pameyer: hi! We're kind of at a strange time right now where we have multiple branches up in the air. Yours could be the 4th or 5th. But once one of them gets merged, we want them all to get merged. To answer your question, please just make a pull request against develop, like usual. :)
18:39 pameyer pdurbin: thanks
18:40 pameyer probably wouldn't be that much harder to set it up to be able to do both glassfish4 and payara5 at the same time (in different containers, obviously)
18:41 pameyer not sure how useful it'd be
18:41 pdurbin meh, overkill, I'd say
18:41 pdurbin Thanks so much for jumping in on this.
18:41 pameyer don't thank me yet - haven't done it yet ;)
18:42 pdurbin pameyer: oh, you can do draft pull requests now. It's new.
18:42 pameyer I was going to comment on how much things have changed in a few months .... then realized it's been a year
18:44 pdurbin It's nice to see you again!
18:45 pdurbin donsizemore17: word on street is that you're a man in search of a bucket.
18:46 donsizemore17 preferably with ice and some vintage wild turkey
18:46 pdurbin a wild turkey pooped on my patio
18:46 pdurbin but I already told you that
18:47 donsizemore17 kevin caught some s3 problems on primefaces8 but the test VMs only have local storage
18:48 pdurbin Right. I forget how much access you have on our AWS account. We should both log in and take a look.
18:49 donsizemore17 i have an access key and secret key (to spin up ec2 instances) but nothing more
18:51 pdurbin There's a pameyer user who has access to ec2 and s3 and a donsizemore user that only has access to ec2.
18:51 pdurbin poikilotherm is full like me
18:54 poikilotherm I'm honored
18:54 donsizemore17 i don't want keys i don't need. denial of plausibility and such
18:54 * poikilotherm picks up screws and driver again
18:58 pameyer weird - I'm suprised I've got access to your aws
18:59 pdurbin pameyer: we've been wondering why our amazon bill is so high. Please try to tone it down a bit. Thanks!
18:59 pdurbin poikilotherm has two accounts actually. And there's a guy who hasn't worked here for a while.
19:10 pdurbin Anyway it sounds like donsizemore17 found the buckets. Good.
19:11 donsizemore17 It's Bouquet! B-U-C-K-E-T!
19:12 pdurbin chowdah
19:34 donsizemore17 @pdurbin knock knock?
19:37 pdurbin donsizemore17: hit me
19:37 donsizemore17 does dataversebot have permission to comment on pull requests?
19:37 pdurbin Thanks more of a poikilotherm question. I don't have the password.
19:38 donsizemore17 ah, i thought it was an IQSS setting
19:38 donsizemore17 or would you rather create an IQSSbot user within IQSS
19:38 pdurbin We can't let poikilotherm have all the fun with dataversebot.
19:41 pdurbin donsizemore17: do you need a bot right now, right now, right now, as my kids used to say?
19:41 donsizemore17 i don't but the PR job wasn't able to report back. i suspect mr. dataverse bot may not have permissions. will pester poikilotherm
19:43 pdurbin That bot used to leave comments on pull requests all the time. Something like, "Will one of the admins verify this patch?"
19:47 donsizemore17 and then it got quiet
19:47 pdurbin :)
19:48 pdurbin I'm looking at https://github.com/IQSS/dataverse/issues/6803
19:50 poikilotherm I'm not sure what permissions dataversebot has in IQSS/dataverse
19:50 poikilotherm Someone with admin rights might check thta
19:50 poikilotherm I'm happy to share the password with IQSS
19:51 poikilotherm There hasn't been any progress on that fronzier so far because no one asked ;-)
19:51 poikilotherm donsizemore17 you con just use the bot from Jenkins. The credentials are stored and ready to use.
19:52 donsizemore17 that's the one i had selected
19:54 poikilotherm In doubt, please look at the k8s jobs. Mr bot is very active for me :-)
19:55 poikilotherm Something in the back of my head is about there had been two credentials...
19:55 poikilotherm But I don't remember exactly and looking it up on my mobile is not so easy ;-)
19:56 poikilotherm And as said: pdurbin or someone else with admin permissions needs to check the permissions on the repo...
19:56 pdurbin Which repo?
19:59 donsizemore17 there is a dataversebot dockerhub user, a dataversebot github user, and a me.
20:01 pdurbin https://github.com/orgs/IQSS/teams?query=%40dataversebot says dataversebot is a member of one team: dataverse-sample-data-write
20:05 poikilotherm donsizemore17 seems like pdurbin is your man for granting mr dataversebot everything you need...
20:05 pdurbin lemme know how I can help
20:25 dataverse-user joined #dataverse
20:27 dataverse-user joined #dataverse

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.