Time
S
Nick
Message
00:33 jri joined #dataverse
03:33 jri joined #dataverse
07:21 jri joined #dataverse
07:24 jri joined #dataverse
10:41 cdsp-rmo joined #dataverse
11:32 mdehollander joined #dataverse
11:35 mdehollander
I am using the drag&drop upload of Dataverse, and I wonder if it is possible to retain the folder structure? Now all folders are expanded and all files are uploaded to the root of the dataset.
11:37 pdurbin
mdehollander: hi! A lot of people ask about this. :( Are you uploading a zip file or individual files?
11:39 pdurbin
If so, one workaround is to double zip your files. Or upload a tarball instead. For a lot of discussion on folder structure, please see https://github.com/IQSS/dataverse/issues/2249
11:50 donsizemore joined #dataverse
12:12 mdehollander
Wow, that is an extensive thread. When I zip the folders, they are automatically expanded with a complaint about identical file names (which is correct). When I zip that zip file again, only the first zip file is shown, not the folder with content. I am on a instance running 4.6.1
12:24 pdurbin_m joined #dataverse
12:25 pdurbin_m
mdehollander: so you expect to see folders
12:26 pdurbin_m
Can you please leave a comment on that issue?
12:28 pdurbin_m
mdehollander: can you think of a webapp that presents folders well?
12:41 mdehollander
Indeed, I expect to see folders. Is that possible. Are there any datasets on Dataverse that contain folders?
12:44 mdehollander
I think Nextcloud, Owncloud, Google Drive, Dropbox, almost all cloud based storage systems can handle a hierarchical folder structure.
12:50 pdurbin_m
Sure. I use Google Drive a lot.
12:51 djbrooke joined #dataverse
12:53 pdurbin_m
mdehollander: I've seen some mockups where we are considering supporting a folder structure but we haven't worked on any code yet.
12:54 pdurbin_m
djbrooke: morning. Do you know if any of those mockups are publicly available?
13:01 pdurbin_m
mdehollander: most people use the double zip work around, I think.
13:04 djbrooke
thanks mdehollander for the comment on 2249. That area is part of a design overhaul of the dataset and file pages, and I think we don't have a solution that works from both the technical and design perspectives yet. Some previous thinking is on page 4 here: https://docs.google.com/document/d/1l7bh0n7tAEAndhzeJWW87qaLeMoAmsLA2XwKDq1A3m8/edit#heading=h .29d4hixvslhm
13:22 Laerte18 joined #dataverse
14:05 pameyer joined #dataverse
14:06 pdurbin
Ah, "/Example/Hierarchy/Structure/README.docx". Yes, this is what I was thinking of. What do you think, mdehollander ? And thanks for the comment!
14:07 jri joined #dataverse
14:10 pameyer
mdehollander: DCM uploads retain folder structure, although they don't show it in the UI (and it's not drag and drop)
14:10 pameyer
it's also in the early stages
14:13 pdurbin
mdehollander: for more on DCM (Data Capture Module), please see http://guides.dataverse.org/en/4.8.6/developers/big-data-support.html
14:27 donsizemore joined #dataverse
14:40 mdehollander
@pdurbin & @pameyer: thanks for the info and links
14:42 mdehollander
@pdurbin, that example shows the complete path for each file isn't it? But when you have hundreds of files in many folders this is still not going to reduce the number of files in the root folder I guess...
14:43 pdurbin
mdehollander: you would rather just see a folder?
14:50 mdehollander
indeed!
14:53 jri_ joined #dataverse
14:55 pdurbin
mdehollander: ok, thanks. In that Google doc I'm looking at all the examples at the bottom (Dryad, Zenodo, etc) and I don't see folders. Do you? There are a lot of examples in there and I may have missed something.
15:02 donsizemore joined #dataverse
15:28 mdehollander
@pdurbin. Indeed, you are right. At least with Zenodo it is also not possible to upload a folder by drag&drop, I just tried.
15:29 pdurbin
mdehollander: ok, have you seen our comparison of repositories? One sec.
15:29 pdurbin
this: https://dataverse.org/blog/comparative-review-various-data-repositories
15:29 mdehollander48 joined #dataverse
15:29 pdurbin
mdehollander: there's a row called "Users are able to control dataset file hierarchy + directory structure"
15:30 pdurbin
mdehollander48: whoops, did you see my last messages?
16:40 dataverse-user joined #dataverse
16:42 dataverse-user
(raman here) donsizemore: pdurbin pointed me to your chat from earlier on the week re: TwoRavens. I'll send it over to James.
16:44 dataverse-user
The TwoRavens in the github repo you found is part of this project: https://www.darpa.mil/program/data-driven-discovery-of-models
16:45 pdurbin
dataverse-user: hi, Raman! Don isn't here right now but I'll try to remember to pass that along to him.
16:47 dataverse-user
For the summer, we're planning on getting an updated version going for Dataverse. We've been prioritizing D3M (grant-related) deliverables for evaluations.
16:47 dataverse-user
(all open source)
16:48 pdurbin
nice
17:31 pameyer
just missed raman :(
18:54 donsizemore joined #dataverse
19:26 donsizemore joined #dataverse
19:42 pdurbin
pameyer: any thoughts on https://github.com/IQSS/dataverse/issues/4605 ?
19:44 pameyer
pdurbin: initial thought is that I'm glad kcondon saw it being broken
19:44 pdurbin
heh. ok. I was wondering if you can reproduce it
19:45 pameyer
not yet sure if config, docs, broken code, etc yet
19:46 pameyer
I'd updated our fork to 4.8.6 and was about to do a full-up test of dcm/rsal; then realized I probably wasn't going to deploy it until 3083 came in and I'd have to update my fork again
19:47 pameyer
not sure if I can reproduce yet
19:47 pameyer
my ansible configs are still on old solr, since that's what our fork's on
19:51 pdurbin
donsizemore opened an issue for new solr at https://github.com/IQSS/dataverse-ansible/issues/17
19:52 pdurbin
and he already pushed some code, if it helps: https://github.com/IQSS/dataverse-ansible/commit/32608f7
19:55 pameyer
leaning towards docker-compose for dataverse/dcm
19:55 pameyer
because that'd been my plan for something more generally testable
19:55 pdurbin
instead of ansible? confused
19:56 pameyer
not for production - for finding out of dcm+dv is broken outside configs specific to our repo
19:56 pameyer
updating ansible would mean I could test it, but probably nobody else
19:56 pdurbin
oh, for testing. sure
19:57 pameyer
dv docker-aio and dcm test stuff means it *should* be fiddle w\ config files and create a compose file
19:57 pameyer
more before ansible, rather than instead of
19:58 pameyer
probably won't happen today with other stuff in the works though
20:00 pdurbin
sure, out of left field
20:24 pdurbin
Whelp, I'm just about out of here. Have a good weekend, everybody. Sunny in Boston.
21:37 pameyer left #dataverse
22:51 pdurbin left #dataverse
