Time Nick Message 11:00 donsizemore @pameyer bringing compute to data was a big push in Discovery Environment - in their architecture they accomplished this over iRODS with Condor ClassAds 14:35 pdurbin I just posted some thoughts on that thread: https://groups.google.com/g/dataverse-big-data/c/FC7uF5uL7RY/m/SXS7dawDAwAJ 14:36 pdurbin And I highly recommend watching the Dataverse/Globus integration (via their app, Synapse) video: https://youtu.be/VYq8Fr_3dhU 15:24 pameyer @donsizemore - sounds interesting; the more compute the better :) 15:25 pameyer @pdurbin - you definately do a better job of community wrangling than me ;) 15:26 pdurbin pameyer: ha! I'm quite grateful you posted yesterday. I tried to post but it didn't go through. 15:26 pameyer I hope I didn't sound too discouraging in that thread 15:26 pdurbin not at all 15:28 pdurbin did you watch the video yet? 15:28 pameyer nope, not yet 15:29 pameyer I have a weird aversion to videos when I'm trying to keep focused on stuff 15:32 pdurbin well, I'm glad you tolerate IRC 15:33 pameyer my irc client doesn't require headphones/audio ;) 15:33 pdurbin In the video I don't believe he demos file hierarchy. 15:33 pdurbin so now I'm wondering, since you brought it up 15:34 pameyer storage ids and in-place compute don't seem to play nicely together 15:37 pdurbin How does in-place compute work with S3? 15:38 pameyer not sure - I don't know enough about how that dataverse storage driver works 15:38 pameyer I'd guess about the same, but that's only a guess 15:38 pdurbin Oh, sorry, I mean outside the context of Dataverse. I'm sure compute happens on AWS. 15:39 pameyer ah, gotcha 15:40 pdurbin maybe S3 gets mounted as a POSIX filesystem or something 15:40 pameyer my (limited) understanding is that POSIX mounts of S3 hurt performance a reasonable amount 15:41 pdurbin sure 15:41 pdurbin so probably people look for tools that support S3, spark or whatever 15:41 pameyer if you're pipelines can talk s3 directly, I think it uses bucket addresses/labels/etc 15:42 pdurbin probably 19:14 pdurbin I'm scared. On my new laptop, echo $SHELL shows /bin/zsh 19:15 donsizemore trade you 19:17 pdurbin heh, suddenly I feel less scared 19:20 pameyer pdurbin: csh is probably still there... 19:21 pdurbin probably, I was using bash before, on my old mac laptop. this new one is also a mac 19:22 pameyer yeah - they switched the default to zsh with 10.15 19:22 pameyer either because they don't like gpu, or they like annoying me 19:22 pameyer gpl, I mean 19:23 pdurbin gotcha, I thought I heard this is coming. I'm running 10.14 on the old one 19:25 donsizemore https://www.howtogeek.com/wp-content/uploads/2019/10/ximg_5da79219c40ee.jpg.pagespeed.gp+jp+jw+pj+ws+js+rj+rp+rw+ri+cp+md.ic.4V1XjxAqkP.jpg 19:25 donsizemore may I suggest /bin/false 19:28 pdurbin :) 19:30 pdurbin donsizemore: ah, thanks for showing me that advanced options exist 19:31 pameyer they dropped 32-bit executables too 19:33 mateolan Hi dataverse folks--we're looking for a SPARQL wrapper for dataverses--or is there some way we can leverage domain specific ontologies for datasets within Dataverse? 19:33 mateolan does such a thing exist--or if not, is it on a product roadmap? 19:35 pameyer I'm not authorative, but as far as I know there's isn't a SPARQL interface (or rdf exporter, for that matter) 19:38 pameyer I've had reasonable luck with the dataverse search APIs for going into metadata - but that might be different than what you're thinking for leveraging ontologies 19:45 mateolan right--ultimately, I'd like to be able to reason over metadata, match column headers to ontology terms, and extract specific rows...withuot having to return a whole dataset...envisioning writing an interface to be able to do ontology annotation suggestions with some human curation... 19:45 mateolan seems like that would open up the dataverse in a big way 19:45 pdurbin mateolan: there is some subsetting functionality via API if that helps. 19:46 mateolan ooh lala @pdurbin, that sounds like a good place to start 19:46 pdurbin You can look for "subset" at http://guides.dataverse.org/en/4.20/api/dataaccess.html . Some external tools make use of it, I believe. 19:48 pdurbin I'd suggest starting a thread on the mailing list. Tell your story, etc. :) 19:48 pdurbin You're very welcome to chat here too, of course, but there are only a dozen people here. There are many more on the mailing list. 19:52 mateolan I'm only finding installation and security email addresses...and a basic twitter handle--guessing there is a specific developer email list? 19:53 pdurbin Well, there is a dev list but your story, your use cases, would be better told on the main list: https://groups.google.com/g/dataverse-community 19:54 mateolan ah, got it, thanks 19:55 pdurbin Sure. Here's the most recent thread on SPARQL, by the way: https://groups.google.com/g/dataverse-community/c/X004wd9ZBKM/m/PYdzYcGJAAAJ 19:56 pdurbin And it looks like SPARQL came up in a community call in May: https://docs.google.com/document/d/1Ve064T5ZnNMxOKLlP9Vu0I1o_pFtcN0vXXgbLXWekv8/edit 19:57 pdurbin But when I hear "domain specific ontologies for datasets within Dataverse" I think of custom metadata blocks: http://guides.dataverse.org/en/4.20/admin/metadatacustomization.html 20:01 pameyer me too - but I tend to forget most things csv or ddi related 20:04 pdurbin pameyer: is your domain specific ontology structural biology? Is that how you think of it? 20:17 pdurbin mateolan: very nice post! https://groups.google.com/g/dataverse-community/c/9T-2YO3czBI/m/rkLo0R21AwAJ 20:18 mateolan thanks pdurbin 20:19 pdurbin Often I try to reply right away but since we talked here, I'll let it sit and hope someone takes the bait. :) 20:20 pdurbin When I think about Dataverse installations that deal a lot with food, this one comes to mind: https://data.cimmyt.org 20:21 mateolan thanks for teh pointer 20:22 mateolan looks cool, is that a white-labeled dataverse? 20:34 pameyer pdurbin: when I think domain specific ontologies I tend to immediately veer off into blue sky about reproducability and probabilistic logic 20:35 pdurbin good 21:00 pdurbin Ok, folks. I'm out. Going on vacation for two weeks. See you on the other side. Have a great weekend! 21:04 pameyer have a good vacation!