IQSS logo

IRC log for #dvn, 2014-03-13

We've moved! Please join #dataverse instead. The new logs are at http://irclog.iq.harvard.edu/dataverse/today

| Channels | #dvn index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

All times shown according to UTC.

Time S Nick Message
00:08 skay joined #dvn
00:50 JonathanNeal joined #dvn
03:51 axfelix joined #dvn
11:28 ruebot joined #dvn
14:33 LyndsySimon joined #dvn
15:53 skay pdurbin: I am hoping everyone who is interested in the data versioning and other data discussions all get together on a mailing list so that everyone can share ideas
15:57 pdurbin skay: new comments! just added! read them, read them! https://docs.google.com/document/d/18WDIS8hrFJvMJBcnRuQ8NfD-VxGq32vJ9WwlEgyyWZs/edit?usp=sharing
16:04 skay pdurbin: I saw! I set hte doc so that it would send me email anytime someone leaves a comment. I hadn't found that setting before. I like google docs more now.
16:04 pdurbin skay: ok. but where are YOUR comments?
16:06 skay :p
16:07 skay I've got the problem where I only want to leave amazingly profound comments
16:13 pdurbin uh huh
16:13 pdurbin that's why IRC is nice
16:13 pdurbin :)
16:17 skay yeah :)
16:18 skay I am thinking I disagree with you about not storing metadata in a metadata kind of file in the repo, but I need to read your doc in more depth to make sure I got the gist of what you meant
16:19 skay I like the niceness of being able to see diffs in text files with info about a line that changes... so for example, saving a pretty printed json file where you could see one field changing
16:19 skay but I probably just skimmed and forgot what you said and what I am saying isn't applicable
16:19 pdurbin skay: but should we let users edit that JSON and then commit and push?
16:20 pdurbin (in theory)
16:20 pdurbin (thought experiment!)
16:21 skay pdurbin: no, I would want to restrict it to being touched via an api that builds it
16:21 skay but if you are using a repo as the store, storing it in textual format where I get to take advantage of diffs is really nice
16:22 skay it's like the idea of those config files that get generated and then say, hey! you! don't edit this! it was generated!
16:22 skay but you still get to access the file and look it like you get to do normal files on a file system
16:22 pdurbin skay: if it's in the git repo you can edit it and commit and push. that's the point. should it (metadata) be in the repo? as JSON or whatever
16:22 skay I think you should restrict actions to the repo, and if that feature does not exist, then make it exist
16:22 pdurbin it's not me setting these rules. it's just how git works
16:23 skay you would have a layer on top of git that does the commit stuff
16:23 pdurbin the commit stuff
16:23 pdurbin skay: prototype it for me :)
16:23 skay you would not allow just any user to interact with git unless you want to allow them to shoot off their feed with a machine gun
16:24 LyndsySimon joined #dvn
16:24 skay take sumatra for example, people track simulation runs that generate data, sumatra tracks the diffs in the way that sumatra defines diffs. you can export a json file of those records
16:24 skay one moment while I find the source code for you
16:25 skay https://bitbucket.org/apdavison/sumatra/src/404ec764a305b5346de2e02569cf167cb59cf762/sumatra/recordstore/serialization.py?at=default#cl-14
16:26 skay now let's consider the case where there is something you are backing by git or some other thing that can store things in a textual way, and now let's consider the case where you have  json thing to store, such as what might be generated by a sumatra user
16:27 skay it will be the case that you will have a UI that users get to have access to, but your UI will sit on top of a service that does not give the user full access to the service being called
16:27 skay sumatra or something else in that role generates some metadata that is json
16:27 skay and stores it in version control
16:27 skay your service layer does the transformation to let the user see it
16:28 skay because your service layer even delivers plain old diffs to the user
16:28 skay the user does not have any write privs, but perhaps if you are feeling nice, you can give them some read privs to the backend store
16:28 skay here user, have a read only thingee that is the backend store of your stuff
16:28 skay GET the stuff!
16:29 skay PUT the stuff? POST the stuff? yeah right I don't think so.
16:29 skay or git protocol or whatever protocol you want
16:29 skay please not jini/pojo
16:30 skay I wouldn't actually take sumatra records and commit changes over them
16:31 skay I was just usng one of those as an example of some metadata about some use of data
16:31 skay I was thinking of taking hte sumatra project and making something like a service interface to it
16:32 skay also, irc is not completely transient when you log the channel so this means I say nonsense and it is on record
16:32 skay later when I know more I will be so embarassed that I said all this nonsense, what was I thinking? etc.
16:32 skay at elast it isn't profound
16:33 skay etc
16:33 skay man, I am going to have an unhappy boss if I don't have something pretty to demo by tomorrow
16:33 skay I should stop thinking about stuff other than doing that
16:35 skay btw, before I go, http://pythonhosted.org/Sumatra/index.html and http://pyvideo.org/video/2039/using-sumatra-to-manage-numerical-simulations-sc
16:35 pdurbin skay: sorry, was fighting with our designer. will catch up in a bit
17:29 pdurbin skay: I still haven't caught up but wanted to link you to this: Git can facilitate greater reproducibility and increased transparency in science « Berkeley Initiative for Transparency in the Social Sciences - http://bitss.org/2014/03/12/git-reproducibility-transparency/
17:32 skay pdurbin: I am not catching up yet, but speaking of interesting blog posts and the fact that you mentioned interest in docker here is: http://bcbio.wordpress.com/2014/03/06/improving-reproducibility-and-installation-of-genomic-analysis-pipelines-with-docker/
17:33 skay arg why do people post links to pdfs
17:33 skay why don't people who want to share information share it in a web friendly form as well as a pdf?
17:34 * skay goes back afk
17:34 LyndsySimon joined #dvn
17:35 pdurbin javaeebot: lucky beyond the pdf conference
17:35 javaeebot pdurbin: http://www.force11.org/beyondthepdf2
17:40 axfelix joined #dvn
19:22 LyndsySimon joined #dvn
19:47 LyndsySimon joined #dvn
20:37 LyndsySimon joined #dvn
22:14 axfelix joined #dvn
23:37 garnett joined #dvn

| Channels | #dvn index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

We've moved! Please join #dataverse instead. The new logs are at http://irclog.iq.harvard.edu/dataverse/today