Time
S
Nick
Message
00:12 bmckinney joined #dataverse
02:11 garnett joined #dataverse
02:16 garnett joined #dataverse
02:17 garnett joined #dataverse
02:44 garnett joined #dataverse
03:12 axfelix joined #dataverse
03:50 garnett joined #dataverse
03:51 garnett joined #dataverse
04:30 garnett joined #dataverse
04:38 axfelix joined #dataverse
05:25 djbrooke joined #dataverse
05:34 LyndsySimon joined #dataverse
05:36 skay_ joined #dataverse
05:36 JonathanNeal joined #dataverse
06:07 skay_ joined #dataverse
06:12 JonathanNeal joined #dataverse
06:51 bmckinney joined #dataverse
07:38 jri joined #dataverse
08:30 bencomp joined #dataverse
08:41 jri joined #dataverse
08:56 bmckinney joined #dataverse
10:24 bmckinney joined #dataverse
10:33 bencomp1 joined #dataverse
12:00 bmckinney joined #dataverse
13:19 djbrooke joined #dataverse
13:20 bsilverstein joined #dataverse
13:37 pdurbin
bmckinney pameyer1: I just created a sprint planning doc for our 2pm meeting: https://docs.google.com/document/d/1Oc6kLLUf8VG1PjGwBLH0qs64einss3-Vcu9-P06un7g/edit?usp=sharing
13:38 pdurbin
djbrooke: have you met Bill and Pete yet? If not, you should swing by the small conference room if you can.
13:39 djbrooke
Will do - thanks!
13:39 bmckinney
cool, thanks
13:40 bmckinney
pdurbin: i seem to remember you were initiating some meetings with Leonid?
13:40 pdurbin
bmckinney: right. He said to grab him whenever. This Thursday or next or whenever. So let's do that.
13:41 bmckinney
pdurbin: excellent
13:41 pdurbin
What do we call that topic? "Is rsync compatible with versioning of files?"
13:43 djbrooke joined #dataverse
14:03 pdurbin
bmckinney: Leonid is under the impression that you have a solution to this problem.
14:05 bmckinney
pdurbin: no
14:06 bmckinney
pdurbin: as i've mentioned before, this isn't a use case for experimental data (you shouldn't be changing x-ray images)
14:06 bmckinney
but we understand it is an issue for others
14:07 jri joined #dataverse
14:08 pdurbin
bmckinney: I tried to disabuse him of the notion. :)
14:09 bmckinney
pdurbin: this will be a good chance to get on the same page
14:10 pdurbin
my hacky solution is to say that all data should be rsync'ed to a folder called "1" and if you publish and change the files, all the files in "1" get copied to a folder called "2" before the rsync can continue... this is slow and stupid but solves the problem
14:11 bmckinney
yes, and pameyer can share his current workarounds for this (it's a rare case i believe)
14:14 pameyer1
it's currently a rare case, and the current workarounds are suboptimal
14:14 pameyer1
it was a nice surprise to find out that some of the metadata people had already drawn a line in the sand between experimental data and reference dataset (which might change)
14:15 jri joined #dataverse
14:16 pdurbin
pameyer1: but you're thinking that some day some of your users might have a use case for using rsync *and* having versions of their datasets, I believe.
14:17 pdurbin
outside of structural biology, perhaps
14:17 pameyer1
we've got multiple datatypes - and one of them is much closer to reference dataset than primary experimental data
14:17 pameyer1
so even within structural biology it's at least a minor use case
14:18 pdurbin
which datatype do you expect to have reference datasets?
14:21 pdurbin
which datatype do you expect to have files that change?
14:21 pameyer1
what we're calling structural models at the moment
14:23 pdurbin
pameyer1: thanks. I left myself a note: https://github.com/pdurbin/dataverse/commit/c7175231df70166129d7866b9e7b2566d3d50031#commitcomment-17806291
14:25 pdurbin
maybe we can write a user story around stuctural models
14:25 pdurbin
and rsync
14:25 pdurbin
and files changing
14:27 pdurbin
On a related note, I recently watched https://youtu.be/Yth7O6yeZRE?t=8h16m about how Dat ( http://dat-data.com ) versions files. I have no idea how well it deals with datasets of 50 GB or more though.
14:29 bsilverstein
pdurbin: for what it's worth I've pinpointed where the account creation email comes from - MailServiceBean.java - and it looks simple enough to limit/edit to confirmed email addresses.
14:29 pdurbin
bsilverstein: yep, that sounds right. Awesome.
14:30 bsilverstein
although from here, would confirming an email address utilize MailServiceBean? would I add a case to the switch..?
14:31 pdurbin
bsilverstein: well, to confirm the email address, you'll need to process a token in a URL . Maybe like https://github.com/IQSS/dataverse/blob/v4.3.1/src/main/webapp/passwordreset.xhtml
14:32 bsilverstein
right, got it. I'll sift through it, thanks Phil!
14:32 axfelix joined #dataverse
14:33 pdurbin
yep, no problem
14:33 pdurbin
pameyer1: back to mitigating password attacks for a moment... great minds think alike: http://irclog.perlgeek.de/crimsonfu/2016-06-09#i_12634167 :)
14:35 pameyer1
funny how that works sometimes...
14:36 pameyer1
re dat - looks like a cool architecture, but without testing it (aka - could be wrong) I assume that javascript + signifcant amounts of data = :(
14:36 pameyer1
and "exponential backoff" sounds like the term I was trying to think of
14:37 pdurbin
pameyer1: actually, in #dat they are telling me they have a command line tool that can upload all of Wikipdedia (which is quite big) to a Dat server *and* versioning is still supported: https://botbot.me/freenode/dat/2016-06-09/?msg=67590264&page=2
14:38 pdurbin
see https://github.com/datproject/docs#sharing-data and https://github.com/datproject/docs#versioning
14:39 pameyer1
sounds like I should poke around in it a bit
14:39 pdurbin
Maybe "Dat" could be in the menu along side rsync, Globus, Aspera, etc. some day.
14:42 pameyer1
maybe - part of the idea of trying to do multi-protocol is that new protocols sometimes show up
14:43 pdurbin
heh. sure
14:44 pdurbin
sounds like we'd be considered an "early adopter" of Dat at this point: https://botbot.me/freenode/dat/msg/67591282/
14:47 pameyer1
"early adopter" can be good or bad sometimes
14:47 pdurbin
skay: you first clued me in to Dat at http://irclog.iq.harvard.edu/dvn/2014-03-05#i_6761
14:48 * skay
waves
14:49 pdurbin
:)
14:49 pdurbin
anyway, back to rsync I guess
15:13 djbrooke joined #dataverse
15:15 djbrooke_ joined #dataverse
15:15 djbrooke_ joined #dataverse
15:21 djbrooke joined #dataverse
15:48 pdurbin
pameyer1: still there?
15:48 pameyer1
for a few more moments
15:49 pdurbin
pameyer1: Leonid is not available to meet today afterall. Next week.
15:49 pameyer1
ok
16:26 djbrooke joined #dataverse
17:00 bmckinney joined #dataverse
17:05 metamattj joined #dataverse
17:13 bmckinney joined #dataverse
17:28 djbrooke joined #dataverse
17:30 djbrooke joined #dataverse
17:51 bmckinney joined #dataverse
17:54 bmckinney joined #dataverse
18:02 djbrooke joined #dataverse
18:05 bmckinney joined #dataverse
18:10 djbrooke joined #dataverse
18:15 bmckinney joined #dataverse
18:19 bmckinney joined #dataverse
18:22 djbrooke joined #dataverse
18:27 djbrooke joined #dataverse
18:52 shauna joined #dataverse
18:57 axfelix joined #dataverse
19:01 djbrooke joined #dataverse
19:08 djbrooke joined #dataverse
19:20 metamattj joined #dataverse
20:08 bmckinney joined #dataverse
20:31 djbrooke joined #dataverse
21:17 axfelix joined #dataverse
21:32 pdurbin
I'm still thinking about Dat: https://twitter.com/philipdurbin/status/741011397312667649 :)
21:35 djbrooke joined #dataverse
23:13 axfelix joined #dataverse
23:44 bmckinney joined #dataverse
23:44 garnett joined #dataverse
23:46 garnett joined #dataverse
