IQSS logo

IRC log for #dataverse, 2019-12-19

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

All times shown according to UTC.

Time S Nick Message
08:40 jri joined #dataverse
10:22 sivoais joined #dataverse
10:33 stefankasberger joined #dataverse
11:12 Youssef_Ouahalou joined #dataverse
11:15 Youssef_Ouahalou Hello everyone, when we add a new metadata, how do we make it appear in ddi output?
11:51 bro joined #dataverse
11:56 pdurbin Youssef_Ouahalou: well, it should appear in the native JSON output. Does it?
11:58 Youssef_Ouahalou Yes it appears in json
12:01 pdurbin Good. The native JSON output is flexible in that all the values are shown. All the other formats included DDI are hard coded. Does the field you're adding map to DDI cleanly? Is there a place for it in DDI?
12:04 Youssef_Ouahalou Can I add it manually?
12:07 pdurbin You would have to fork the code and edit some Java. Could this field be used by other installations of Dataverse? Or is it somehow specific to your institution? I'm wondering if making a pull request makes sense, to bring the feature to everyone.
12:08 MrK joined #dataverse
12:10 Youssef_Ouahalou I think it is specific to our institution, it would be interesting if it were more flexible
12:13 Guest33048 Hi everyone! I have a question about the controlled vocabularies and their translations. I've manage to translate some of my controlled vocabularies by doing as said in the doc (http://guides.dataverse.org/en/latest/admin/metadatacustomization.html?highlight=properties%20underscore) but I couldn't manage to make it work for values that have numbers or special characters. So I wondered if I could use the identifier from the controlled vocabulary part
12:15 pdurbin Guest33048: that's... strange. Numbers and special characters should work, I think. Can you share your TSV file?
12:15 pdurbin Youssef_Ouahalou: are you saying it would be interesting if DDI is more flexible?
12:16 Youssef_Ouahalou To enter new ddi fields
12:20 Guest33048 pdurbin: oh, then I guess I did somethings wrong (I've made many tries for different problems so I think I must have made a mistake). I'll try again. The only things to convert are spaces and uppercase right? So, for example, "Word 1: word2" should become "word_1:_word2"?
12:20 pdurbin Right but DDI is a standard (unlike Dataverse's native JSON format). So you have to map your institution-specific field to the standard somehow, right? You can't just invent a DDI field, I think. That said, there was a recent thread on the mailing list about some upcoming flexibility in DDI. Let me go find it, Youssef_Ouahalou
12:22 pdurbin Guest33048: I'm sorry but without seeing your TSV I'm having a little trouble picturing in my head what it looks like. But it's also early and I just had my first sip of coffee. :)
12:25 pdurbin Guest33048: you'd be very welcome to create a GitHub issue and attach your TSV to it. You might need to add .txt to it in order to attach it: https://github.com/IQSS/dataverse/issues
12:26 Guest33048 pdurbin: I've put an extra underscore in my property file. Sorry for the inconvenience! However can you confirm that the identifier in the TSV cannot be used as a property name?
12:27 pdurbin Youssef_Ouahalou: this is the thread I'm thinking about but it has more to do with supporting more scientific disciplines than supporting fields that are specific to an institution: https://groups.google.com/d/msg/dataverse-community/uKretKox_io/VTZsb49MAQAJ ("custom metadata blocks now easier to spin up and evaluate") ... Guest33048 you may be interested in that thread too.
12:29 pdurbin Guest33048: when I'm at my desk it would be easier to confirm. Can you please email support@dataverse.org so that I or someone else can get back to you about this?
12:31 Guest33048 pdurbin: I'll do that. Thank you for everything, and for the thread that I bookmarked :)
12:31 Youssef_Ouahalou Ok thank you very much i will read it,thank you for your answers
12:32 pdurbin Guest33048: thanks! If you have any questions about that thread, please let me know! There's a lot to unpack. :)
12:32 pdurbin Youssef_Ouahalou: perfect, thanks. Also, I think your boss emailed me. I'm reading it now. :)
12:34 Youssef_Ouahalou Hahhaha yes I think ☺
12:40 pdurbin Youssef_Ouahalou: I just found this: https://fosdem.org/2020/schedule/track/hpc_big_data_and_data_science/
12:44 Youssef_Ouahalou Is this about your presentation?
12:50 pdurbin Well, the lightning talks still haven't been announced. So I still don't know if my talk has been accepted. I'm just poking around the FOSDEM site trying to figure out where it might be good to spend some time. There's also a room for Java and a room for PostgreSQL.
12:52 Youssef_Ouahalou yes I also saw that, it could be interesting. Hoping that your speech will be accepted
12:52 poikilotherm joined #dataverse
12:55 poikilotherm Mornin' guys. What's up?
12:56 MrK Hi
12:56 Youssef_Ouahalou Morning nice and you
13:01 Benjamin_Peuch joined #dataverse
13:03 Benjamin_Peuch joined #dataverse
13:04 Benjamin_Peuch boop
13:04 Benjamin_Peuch Oh this works now. Hello everybody, this is Ben from the State Archives of Belgium
13:05 Benjamin_Peuch Hello Philip. I'm Youssef's coworker :)
13:06 poikilotherm Welcome Benjamin_Peuch
13:09 Benjamin_Peuch Thanks, poikilotherm. We are glad that we finally got down to setting up a Dataverse and seeing how far we can adapt it to our needs
13:10 Benjamin_Peuch We are planning to launch it in a couple of months. Youssef has worked hard to this end, and I'm told he got a lot of support from you all
13:20 pdurbin Benjamin_Peuch: hi! I got your email!
13:21 pdurbin Yes, lot of us in here are helping Youssef_Ouahalou and others get their installations of Dataverse launched. :)
13:22 pdurbin stefankasberger: hey, did you get my "mysterious message from Vienna" email? :)
13:23 Benjamin_Peuch We greatly appreciate the support, pdurbin. :)
13:24 Benjamin_Peuch Youssef told me about the exchange you just had about adding extra fields in Dataverse.
13:24 Benjamin_Peuch We do plan to do this properly, so the output in DDI would absolutely be compliant with DDI-Codebook 2.5.
13:25 Benjamin_Peuch I must say that some of those extra fields we want to add are meant to transfer to another XML language, Encoded Archival Description (EAD), for archival purposes.
13:26 Benjamin_Peuch Still, we have identified which DDI elements we can use as intermediary vessels to this end. They would simply be <notes> element in <stdyDscr>'s <citation>.
13:26 pdurbin Benjamin_Peuch: I'm flying back at 6am on that Monday. Friday *might* work. I'm arriving from Lisbon/PIDapalooza at 6:15am on that Friday. I wonder if it would make sense to host a "fringe" event about Dataverse at your institution: https://fosdem.org/2020/fringe/ . I need to spend more time looking at my calendar though. :)
13:26 Benjamin_Peuch That would be wonderful! We would make sure to prepare a lot of coffee. :)
13:27 Benjamin_Peuch We know of Dataverse users in France, Austria (hello stefankasberger!) and the Netherlands. They might be interested, especially if they plan to attend FOSDEM'20?
13:27 pdurbin My hotel is only a 29 minute walk from your institution (though I'd probably take public transportation). :)
13:27 Benjamin_Peuch Handy!
13:28 pdurbin I've been waiting to start a thread on the dataverse-community list about FOSDEM until I know if my lightning talk has been accepted or not. A lot of the rest of the schedule is up already. Longer talks, dev rooms, etc.
13:28 Benjamin_Peuch I should also mention we took good note of the advice in DV's manual, and we do plan to publish our metadata input.
13:29 pdurbin MrK: so this is another potential meetup in Europe. Perhaps. If we can pull it together. :)
13:29 Benjamin_Peuch Yes... Fingers crossed. I really hope you got to make this presentation.
13:30 Benjamin_Peuch *get
13:30 pdurbin Benjamin_Peuch: the thing I'm trying to figure out about your custom metadata fields is this... Can they be used by other installations of Dataverse? If so, we should work toward a pull request so everyone can benefit.
13:30 Benjamin_Peuch That is also a thing indeed, pdurbin.
13:31 pdurbin Otherwise, I fear you'd be forced to run a fork.
13:31 Benjamin_Peuch We had quite a few modifications for Dataverse in the gears until we realized that it had not been developed so as to be very heavily customized, especially regarding the core metadata (Citation and Terms essentially, I would say).
13:32 Benjamin_Peuch I must admit we interpreted the notion of open source a bit too freely. Indeed that was also my conclusion after you voiced concerns about departing too much from the original software (and our head of IT developments was of the same mind): it would pretty much amount to forking.
13:32 pdurbin Ok. Lots of people have forked Dataverse. You aren't alone. But ideally, we would merge changes into upstream, if we can.
13:34 Benjamin_Peuch Yes, that was also our plan. We thought we would take another approach and publish information about our needs (as archivists, data specialists) and that of our users (we only have a few at the moment, but they're qualified social scientists) so that it's open for discussion, and we can see from here what can be integrated in DV in the short, medium or long term.
13:34 Benjamin_Peuch I know Youssef already opened a few issues on your GitHub and I don't want to spam you, so I'm going to look into it and synthesize our points.
13:35 stefankasberger @pdurbin: Yes. It was Lars Kaczmirek, CEO of AUSSDA (my boss).
13:35 pdurbin stefankasberger: oh! I never got a follow up email from him. Thanks!
13:36 pdurbin Benjamin_Peuch: honestly, often is easier to get quick answers here than wading through old GitHub issues. But you're welcome to go wading. :)
13:37 Benjamin_Peuch Oh okay. Does that include saying "Hey, we want this! Implement that!"? :p
13:38 pdurbin Benjamin_Peuch: something else I'd like to put on your and Youssef_Ouahalou's radar is that DANS (I think) is transforming Dataverse's native JSON to various XML using a crosswalk outside of Dataverse. It's on GitHub somewhere.
13:38 stefankasberger Yes, cause I will get in touch with you, once i have time. Would it be possible to have a short call tomorrow to talk about some things?
13:38 poikilotherm pdurbin will most likely say: please open an issue for that @Benjamin_Peuch
13:38 pdurbin stefankasberger: tomorrow is looking quite open for me. After that I'm on holiday until Jan 2.
13:39 Benjamin_Peuch Oh that sounds very interesting. Thanks pdurbin. I assume they might be doing this with the DataverseEU project.
13:39 pdurbin Oh, and we have Zoom at Harvard now, which is nice. I hosted my first Zoom with the new maintainer of dataverse-client-r on Tuesday.
13:39 stefankasberger Me the same. I will get in touch with you, okay?
13:41 pdurbin stefankasberger: sure, let's pick a time.
13:41 pdurbin Benjamin_Peuch: I'm having trouble finding the crosswalk repo :(
13:41 stefankasberger 11/12h CET?
13:42 donsizemore joined #dataverse
13:42 pdurbin If memory serves, Slava was transforming Dataverse's native JSON to an XML format used in their in house system called EASY.
13:42 Benjamin_Peuch pdurbin: That's okay. I'm in touch with Marion and Slava. I can ask them directly. Thanks. :)
13:42 pdurbin Benjamin_Peuch: great. If you find the repo, please open an issue so we can add it to the guides.
13:50 Benjamin_Peuch Will do.
13:58 Benjamin_Peuch left #dataverse
14:02 donsizemore @pdurbin may i pester you about the python3 installer when you have a minute?
14:10 poikilotherm donsizemore: could you please just use ansible for that and bundle it?
14:12 MrK pdurbin: what kind of meeting :P?
14:12 poikilotherm donsizemore and pdurbin: or maybe use sth like https://github.com/jordansissel/fpm to create real packages, with ansible behind the scenes?
14:13 poikilotherm That's how gitlab does it: they provide packages with a bundled chef installer doing all the hard work for them
14:31 stefankasberger joined #dataverse
14:50 pdurbin Wow, -5F (-21C) with wind chill. Glad I wore snow pants to bike in. I hear in Australia they have record heat.
14:51 pdurbin donsizemore: please pester away
14:51 pdurbin MrK: nothing concrete yet. :)
14:59 donsizemore @pdurbin so, one of the long-running criticisms of the ansible role is that it's not idempotent, because it's a wrapper for the dataverse installer, which itself is not idempotent
15:00 donsizemore @pdurbin i can remove some portions of the ansible role to allow the installer to do it all (fine) but that's a step backwards from an idempotence perspective
15:01 pdurbin yikes
15:01 pdurbin So how do we move forward instead of backward? :)
15:02 donsizemore regarding the installer, one first step might be the addition of a --no-db flag (in addition to the --db-only) flag
15:03 donsizemore under perl (and now python) the installer insists on creating the database as an admin user and creating the tables up front
15:04 pdurbin Well, aren't the tables created by the deployment of the war file? I'm agreeing with you but trying to get more specific.
15:04 donsizemore this would begin to pave the way for normalizing the upgrade process
15:04 donsizemore the perl installer crabs "a database exists! i'll only install onto a squeaky-clean postgres" or something similar
15:05 donsizemore so a --no-db flag would relieve the first such blocker to repeated runs. (or i can just let ansible call the script directly, but thought i'd ask)
15:06 pdurbin donsizemore: should we get on a Zoom with Leonid?
15:07 donsizemore that'd be fine with me, just thinking about repeated installs, ("using Ansible to manage upgrades" as has been long requested), and direction/design
15:09 pdurbin Sure. And he might want to pick your brain about the rewrite. He started from your work. Also, I recently heard about a POP concept I'd like your take on. Let me go find it.
15:10 MrK pdurbin: Installer is now in python?
15:10 pdurbin MrK: in development. Do you prefer Perl?
15:10 MrK pdurbin: No I'm happy its in python :p
15:11 pdurbin donsizemore: here's where I heard of POP: Making Complex Software Fun And Flexible With Plugin Oriented Programming - Episode 240 - https://www.pythonpodcast.com/plugin-oriented-programming-episode-240/
15:12 pdurbin My question is if you (and others here) think it's overkill to consider a POP architecture for the installer.
15:20 donsizemore i mean, this is proper design =) but as you say, the installer is just an installer.
15:21 pdurbin yeah
15:21 pdurbin Let me walk down the hall and see if Leonid is around. And if anybody made coffee.
15:26 poikilotherm joined #dataverse
15:30 pdurbin donsizemore: good news. There's coffee and Leonid is happy to do a video call tomorrow any time after standup. Will you be around? Should we invite poikilotherm and MrK and anyone else who has strong opinions about the installer? :)
15:30 poikilotherm :-D
15:31 poikilotherm The strongest opinions are about those pesky resource creations inside glassfish
15:31 pdurbin MrK poikilotherm: Mozilla just announced they're moving from IRC to Matrix: https://discourse.mozilla.org/t/synchronous-messaging-at-mozilla-the-decision/50620
15:31 poikilotherm Everything is very likely not to tangle me for K8s ;-)
15:33 poikilotherm +else
15:34 donsizemore @pdurbin i have a lunch close to noon but otherwise i'll be here
15:35 pdurbin donsizemore: ok, should we squeeze it in before your lunch?
15:35 donsizemore @pdurbin fine by me. it's just a design question, really (and i like to pick a path before picking up the rake and lopping shears)
15:39 pdurbin donsizemore: cool. I just started a doc with talking points. Can you please add some bullets? https://docs.google.com/document/d/1CSPTkFT8i38a0rZShKdoHL4UchiD_njc44-cqxVFc7k/edit?usp=sharing
15:40 donsizemore @pdurbin that's my basic question =) and it's not really my question (I pretty much use Ansible as a one-shot installer) but a bunch of community members have asked for idempotence
15:42 pdurbin Ok. Well, I already put a bug in Leonid's ear about idempotence. And he mentioned there are some flags already that might help. If we think of other talking points, we can add them. If not, we can just use that doc for notes.
15:46 poikilotherm pdurbin: it was too tempting to add some points ;-)
15:48 poikilotherm pdurbin: I don't think I can make it - 1130 Boston = 1730 over here...
15:51 pdurbin poikilotherm: that's why I created the doc. Thanks! Can you put your name next to your questions?
15:51 poikilotherm Done
15:52 pdurbin thanks!
15:52 poikilotherm Shall I do a quick browsing through the installers to nail down some more questions?
15:53 pdurbin If you mean the new branch, sure, please take a look: https://github.com/IQSS/dataverse/compare/3937-rewrite-installer-in-python-again
15:53 poikilotherm No I meant the old installer and its pain points
15:53 pdurbin You're looking for pain in that old Perl script?
15:53 poikilotherm Nope
15:54 poikilotherm Things like setup-all.sh etc
15:54 poikilotherm Those things have to DIE, IMHO
15:54 Benjamin_Peuch joined #dataverse
15:56 pdurbin Well, at least it's possible to install Dataverse from the command line. Better than forcing people to use a GUI. :)
15:56 poikilotherm :-D
15:56 pdurbin But yeah, let's make the installation process simple and awesome.
15:57 Benjamin_Peuch :thumbup:
15:57 pdurbin Are we inspired by any software that's simple to install?
15:57 poikilotherm It would be awesome to have a cli tool called "dataverse-adm"
15:57 poikilotherm Which does bundle simple things for you
15:57 poikilotherm Like those curls to block api endpoint, set unblock key etc
15:59 poikilotherm Or activate FAKE provider
15:59 poikilotherm Hmm wondering if this might be a good fit into pyDataverse
16:01 pdurbin sort of like asadmin?
16:01 poikilotherm Yeah
16:01 pdurbin And it calls into Dataverse APIs?
16:01 poikilotherm Don't make people lookup curls, give them small helper tools
16:02 poikilotherm This is all heavily inspired by Gitlab...
16:02 pdurbin great idea
16:03 poikilotherm Lots of installer stuff could be done in such a tool
16:03 poikilotherm The new installer is another HUGE beast, which is not easy to maintain
16:03 pdurbin What does GitLab write their adm tool in?
16:05 pdurbin poikilotherm: did you see my post above about POP? Would that help make it less of a HUGE beast? Or should we use some other language besides Python? How can we make the new installer awesome?
16:06 poikilotherm Gitlab seem to have a wrapper around rake tasks, so this is Ruby
16:06 pdurbin bleh, ruby ;)
16:07 pdurbin So you have to gem install it?
16:07 poikilotherm I'm not saying this should be done in ruby
16:07 poikilotherm Nope, in most cases (if not using K8s or similar) you will use packages (rpm/deb) and this is a simple binary shipped in the packages
16:08 poikilotherm Its just a wrapper, you could use rake from a bundle install (as a gem)
16:10 donsizemore @pdurbin did you want? can we push it back until afternoon EST?
16:11 pdurbin donsizemore: sure, afternoon is fine. It's too late for poikilotherm anyway. What time works for you. Not too late on a Friday before a more-than-a-week holiday, please. :)
16:12 pdurbin poikilotherm: ok, just a yum install or an apt-get install away. Good.
16:14 poikilotherm pdurbin: I added a note about my idea to the doc
16:15 * pdurbin looks
16:17 donsizemore @pdurbin any time after say 1300 should be fine?
16:23 poikilotherm Read you guys tomorrow
16:43 pdurbin donsizemore: at standup just now Gustavo reminded us that you're on Slack. I just started a direct message thingy with you, me, Leonid, Kevin, and Gustavo.
19:17 pdurbin Looks like Leonid just left a comment here: https://github.com/IQSS/dataverse/issues/3937#issuecomment-567605666
19:44 bricas joined #dataverse
19:56 dataverse-user joined #dataverse
19:56 dataverse-user hi for all
19:56 pdurbin how's it going, dataverse-user? :)
19:59 dataverse-user fine thanks,  have a question about harvard dataverse. If i want to storage the research data of my project in the harvard dataverse repository. how much its it? ,
20:00 pdurbin Harvard Dataverse offers free data hosting if the data is under a certain size.
20:03 pdurbin Harvard Dataverse is launching a new support/about/reference website soon that will answer questions like this. It's not quite ready but here's the issue where the work is being tracked: https://github.com/IQSS/dataverse.harvard.edu/issues/26
20:04 pdurbin dataverse-user: does that help?
20:32 dataverse-user @pdurbin: what happend if the data its different size or new institution not in  consortium its required dataverse storage?
20:37 pdurbin I think the quota is 1 TB right now. Your institution does not need to be part of any consortium. Individuals can even upload data.
20:44 pdurbin donsizemore: this is new to me, might be of interest: https://github.com/jggautier/dataverse-metadata-scripts
21:02 pdurbin dataverse-user: I assume you're talking about http://dataversecommunity.global . Not sure.

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.