Time
S
Nick
Message
00:48 jri joined #dataverse
03:48 jri joined #dataverse
07:21 jri joined #dataverse
10:01 poikilotherm joined #dataverse
12:18 donsizemore joined #dataverse
13:12 andrewSC
pdurbin: interesting!
13:13 andrewSC
I'll keep an eye on the conversations but I don't think I'll be able to participate much in them
13:13 andrewSC
the lab i work out of is a bit of a gray area in terms of what is actually publicly published since we work closely with the intelligence community
13:16 andrewSC
just looking at the image in that gh issue the part that we're moving towards is having a couple of basic frameworks/automated workflows to get people spun up quick without having to trudge through all the scaffolding work
13:17 andrewSC
the thing is it's pretty specific to our environment and how we operate internally
13:25 pdurbin
andrewSC: wow, intelligence community. neat. Tell us more about the scaffolding stuff. There's a workbend from NDS that may or may not be of interest.
13:25 pdurbin
workbench*
13:26 pdurbin
poikilotherm: did you see http://irclog.iq.harvard.edu/dataverse/2018-10-03#i_74583 ?
13:41 andrewSC
pdurbin: neat, I'll have to take a look into the nds labs workbench some more--first time hearing of it!
13:41 andrewSC
wrt the scaffolding stuff, there's been a pattern of how we develop prototypes at the lab--some revolve around viz stuff others are more heavy lift operations with some sort of frontend to manage it
13:42 pdurbin
andrewSC: can I assign https://github.com/IQSS/dataverse/issues/4152 to you? :)
13:42 andrewSC
I appreciate the thought--I can't take that on at the moment though
13:43 pdurbin
no worries :)
13:43 pdurbin
I just met the developer a few weeks ago. The project is still kicking.
13:43 andrewSC
basically we have this need to standardize how we deploy these prototypes and even before that, build them.
13:43 andrewSC
nice!
13:43 pdurbin
Actually, it's the same guy who opened the Whole Tale issue.
13:43 andrewSC
oh interesting
13:44 andrewSC
but yeah.. the scaffolding stuff just revolves around having something that people can leverage to get started quicker rather than wade through aws docs, travis ci docs, docker, docker compose etc docs
13:44 pdurbin
He was saying he'd like to move workbench to helm charts some day.
13:45 jri joined #dataverse
13:46 andrewSC
basically let people get to the thing they are actually good at and remove that barrier of getting spun up as much as possible
13:46 pdurbin
andrewSC: yeah, I'm wondering if this new script donsizemore and I have been working on will help people kick the tires on Dataverse more easily: http://guides.dataverse.org/en/4.9.3/developers/deployment.html#deploying-dataverse-to-amazon-web-services-aws
13:48 andrewSC
hmm interesting
13:50 pdurbin
I mention the new script in passing at http://guides.dataverse.org/en/4.9.3/installation/prep.html#advanced-installation
13:52 andrewSC
i think we're leveraging the ansible repo to do our aws deployment but I'd have to double check--I'm a bit removed from the migration process atm
13:52 pdurbin
cool
13:52 andrewSC
this is a little bit OT, but what are your thoughts on Java as a technology/lang for a large project like this?
13:52 donsizemore
@andrewSC pull requests welcome! if i had more time it would get more love
13:52 andrewSC
pros/cons?
13:53 andrewSC
donsizemore: :))
13:53 andrewSC
yeah i was just looking throught the actual bash script now
13:54 andrewSC
I think the point I'm trying to get to/ask is have there been struggles dev'ing DV because "It's a java thing"? does that make sense?
13:54 pdurbin
andrewSC: I feel like Java is good for largish teams. There are mechanisms for staying organized. Why?
13:55 andrewSC
also want to clarify not trying to start a flame war or any bs like that--I'm evaluating langs for a little side project of mine and i think long term java might be the ticket but i haven't had much experience with large scale java dev/deployments
13:55 andrewSC
pdurbin: ^^
13:56 pdurbin
I think Java is fine. I'm not sure about the dependency on Glassfish.
13:56 andrewSC
i mean with that being said it's "pick the right tool for the job"
13:57 andrewSC
but i've gotten everything from "write it using PHP " to "C# is the ticket" lol
13:57 andrewSC
so just trying to gather opinions :)
13:57 pdurbin
PHP I've done. I've never hacked on C# but I hear it runs on Linux these days.
13:58 andrewSC
yeah a buddy of mine keeps trying to sell me on it (he also works at msft lol)
13:58 andrewSC
so *shrugs*
13:58 pdurbin
I would say the biggest struggle is getting a dev environments set up. Every summer we have interns and it's tough. Tougher for contributors who aren't on our floor.
13:58 andrewSC
mmmmmmmmmm
13:58 pdurbin
I just pray that new contributors don't assume it's their fault and ask for help.
13:58 andrewSC
is it just getting all the deps/env setup so actual dev can take place?
13:58 andrewSC
lol
13:59 andrewSC
oh wow duh you literally just answered my question..
13:59 andrewSC
gotcha gotcha
13:59 pdurbin
yeah, the dev environment
14:00 pdurbin
It would be nice to streamline the onboarding process for new developers. Ideas are welcome.
14:00 andrewSC
mmmmmmmmmmmm
14:00 pdurbin
andrewSC: so the other languages you're considering are PHP and C#? Anything else?
14:01 donsizemore
@andrewSC the perl you write today will still run flawlessly in 20 years. just sayin
14:01 poikilotherm
Doesn't depend the language to choose on the project needs?
14:01 andrewSC
well, I've been suggested PHP and C# but idk if i'm keen on them as much ;) python has been strong in my mind, java as well.
14:02 andrewSC
poikilotherm: it does! hence the "right tool for the job" statement i made earlier
14:02 poikilotherm
E. g. writting a kernel driver is just impossible with Java...
14:02 andrewSC
basically something to serve an api since the project will have two interfaces, a phone app and a website
14:02 andrewSC
poikilotherm: right lol
14:03 andrewSC
donsizemore: lol
14:04 andrewSC
obv any lang in that list "works" what i'm thinking is longer term, maintainability, scalabiltiy. all those sorts of things
14:04 andrewSC
also static typing..
14:04 andrewSC
;)
14:05 poikilotherm
If you are into Business Intelligence, Java is the definite way to go for Web Devs
14:05 poikilotherm
There is a ton of projects that help you get things done with that
14:06 poikilotherm
Those don't exist in other "language worlds" or are clumbsy to get baked in
14:06 poikilotherm
E.g. using stuff for BPML is way easier in Java than in Python/PHP /... (dunno for C#)
14:07 poikilotherm
If you are just in REST, there are plenty of tools and frameworks for any kind of language out there...
14:08 poikilotherm
For me the language choice is a matter of human resources and what kind of libs/... exist to make devs life easier and get thing out of the door.
14:08 andrewSC
poikilotherm: mhmmm right right
14:10 andrewSC
i prefer python since it's mature enough however i'm concerned about rest api perf long term as well as not having robust static typing
14:11 andrewSC
the latter is something i've come to know I'll want long term/as the project grows
14:13 andrewSC
there's also considerations of marketability and being able to hire realtively easily if it were to get to that point
14:13 andrewSC
anyways /end ot discussion lol
14:13 andrewSC
thanks for the thoughts :)
14:13 poikilotherm
Just give Django a look ;-)
14:13 andrewSC
;)))
14:14 poikilotherm
That is a very mature and well-known solution with a lot of attracation for devs
14:14 andrewSC
mhmmm
14:14 poikilotherm
But keep in mind that you will need the client side, too
14:14 poikilotherm
And Vue.js / Angular / React etc are a totally different story
14:15 andrewSC
right i think i have settled on client side stack for the web portion
14:15 andrewSC
mhmmmmm
14:15 poikilotherm
Most of the time you will need at least two devs, one for front and one for backend...
14:15 andrewSC
right...
14:15 poikilotherm
Most of the devs are good on only one side of the story
14:34 pdurbin
andrewSC poikilotherm donsizemore: for what it's worth, I wrote our new metrics aggregator in Python: https://github.com/IQSS/metrics.dataverse.org
14:36 Mahsa joined #dataverse
14:38 Mahsa
Hi all, I have a quick question about handles. We are using handles for our dataverse. We would like to use our local handle server and not the Global Handle Registry. So my quick Question is
14:39 pdurbin
Mahsa: hi! Is what? :) I think you were cut off.
14:39 Mahsa
Is it only a databse field value change to replace the default ( https://hdl.handle.net/) for the hanldes with sth else,
14:40 Mahsa
hi :-) sorry, was still typing when hit enter
14:41 Mahsa
So by default dataverse creates handles based on this address: https://hdl.handle.net/
14:41 pdurbin
Mahsa: I'm looking at http://guides.dataverse.org/en/4.9.4/installation/config.html#configuring-dataverse-for-handles and I can't figure out where to set the hostname for your Handle server.
14:41 Mahsa
Yes, exactly, That is my question
14:42 Mahsa
I am wondering if it is in the DB somewhere that we can change
14:42 Mahsa
?
14:43 pdurbin
Mahsa: sure, Can you please create a GitHub issue about how "https://hdl.handle.net " is hard coded at https://github.com/IQSS/dataverse/blob/v4.9.4/src/main/java/edu/harvard/iq/dataverse/HandlenetServiceBean.java#L396 ?
14:43 Mahsa
Sure, I will just go ahead and create one then
14:43 Mahsa
Thanks for redirecting me to it
14:44 pdurbin
sure thing
15:02 pdurbin
nice, created already: https://github.com/IQSS/dataverse/issues/5137
15:06 pdurbin
Now we can get back to talking about languages and front end frameworks. :)
15:11 jri joined #dataverse
15:56 poikilotherm joined #dataverse
16:28 jri joined #dataverse
17:29 donsizemore joined #dataverse
18:20 pdurbin
donsizemore: question about https://github.com/IQSS/dataverse/issues/3205
18:41 donsizemore
@pdurbin question about https://github.com/IQSS/dataverse/issues/4593!
18:41 donsizemore
but you first
18:43 pdurbin
donsizemore: can you please check with Thu-Mai if Data Explorer already gives her the feature she's asking for? And if not, does she think that it's something Data Explorer could do in the future? And if so, can she create and issue on the Data Explorer side so we can close the issue on the Dataverse side?
18:43 pdurbin
NESSTAR import question? go for it :)
18:45 donsizemore
@pdurbin i started with the 3.n migration document before i found that issue. in true danger-prone don form, i'm not getting very far: {"status":"ERROR","code":404,"message":"'/api/v1/batch/migrate/' endpoint does not exist on this server.
18:45 donsizemore
@pdurbin that's on 4.8.6, trying against localhost:8080 and the FQDN , passing the admin account's API token either as a key or an HTTP header
18:48 jri joined #dataverse
18:50 Jim__ joined #dataverse
18:52 donsizemore
@pdurbin also, i passed the note along to thu-mai
18:54 pdurbin
thanks! so you're following the curl command at the top of that issue? or something in the api guide?
18:55 donsizemore
both/either
18:56 pdurbin
what are you trying to import? a NESSTAR dataset?
18:57 donsizemore
11 dataset files in codebook 2.5 XML . pointing it at the directory with a ?dv=alias as well. perplexed that dataverse is throwing a 404
18:57 pdurbin
Huh. Do we support codebook 2.5? I have no idea.
18:58 pdurbin
DdiExportUtil is telling me that we export DDI 2.5, at least
18:58 pdurbin
I don't know about import.
18:59 pdurbin
I thought the version we supported for import from DVN 3 was a lower number.
19:00 pdurbin
We imported DDI 2.0 according to scripts/issues/907/batchImportDv/version1.xml which I used in a test: https://github.com/IQSS/dataverse/blob/v4.9.4/src/test/java/edu/harvard/iq/dataverse/api/BatchImportIT.java#L64
19:01 pdurbin
If my commit messages can be trusted, I didn't get that test working again until 4.9: https://github.com/IQSS/dataverse/commit/b8090f0
19:01 donsizemore
would that cause a 404?
19:02 donsizemore
oh, wait. this test server is running 4.9.
19:02 donsizemore
^^ 4.9.4
19:04 pdurbin
hmm, I'm getting the same error: "message": "'/api/v1/batch/migrate/' endpoint does not exist on this server
19:04 pdurbin
when I run `mvn test -Dtest=BatchImportIT` on a server running 4.9.4
19:04 donsizemore
*phew*
19:04 pdurbin
Expected: WORKFLOW_IN_PROGRESS
19:04 pdurbin
Actual: ERROR
19:04 Jim__
@pdurbin - w.r.t. #4816 - not sure what you want. The updates to the formats were part of #4777 and it's follow-on fixes that went out. Do you want to review those changes to close this issue out? Or?
19:04 pdurbin
T-T
19:05 pdurbin
Jim__: hi! Sorry to confuse you. It's a process thing. I don't test or close issues. That's QA's job. But I can move it to QA. Does that make sense?
19:06 Jim__
Yeah - afaik there's nothing more to do...
19:08 donsizemore
@pdurbin it's a 4.9 thing: [dlshttp://localhost:8080/api/batch/migrate {"status":"ERROR","message":"Please provide a key query parameter (?key=XXX) or via the HTTP header X-Dataverse-key"}
19:08 pdurbin
Jim__: great! I moved it over and added "connects to" so QA can find it. QA looks for the associated pull request on the card in Waffle. Does that make sense?
19:09 donsizemore
@pdurbin so i just need to pick on a test server <= 4.8.6
19:09 pdurbin
donsizemore: you're saying that the "migrate" API stopped working or moved or something from one version to another?
19:10 donsizemore
@pdurbin from this end, the migrate API throws a 404 in 4.9
19:10 pdurbin
yeah and a 404 also for 4.9.4 for me
19:11 donsizemore
[dlshttp://localhost:8080/api/batch/migrate {"status":"ERROR","code":404,"message":"'/api/v1/batch/migrate' endpoint does not exist on this server.
19:11 pdurbin
but you're saying it works in 4.8.4, sounds like
19:11 donsizemore
it exists on 4.8.6 (i just have to stand up a 4.8.6 test server)
19:11 pdurbin
not great for APIs to disappear :(
19:11 donsizemore
well, hopefully most folks have migrated from 3.6 by now =)
19:12 pdurbin
hopefully but https://github.com/IQSS/dataverse/issues/4996 is still open
19:14 donsizemore
@pdurbin would you like for me open an issue?
19:15 donsizemore
i don't need it for me (the target server in this cause is still running 4.8.6) but unless it was removed intentionally...
19:15 pdurbin
donsizemore: yes please
19:46 donsizemore joined #dataverse
19:47 donsizemore
@pdurbin back - a fire alarm sent us scurrying to the coffeeshop
19:47 donsizemore
@pdurbin also, i just got my demo.dataverse account from martin fenner so i'll stick that in dataverse-ansible and get demo publishing working again
19:50 pdurbin
donsizemore: you probably shouldn't
19:51 pdurbin
shouldn't it be treated as a secret and not checked in to the code?
19:52 pdurbin
donsizemore: I'm saying that based on "DataCite is happy to give credentials to the DataCite test system to everyone who asks for that, but we would not be happy with a standard test account and no password as EZID is doing." https://github.com/IQSS/dataverse/issues/5024#issuecomment-423610278
19:52 donsizemore
@pdurbin test DOIs will be deleted 30 days out
19:53 donsizemore
@pdurbin i was wondering about that, but jon suggested i just stick the values in the role and run with it. i can ask martin for clarification
19:53 donsizemore
@pdurbin regarding #3205, thu-mai sez "yes, data explorer does give us what i wanted"
19:53 pdurbin
if you would. thanks!
19:54 pdurbin
oh! great! I guess I'll close that issue then. thanks!
20:01 donsizemore
@pdurbin one in, one out ;)
20:03 pdurbin
heh. I'm leaving a comment on the one you just put in (thanks): https://github.com/IQSS/dataverse/issues/5138
20:13 donsizemore
@pdurbin martin sez don't publish credentials so i'll stick vars in and update the README usage
20:17 pdurbin
donsizemore: perfect. Now how do I get the creds from my ec2 script to your ansible code?
20:18 pdurbin
moar extra-vars?
20:18 donsizemore
--extra-vars
20:19 pdurbin
jink
20:19 pdurbin
jinx
20:19 donsizemore
jinkies!
20:19 pdurbin
heh, I was just googling how to spell that
20:20 pdurbin
donsizemore: am I going to be able to choose between EZID and DataCite with vars?
20:20 pdurbin
I'll just go look.
20:21 pdurbin
it seems like we'd have to unnest more vars :/
20:22 pdurbin
for what it's worth a friend says unnesting isn't necessary: http://irclogs.shortcircuit.net.au/%23crimsonfu/2018-09-21.log.html and http://irclogs.shortcircuit.net.au/%23crimsonfu/2018-09-22.log.html but who knows
20:29 donsizemore
@pdurbin absolutely. i don't mind unnesting vars, as you know i'm no purist. i want stuff to work so i can go to the gym =)
20:35 pdurbin
donsizemore: meh, please just leave it.
20:35 pdurbin
I have some test DataCite credentials I can send.
20:36 pdurbin
It'll be a bit of a pain to go around to everyone's laptop and put them on.
20:36 pdurbin
It was nice how with EZID we could just have the creds checked into the code.
20:39 pdurbin
I never know if you're going to talk about going to the gym or having margaritas by the pool.
20:40 pdurbin
now this is what I call an installation: https://www.harvard.edu/inauguration-lawrence-s-bacow/installation-schedule-events
22:11 donsizemore joined #dataverse
22:49 jri joined #dataverse
