Time
S
Nick
Message
08:16 jri joined #dataverse
08:22 poikilotherm joined #dataverse
08:58 MrK joined #dataverse
09:20 stefankasberger joined #dataverse
09:24 Youssef_Ouahalou joined #dataverse
09:26 Youssef_Ouahalou
Hello everyone :)
09:26 poikilotherm
Morning
09:29 Youssef_Ouahalou
I need to delete items in the subject list of the citation metadata, I have to do it directly in the tsv file?
09:30 poikilotherm
You can create a copy and delete them there. Anyway: you will need to update your database when you want to modify an existing installation
09:32 poikilotherm
I'm not sure if reloading the TSV file with an empty vocabulary would do that for you
09:32 poikilotherm
And I believe that no existing metadata records will be updated
09:32 poikilotherm
No idea what happens when you do that - unsure if this is test covered... ;-)
09:35 Youssef_Ouahalou
@poikilotherm: Thank you so much for your answer
09:36 Youssef_Ouahalou
do you know the file I need to edit in the database?
09:37 poikilotherm
It's no file in the database. When loading the TSV via an API call, it is processed and stored as a proper database structure
09:37 poikilotherm
Let me do a quick check on the code
09:39 Youssef_Ouahalou
And you're right I updated the file with this command and nothing was changed . curl -H "X-Dataverse-key:*****" -X POST -H "Content-type:application/json" -d "[\"citation\"]" http://localhost:8080/api/dataverses/:root/metadatablocks
09:40 Youssef_Ouahalou
I send you the complete file?
09:49 poikilotherm
Ok looking at http://github.com/poikilotherm/dataverse/blob/105e62fb13368c496789002b3974ec630188b92c/src/main/java/edu/harvard/iq/dataverse/api/DatasetFieldServiceApi.java#L376-L416 it seems like you can only ADD or UPDATE existing records, but not delete old ones.
09:50 poikilotherm
This is not an idempotent API , so you cannot describe the state you want...
09:50 poikilotherm
As far as I can see, there is no delete endpoint
09:51 poikilotherm
If you feel like it, you could extend the API to be able to delete single fields or even delete single vocabulary values
09:52 poikilotherm
At least you could create an issue on Github, describing your need
09:53 poikilotherm
If you create a new installation, you can just modify the file and let setup load your customized citation file
09:54 poikilotherm
Just make sure you don't load an upstream changed file during upgrade... Otherwise you will be in trouble ;-)
10:01 MrK
Metadata is append only, you can only update or create new.
10:06 Youssef_Ouahalou
Do you think that if I reinstall with my citation file modify it should make the changes I would like?
10:08 MrK
Yeah, if it's correctly modified.
10:09 Youssef_Ouahalou
Okay thank you so much :)
10:27 stefankasberger
@j
10:29 stefankasberger
@pdurbin: I have a question regarding the API and Waiting Times for Requests. I do now a mass import with some bigger files (100MB SAV files). It seems, as during the ingest/file conversion, the API does not work anymore. This is the concrete issue, but in general it would be nice to know, how to handle waiting times for all kind of requests (delete datasets, create datasets, etc.) and how to work with the API the best.
10:29 stefankasberger
Is there for example a request method, to check if the API is working, or do you recommend static waiting times?
11:32 pdurbin
Youssef_Ouahalou: before modifying citation fields like "subject", you should read the warning that starts with "Generally speaking it is safer to create your own custom metadata block rather than editing metadata blocks that ship with Dataverse, because changes to these blocks may be made in future releases of Dataverse."
11:33 pdurbin
at http://guides.dataverse.org/en/4.18.1/admin/metadatacustomization.html#introduction
11:33 pdurbin
Youssef_Ouahalou: I think you should create a GitHub issue about the pain point you have. You aren't the first who has wanted to remove subjects but I'm not sure if there is a clear issue tracking this.
11:36 pdurbin
stefankasberger: for ingest I recommend checking for locks in a loop until it has cleared. Something like this: https://github.com/IQSS/dataverse-sample-data/blob/ca7eca8d93da42ca1735551001684b34cc9a6b6b/create_sample_data.py#L14
11:41 pdurbin
poikilotherm: nothing in code review or QA. How's your pull request coming? :) Need any help?
11:41 pdurbin
MrK: any pull requests you're working on? :)
11:41 poikilotherm
pdurbin: hacking on the docs since this morning
11:41 poikilotherm
Stay tuned
11:45 pdurbin
\o/
11:57 poikilotherm
pdurbin: I believe that http://guides.dataverse.org/en/latest/installation/oauth2.html contains some wrong advice
11:57 poikilotherm
"After restarting Glassfish you should see the new provider under “Other options” on the Log In page,"
11:57 poikilotherm
IIRC you don't need to restart
11:57 poikilotherm
The API call is triggering the factories to load the providers
11:58 poikilotherm
Do you want to leave that inside to be completely sure or should we change that?
11:58 poikilotherm
I would regret to write this in the OIDC docs...
11:59 poikilotherm
I fetch metadata when the provider is loaded, so when you provide a wrong configuration, you will instantly see messages in the logs, not just after a restart
12:01 pdurbin
poikilotherm: hmm, up to you. I'd say maybe create an issue about it and leave it for later.
12:27 MrK
pdurbin: Well yeah but in my repo :P
12:30 pdurbin
T-T
12:45 MrK
As I said before it's impossible to merge :<
12:52 donsizemore joined #dataverse
12:52 poikilotherm
pdurbin: I lost track... is there a new template for PRs?
12:53 poikilotherm
pdurbin: we discussed about describing scope, things to test etc etc
12:53 poikilotherm
pdurbin: https://github.com/IQSS/dataverse/issues/6331 did not see more updates...
12:54 poikilotherm
Oh sorry, I meant https://github.com/IQSS/dataverse/issues/6226
12:57 poikilotherm
I'd be happy to test following https://github.com/IQSS/dataverse/blob/728106c53909f3b813476e257624786487c380e0/.github/PULL_REQUEST_TEMPLATE.md
12:59 pdurbin_m joined #dataverse
12:59 pdurbin_m
poikilotherm: sure, please try the new template
13:00 pdurbin_m
MrK: you've reached the point of no return? T-T
13:00 MrK
pdurbin_m: Yeah, so time ago.
13:01 pdurbin_m
MrK: do you still call it Dataverse? :)
13:01 MrK
pdurbin_m: Yeah :P.
13:04 poikilotherm
MrK: too bad... :-(
13:05 MrK
This year was mostly refactoring.
13:05 poikilotherm
Great!
13:05 poikilotherm
I also did a load of refactoring
13:06 poikilotherm
But I really insisted on getting it upstream... ;-)
13:07 MrK
The problem is that we would have to wait for pr's and we wouldn't be able to control the quality :P and I'm pretty hard on that.
13:07 MrK
As well as the team tbh.
13:07 poikilotherm
Well pdurbin, djbrooke and kcondon are really trying to get us moving fast
13:08 poikilotherm
Especially our chief community officer pdurbin is doing a very good job on this
13:08 poikilotherm
But I hear you
13:09 poikilotherm
It's just that it would be great to get things moving together.
13:09 MrK
Yeah, I know.
13:09 poikilotherm
Community pushing forward with more experts should at least in theory have a huger impact
13:09 poikilotherm
hugher impact
13:13 pdurbin_m joined #dataverse
13:15 pdurbin_m
MrK: I'm definitely interested in learning better practices from you and your team.
13:15 pdurbin_m
I hope we can improve the quality of the code together.
13:16 pdurbin_m
poikilotherm: thank you for working so hard to get your refactoring merged upstream.
13:20 pdurbin_m
MrK: any interest in getting your installation on the map at https://iqss.github.io/dataverse-installations/ at least?
13:22 pdurbin_m
donsizemore: are we on for this afternoon? Guess what. We have Zoom now. I hear.
13:28 pdurbin_m
MrK: I'm curious why you (or whoever) decided to build from Dataverse instead of starting from scratch. What are the significant features? If that makes sense.
13:29 donsizemore
@pdurbin_m sounds good to me. i've been doing a bunch of odum-stuff, tho
13:31 MrK191 joined #dataverse
13:31 pdurbin_m
MrK191: ^^ :)
13:32 pdurbin_m
donsizemore: how dare you. Any thoughts on the next thing to do for automated testing?
13:32 pdurbin_m
poikilotherm: same question :)
13:33 poikilotherm
pdurbin_m: we didn't want to reinvent the entire wheel
13:33 poikilotherm
Too much repo software out in the wild
13:34 donsizemore
@pdurbin_m i was on the jacoco train
13:34 poikilotherm
The ability in Dataverse to create custom metadata, creating hierarchies and being written in Java were three very important points ;-)
13:34 MrK191
pdurbin_m: As soon as the installation will be public I will contact you about the map.
13:35 MrK191
pdurbin_m: The
13:35 pdurbin_m
poikilotherm: by hierarchies you mean dataverses within dataverses?
13:35 pdurbin_m
MrK191: thanks!
13:35 MrK191
application had requirement to start from some other project, and there was a lot of features which we needed.
13:35 stefankasberger
@pdurbin: Thanks!
13:36 pdurbin_m
MrK191: interesting requirement :)
13:36 pdurbin_m
stefankasberger: I forget if checking for locks is in pyDataverse or not.
13:37 pdurbin_m
donsizemore: good train to be on, I'd say. :)
13:38 stefankasberger
No, it was not.
13:38 stefankasberger
But is the lock related to a Dataset, or in general for the API ?
13:39 stefankasberger
its realted to the dataset as i see. and thats the most reasonable, prefered way i guess, right?
13:39 pdurbin_m
stefankasberger: locks are at the dataset level and there are various types.
13:40 pdurbin_m
donsizemore: did you see that the new R maintainer wants better automated tests? I thought maybe we could help.
13:41 pdurbin_m
donsizemore: https://github.com/IQSS/dataverse-client-r/pull/28#issuecomment-557936232
13:42 poikilotherm
pdurbin_m: I just booked flight and hotel for Norway. It's a pity you can't make it...
13:52 pdurbin
poikilotherm: yeah, I tried to get invited but failed. :) Oh well. :)
14:09 bricas joined #dataverse
14:19 bricas joined #dataverse
14:27 pdurbin
stefankasberger: I got the email you sent to me and donsizemore. I'm a little confused about Data Pipeline. That is to say, I'm not sure I understand the use case. What problem does it solve? :)
15:43 poikilotherm
pdurbin: just before heading home / picking up kids I created https://github.com/IQSS/dataverse/pull/6433 for you
15:44 pdurbin
poikilotherm: thanks!
15:44 poikilotherm
I'd love to see a review, dunno if you want to add more people in the mix of reviewers.
15:55 stefankasberger
It is just stitching together the pieces of a data migration project. it connects the api module with the data models with the data import and OAISTree.
15:55 stefankasberger
Its the application of the module for a whole data migration project, from data import to dataverse deletions.
15:57 stefankasberger
@pdurbin: when was the locks endpoint added to Dataverse?
16:00 pdurbin
stefankasberger: ok, so it's all for data migration. Perfect. Thanks. Let me go look.
16:01 pdurbin
stefankasberger: the locks endpoints were added in Dataverse 4.9.3 in https://github.com/IQSS/dataverse/pull/4986
16:21 stefankasberger
@pduring: what would be the recommended way to check api availability before 4.9.3?
16:22 stefankasberger
Regarding data migration: maybe i can show this in a community call?
16:25 pdurbin
stefankasberger: since Dataverse 4.0 you can check for a "locked" boolean via SWORD: http://guides.dataverse.org/en/4.18.1/api/sword.html#display-a-dataset-statement
16:26 pdurbin
And sure, we're always looking for topics for community calls! More people tend to call in when there's actually something on the agenda. :)
16:41 stefankasberger
great, thanks.
16:43 pdurbin
donsizemore: on the agenda for 3pm is getting off Glassfish 4 :)
18:03 donsizemore
@pdurbin yes yes yes!
20:01 poiki-mobile joined #dataverse
20:02 poiki-mobile
Ola pdurbin
20:02 poiki-mobile
I just saw as that my PR was already merged...
20:02 poiki-mobile
O.O
20:03 poiki-mobile
Will post some answers to your questions tomorrow...
20:03 poiki-mobile
I still shocked this went through so fast...
20:03 poiki-mobile
+am
20:05 pdurbin
Oh! Merged already? Really? :)
20:09 pdurbin
poiki-mobile: I just changed the title of the pull request at least :)
20:12 poiki-mobile
Thx. I was in a hurry earlier...
20:12 pdurbin
so was QA apparently :)
20:12 poiki-mobile
kcondon and you seem pretty confident my stuff is working ok :-D
20:13 poiki-mobile
I can barely believe kcondon tested it with Google...
20:13 poiki-mobile
Will do that tomorrow anyway, had no chance today
20:14 poiki-mobile
About that question on testing... Do you think you guys be cool with adding things like Testcontainers based integration tests?
20:15 poiki-mobile
I do see potential for more of these... and that current stuff would be excellent for testing and learning about it
20:16 poiki-mobile
You'd get more shiny numbers for test coverage... ;-)
20:22 pdurbin
poiki-mobile: sure! Speaking of testing Gustavo and I are doing a Zoom call with donsizemore right now about automated testing. And moving off Glassfish.
20:22 pdurbin
Your name just came up. You'd be welcome to join in. :)
20:22 donsizemore
@poiki-mobile https://unc.zoom.us/my/sizemore
20:22 poiki-mobile
Meh. Located at construction site right now
20:22 poiki-mobile
I'd love to join, but my data plan...
20:23 pdurbin
heh
20:40 pdurbin
donsizemore: if you want, please just send me a CSR like last time.
20:40 pdurbin
poiki-mobile will be excited about the hostname, I suspect :)
20:41 poiki-mobile
Hum?
20:52 pdurbin
I wouldn't want to spoil the surprise.
20:52 pdurbin
how non-SLOPI of me
20:52 pdurbin
donsizemore: hey, I just added you to cc on an email about OCR . How was that call today anyway? I've been in and out a lot today.
21:23 poiki-mobile
pdurbin did you guys come to a conclusion about Glassfish vs Payara vs some other server?
21:26 pdurbin
let's just say I don't expect the hostname to have glassfish in it
21:32 poiki-mobile
LOL
21:33 * poiki-mobile
watches his inbox
21:37 poiki-mobile
Did you guys talk about how to gather things we all want changed somehow and how to tackle everything down?
21:42 pdurbin
uh, what things?
21:42 pdurbin
What do you want to change?
21:45 poiki-mobile
Many things.
21:46 pdurbin
It was a pretty high level and quick discussion.
21:46 poiki-mobile
Dependcy cleanup for example
21:46 pdurbin
nope, didn't come up
21:46 poiki-mobile
Resource creation
21:46 pdurbin
nope
21:46 poiki-mobile
Jakarta EE
21:47 pdurbin
we want that
21:47 poiki-mobile
Doing the breaking change with 9 seems like a good idea in one go... Do all the hard things now, don't pester people too much
21:48 poiki-mobile
It's going to be though for the next year at least...
21:48 poiki-mobile
Plus Java 11...
21:49 pdurbin
You *do* want a lot of changes. :)
21:51 poiki-mobile
Yes
21:51 poiki-mobile
it's gonna hurt
21:51 poiki-mobile
But the pain is coming anyway...
21:52 poiki-mobile
IMHO we should create some kind of roadmap in whatever way ;-)
21:59 poiki-mobile
Forgive me if I sound like a pushy prank... It's just me trying to get things moving for more than a year and many things moved in the eco system in the mean time ;-)
21:59 donsizemore
they're all ansible group_vars to me ;)
22:00 poiki-mobile
donsizemore: LOL
22:01 poiki-mobile
donsizemore: I feel you... for me it's just a tag patch away ;-)
22:01 donsizemore
@pdurbin we indeed talked about Melissa Dell!
22:28 poiki-mobile
Alright guys, I'm going to switch off the lights now and head for my bed for the day... Read you tomorrow
23:33 xarthisius joined #dataverse
23:33 xarthisius joined #dataverse
