Time
S
Nick
Message
00:21 axfelix joined #dataverse
02:53 axfelix joined #dataverse
02:56 cnk joined #dataverse
03:37 garnett joined #dataverse
04:23 kzisme joined #dataverse
04:23 kzisme joined #dataverse
04:44 garnett joined #dataverse
07:18 jri joined #dataverse
07:50 bencomp joined #dataverse
12:33 jri
Hi all, This week I'm making the production cluster for our dataverse \o/ I just want to know if the solr index becomes inconsistent, if there a "button" or a command to recreate the index ?
12:34 jri
And if the solr is really essential or if it could be down for some minutes ?
12:36 jri
(because creating a real cluster for solr means deploy most machines which is a little overwhelming for our usage)
12:41 pdurbin
\o/
12:41 pdurbin
jri: have you already read http://guides.dataverse.org/en/4.3/installation/administration.html#solr-search-index ?
12:41 jri
pdurbin: oops
12:42 jri
I miss this one
12:42 pdurbin
it's new
12:42 jri
:)
12:43 jri
It's what I need then
12:43 pdurbin
jri: here's a pull request about fancy Solr cluster stuff: https://github.com/IQSS/dataverse/pull/2985
12:43 jri
Do you use SolrCloud at Harvard ?
12:44 jri
Cause I was looking for something similar to this: http://guides.dataverse.org/en/4.3/_images/3webservers.png
12:44 pdurbin
yeah. cold spare. that's what we do
12:49 jri
So it's not SolrCloud ?
12:49 pdurbin
jri: no, we don't use SolrCloud but that's exactly what the pull request above is about. See also https://groups.google.com/d/msg/dataverse-community/0Jp-hhNdM38/g1e3FJA5EQAJ
12:51 jri
I think I'll make two solr node with replication and failover, and in the case of master fail and slave handle writes :/
12:51 jri
SolrCloud will be an option if we grow up
12:52 pdurbin
jri: maybe you could contribute some docs on what you set up
12:52 jri
Interesting Pull Request and discussion
12:53 pdurbin
jri: there's a very detailed Google Doc about the pull request at https://docs.google.com/document/d/1tfYtmoKPy-Ci7s6TOLa-PG-Ez9K_BQB5gGYKCAHbFO8/edit?usp=sharing
12:59 jri
pdurbin: oh thanks! All the new stuff about solrCloud like SSL, kerberos are very interesting. I keep that in bookmarks, I am tempted to do an unreasonable cluster just to play with this...
13:00 pdurbin
heh
13:00 jri
But dataverse is our only use case with solr :(
13:00 pdurbin
jri: you could build a war file based on that pull request and give the author feedback. I'm sure he'd appreciate it.
13:02 jri
If I can find time after set up my little cluster, why not :) it could be fun
13:03 pdurbin
I'm too snowed under to look at it. :(
13:05 jri
Arf I know that...
13:07 pdurbin
arf
13:07 pdurbin
:)
13:10 pdurbin
jri: if you want I can add you to an internal support ticket where someone is trying to help the author of that pull request.
13:12 jri
Hmmm, is possible to just be a kind of follower for now ? because I'm afraid to have nothing to said for help before testing it
13:12 pdurbin
jri: you could click "subscribe" on the pull request
13:13 jri
Ah! got it
13:14 jri
I'll follow that and participate it i've something to say
13:15 jri
pdurbin: you helped me greatly again :) thanks so much
13:16 jri
for your time
13:21 pdurbin
oh sure
13:21 pdurbin
:)
13:34 bmckinney joined #dataverse
13:48 pdurbin
bmckinney: mornin. hope I didn't miss much yesterday
13:49 bmckinney
pdurbin: nope, not a lot to report really
13:50 bmckinney
pdurbin: i'm blocked on integration testing!
13:50 pdurbin
rest assured :)
13:50 bmckinney
simple setup and teardown testing using native apis
13:56 bmckinney
in teardown i delete the test dataset and it gives me success: { "status": "OK", "data": { "message": "Dataset 6 deleted" } }
13:56 bmckinney
i attempt to delete test dataverse and it tells me it's still there: { "status": "ERROR", "message": "Cannot delete non-empty dataverses: Root Dataverse, testDv Dataverse, Darwin's Finches, " }
13:56 bmckinney
still in the db
13:57 bmckinney
strange
13:57 pdurbin
!
13:57 pdurbin
sounds like the "delete dataset" api endpoint is lying to you
13:59 pameyer joined #dataverse
14:00 bmckinney
yeah, seems like there are multiple ways to delete and i've tried them all (by id, by version, using "destroy")
14:01 pdurbin
you have to use "destroy" if the dataset has been published
14:01 bmckinney
they all tell me they worked but not actual commit of the transaction
14:02 bmckinney
ah, another good reason for swagger - many apis are a dark art
14:02 pameyer
"swagger"?
14:02 bmckinney
only known via oral tradition
14:02 pameyer
gotcha
14:03 pdurbin
are we allowed to link to the swagger as sbgrid?
14:03 bmckinney
pameyer: see: https://dv.sbgrid.org/swagger/index.html
14:03 bmckinney
pdurbin: don't understand the question
14:05 pdurbin
I thought pameyer was keeping that instance under wraps. This channel publicly logged.
14:06 bmckinney
adding swagger to the project is the easy part, annotating the APIs with useful info (where there currently is none) is the hard part
14:06 pameyer
it's externally visible - just don't want anything thinking it's live before it is
14:07 pdurbin
at least it says "Beta: for testing only" at the top. :)
14:07 pdurbin
should be enough
14:08 pdurbin
pameyer: if you want you can make it so that when you click the warning more information appears in a popup like we do at https://demo.dataverse.org
14:08 pameyer
pdurbin: not a bad idea
14:09 bmckinney
pdurbin: dare i ask, docs for that?
14:10 pdurbin
bmckinney: :StatusMessageText has not been documented. -- https://github.com/IQSS/dataverse/issues/2944
14:12 pdurbin
should work though
14:15 bencomp1 joined #dataverse
14:19 axfelix joined #dataverse
14:38 bmckinney joined #dataverse
14:57 bmckinney26 joined #dataverse
14:59 axfelix joined #dataverse
15:05 jri
pdurbin: me again... I'm at the install of Dataverse, but the install script is not ready to a install into a glassfish cluster :/ How do you do at harvard ? Have you three glassfish standalone servers and apply config on each one ?
15:06 jri
Or do you deploy and configure dataverse in the cluster with hands (without the script)
15:06 jri
?
15:17 pdurbin
jri: uh... I'd say that's an "advanced installation" ( http://guides.dataverse.org/en/4.3/installation/prep.html#advanced-installation ) and that you should email support
15:23 jri
pdurbin: yes it is
15:23 jri
People behind the mailing list are also sysadmins ?
15:26 pdurbin
jri: if you email supporthttps://help.hmdc.harvard.edu which will be looked at by the people who help run https://dataverse.harvard.edu
15:26 jri
Ah! cool!
15:26 pdurbin
I wouldn't call that a mailing list though. It's a ticketing system.
15:26 jri
Ok
15:27 jri
I'll try to get some info through this way
15:27 pdurbin
the ticket should find it's way to people who know how to explain what to do
15:27 jri
Ok :)
15:27 jri
Good to know
15:28 jri
I was thinking that was people who are in this chan
15:29 pdurbin
nope. the two people I'm thinking of aren't in this channel
15:29 jri
ok :)
15:35 bmckinney26
pdurbin: so i think i see the problem, DeleteDatasetVersionCommand fails to delete related metadatablocs/datasetfields
15:36 pdurbin
bmckinney26: can you please open an issue about this? This happens in 4.3 I assume.
15:37 bmckinney26
somewhat surprising that this wasn't discovered since one of the search tests includes metadatablocks (scripts/search/tests/data/dataset-finch1.json)
15:38 pdurbin
bmckinney26: sorry, which search test?
15:38 bmckinney26
pdurbin: not sure, i'm just borrowing the json for my test
15:38 pdurbin
oh
15:38 pdurbin
I do feel like maybe I've seen this bug before. Not sure.
15:39 bmckinney26
used in create-all-and-test.sh, UtilIT, DdlExportUtilTest
15:40 bmckinney26
no setup/teardown apparently
15:41 pdurbin
scripts/search/tests/delete-all-and-test works. I was just using it yesterday
15:41 bmckinney26
and you remove the dataset?
15:41 pdurbin
yep. "all" :)
15:43 bmckinney26
is see you revoking a role but issuing no delete operation on the dataset
15:43 bmckinney26
sloppy!
15:44 bmckinney26
;)
15:44 pdurbin
bmckinney26: you don't see the deletes of the dataset? see the big DELETE? :)
15:45 pdurbin
https://github.com/IQSS/dataverse/blob/v4.3/scripts/search/tests/delete-all-and-test
15:45 pdurbin
one dataset is deleted via SWORD. the other is deleted via the native api
15:45 pdurbin
so not sloppy :)
15:45 pdurbin
you are mistaken :)
15:46 bmckinney26
whoops, looking at wrong script - my bad!
15:47 bmckinney26
create-all-and-test uses finch1.json with no delete
15:49 pdurbin
right. because I want those datasets to stick around so I have some test data to play with
15:50 pdurbin
you can see them at http://phoenix.dataverse.org (which calls the same script)
15:50 bmckinney26
can you point to where the datasetfields get zapped?
15:50 bmckinney26
maybe there is no bug and i'm missing something?
15:50 pdurbin
buh. probably the command you mentioned
15:53 bmckinney26
let's discuss thursday, i'll remove the offending metadata blocks in my test json so i can keep moving for now...
15:54 pdurbin
ok
15:55 pdurbin
I'm surprised that your new metadata block is causing trouble. Maybe it needs to be fixed up a bit.
15:56 bmckinney26
Not mine really, just reusing dataset-finch1.json
15:57 pdurbin
oh! huh
15:58 pdurbin
yeah, those scripts work fine on phoenix. not sure what's different on your machine
16:01 cnk joined #dataverse
16:21 pdurbin
bmckinney26: speaking of the JSON used to create datasets, I just left this comment: https://github.com/IQSS/dataverse/issues/3060#issuecomment-205873515
16:37 cnk joined #dataverse
17:00 cnk joined #dataverse
17:08 jri joined #dataverse
17:40 bmckinney26 left #dataverse
18:17 garnett joined #dataverse
18:52 bmckinney joined #dataverse
19:16 axfelix joined #dataverse
19:54 bmckinney
pdurbin: mystery solved - i was using a test version of persistence.xml for my arquillian integration tests and the doidataciteregistercache table+sequence was missing
19:55 bmckinney
i had commented out the ddl-generation property in persistence.xml
19:55 pdurbin
ah
19:58 bmckinney
strange that it would silently break some things and not others, but there you go
20:08 bmckinney left #dataverse
20:17 pdurbin
it moves in mysterious ways
20:54 bencomp joined #dataverse
21:01 axfelix joined #dataverse
23:01 cnk joined #dataverse
23:15 garnett joined #dataverse
