Time
S
Nick
Message
06:41 Slava1 joined #dataverse
07:32 jri joined #dataverse
07:55 Slava1 joined #dataverse
10:17 Slava1 joined #dataverse
11:58 Slava1 joined #dataverse
13:31 Sherry joined #dataverse
13:31 Sherry
Is demo.dataverse down for a reason?
13:32 pdurbin
It's down? Uh oh. :)
13:32 Sherry
I've been getting 503 errors after lots of "spinning"
13:32 Sherry
And Sonia doesn't like me putting "test" files on Harvard's :-)
13:34 pdurbin
Sherry: can you please email support
13:36 Sherry
will do
13:36 pdurbin
Thanks! Please let me know the ticket number you get.
13:56 Slava1 joined #dataverse
13:59 donsizemore joined #dataverse
14:00 donsizemore
@Sherry feel free to beat up on https://dataverse-test.irss.unc.edu though we're still sitting on a custom build of 4.11
15:12 skarcher joined #dataverse
15:12 skarcher
I'm getting a 503 from the Demo Dataverse currently -- are you aware of problems there?
15:39 jri_ joined #dataverse
15:43 Sherry joined #dataverse
15:44 pdurbin
skarcher: hi! Yeah, I think Sherry opened a ticket.
15:44 Sherry
Sorry, was in a meeting when this arrived - at 10:04 reported Demo dataverse down See all of this ticket: https://help.hmdc.harvard.edu/Ticket/Display.html?id=281885
15:44 pdurbin
Sherry: thanks!
15:48 skarcher
pdurbin: one other Question. I'm seeing a lot of .zip files (for recent projects) in Harvard Dataverse. The only way to get those in is still double-zipping, right? So people are doing that on purpose?
15:51 pdurbin
Yeah, double-zipping is still the workaround. Let me dig up an issue you can put n your board for QDR if you'd like.
15:52 pdurbin
Oh, this is the issue I was thinking about but bjonnh closed it: https://github.com/IQSS/dataverse/issues/3439 :) Because he's all set now that file hierarchy is supported. So I guess a fresh issue is in order, skarcher, if you don't mind.
15:54 skarcher
Thanks -- no, this is working perfectly for us; I don't really have a use case. On the contrary, I think people shouldn't be doing this. I'm just advising someone working with Harvard DV and was looking around and surprised.
15:55 pdurbin
Ah, ok. Makes sense. Yeah, I'm not sure what to say expect that a "About Harvard Dataverse" website is in the works and I assume it'll have something to say about preferred formats. I assume it will discourage uploading zip files.
15:59 pdurbin
Here's where we are tracking the creation of that new website, if that's of interest: https://github.com/IQSS/dataverse.harvard.edu/issues/26
15:59 pdurbin
skarcher: if you want you could leave a comment saying that zip files should be discouraged.
16:02 joelmarkanderson joined #dataverse
16:05 joelmarkanderson
pdurbin: you avail? got a problem today
16:06 pdurbin
joelmarkanderson: doing many things but how can I help? :)
16:06 pdurbin
(including Slacking with xarthisius about external tools) :)
16:07 joelmarkanderson
i'll describe it, but i can take a number and wait!
16:08 pdurbin
joelmarkanderson: the best way to get a number (a ticket number) is to email support
16:08 joelmarkanderson
punch line: my admins reindexed solr, and now we seem to be missing some datasets on the main page (~120 or so)
16:08 pdurbin
But someone else here might be able to help. We're a community. :)
16:09 joelmarkanderson
i can see them in psql still though
16:09 pdurbin
joelmarkanderson: ok, others have seen this too and I'd recommend trying to index one of those datasets individually
16:09 pdurbin
curl http://localhost:8080/api/admin/index/dataset?persistentId=doi:10.5072/FK2/AAA000
16:09 pdurbin
http://guides.dataverse.org/en/4.17/admin/solr-search-index.html#reindexing-datasets
16:15 pdurbin
joelmarkanderson: we never did track down the cause of https://github.com/IQSS/dataverse/issues/5575 but it might be related
16:16 joelmarkanderson
we did upgrade 4.12->4.16 this week
16:16 pdurbin
gotcha
16:16 joelmarkanderson
was never a problem before
16:16 pdurbin
ok
16:16 pdurbin
Does a manual reindex of a single dataset help?
16:16 joelmarkanderson
working to find one...
16:17 joelmarkanderson
57/177 are on the main page...
16:17 pdurbin
You can also reindex by database id if that's easier.
16:17 joelmarkanderson
which id? dataset?
16:17 pdurbin
yeah
16:18 pdurbin
curl http://localhost:8080/api/admin/index/datasets/7504557
16:18 pdurbin
(from the guides link above)
16:18 pdurbin
oh, there's an undocumented api that might help you find them
16:19 pdurbin
let me see if it even still works on my test server
16:19 pdurbin
curl http://localhost:8080/api/admin/index/status
16:20 joelmarkanderson
{"status":"OK","data":{"contentInDatabaseButStaleInOrMissingFromIndex":{"dataverses":0,"datasets":128},"contentInIndexButNotDatabase":{"dataverses":0,"datasets":0,"files":0},"permissionsInDatabaseButStaleInOrMissingFromIndex":{"dvobjects":1361},"permissionsInIndexButNotDatabase":{"dvobjects":0}}}
16:21 pdurbin
looks like it doesn't show you which ones :(
16:21 joelmarkanderson
that's ok
16:21 pdurbin
maybe just iterate through all the IDs in your "dataset" table
16:24 joelmarkanderson
it's complaining about my metadata block information
16:24 joelmarkanderson
{"status":"ERROR","message":"Problem indexing datasets/9: javax.ejb.EJBException org.apache.solr.client.solrj.impl.HttpSolrClient.RemoteSolrException Error from server at http://localhost:8983/solr/collection1: ERROR: [doc=dataset_9] unknown field 'tag' "}
16:25 joelmarkanderson
tell me when to throw this into a ticket
16:25 pdurbin
Hmm, I wonder when "tag" was added. The release notes should indicate when you need to update Solr's schema.xml file.
16:25 joelmarkanderson
"tag" was added weeks ago
16:26 pdurbin
Ooooooh. That's your custom metadata block.
16:26 joelmarkanderson
yes; i wonder if the admins overlooked this during version upgrade
16:26 pdurbin
So you need to adjust your schema.xml to accomodate your custom "tag" field.
16:27 joelmarkanderson
it should be there
16:27 joelmarkanderson
let me check
16:27 pdurbin
ok
16:31 joelmarkanderson
it's there
16:31 joelmarkanderson
curl http://localhost:8080/api/admin/index/solr/schema?
16:31 pdurbin
very strange
16:36 pdurbin
I guess you could try removing the "tag" field from dataset 9 and checking to see if it gets indexed when you publish it.
16:38 pdurbin
(re-publish it)
16:45 pdurbin
But yeah, you should probably open a ticket. Please be advised that a lot of people took the day off. And I'm off Monday (holiday) and Tuesday.
16:45 joelmarkanderson
ok thanks
16:50 jri joined #dataverse
17:10 joelmarkanderson
pdurbin: so it seems that curl'ing the solr schema was not accurately reflecting the current schema.xml; i bet it was overwritten during patch night
17:19 joelmarkanderson
pdurbin: all is back up now; thanks for your investigative help
17:29 pdurbin
joelmarkanderson: sure! Great news!
17:30 joelmarkanderson
i wonder why the schema curl caches old data?
17:30 pdurbin
No idea. You can ask in #solr. :)
18:26 Slava1 joined #dataverse
19:02 pdurbin
Slava1: great tweets. Thanks!
19:16 bjonnh
pdurbin: o/
19:16 pdurbin
Heh. How's it going, bjonnh?
19:16 bjonnh
pretty good
19:16 pdurbin
skarcher: I'm spinning up a new server now to hopefully stand in for the demo server over the holiday weekend. :)
19:23 skarcher
SU is woke. We don't do Columbus day ;)
19:25 pdurbin
skarcher: great! That means you're in charge in here on Monday. :) (And Tuesday, when I'm out.)
20:17 pdurbin
skarcher: we just updated DNS so that https://demo.dataverse.org is pointed at the server I just stood up. Cert errors are expected but I guess it's better than being completely down. Did you need it for something?
20:17 skarcher
I wanted to test out .zip expanding on a "clean" install
20:18 skarcher
but I think I have all answers I need currently. Thanks!
20:18 pdurbin
Ah. Ok. Well, it's a fresh install of Dataverse 4.17 and you're welcome to bang away at it.
20:21 pdurbin
skarcher: also, I'm looking at https://doi.org/10.25504/FAIRsharing.bmz5ap ... this is new to me. (I've heard of re3data.)
21:19 pdurbin
Ok folks, I'm starting my four day weekend. See you on Wednesday! Have a great weekend!
21:19 pdurbin left #dataverse
