IQSS logo

IRC log for #dataverse, 2021-03-10

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

All times shown according to UTC.

Time S Nick Message
01:44 bjonnh joined #dataverse
07:11 Virgile joined #dataverse
09:55 juancorr joined #dataverse
10:47 poikilotherm2 joined #dataverse
11:06 GitterIntegratio joined #dataverse
12:12 donsizemore joined #dataverse
13:41 pkiraly joined #dataverse
13:49 pkiraly Hi, I run integration tests locacally, and it seems that several ones are broken.
13:49 pkiraly [ERROR] Tests run: 146, Failures: 26, Errors: 3, Skipped: 8
13:56 poikilotherm2 Hi pkiraly :-)
13:56 poikilotherm2 Are you sure you are running the IT tests? I see there are skipped tests...
13:57 poikilotherm2 And can you post a list of failing tests?
13:57 poikilotherm2 That would be really helpful to see what happenes
14:01 donsizemore Jenkins seems to skip 15 these days https://jenkins.dataverse.org/job/IQSS-dataverse-develop/728/testReport/
14:12 pkiraly I run the procedure described in conf/docker-aio/readme.md. The skipped tests must be defined in the scripts mentioned in that file.
14:47 pdurbin pkiraly: you might want to also look at https://guides.dataverse.org/en/5.3/developers/testing.html#getting-set-up-to-run-rest-assured-tests
14:50 poikilotherm2 Morning pdurbin :-)
14:51 poikilotherm2 I'm getting my hands dirty in Solr code...
14:51 pdurbin nice!
14:51 pdurbin Java everywhere
14:52 poikilotherm2 I do have some questions about the code you wrote back in the day...
14:52 poikilotherm2 :-D
14:52 pdurbin uh oh
14:53 pdurbin Any code I wrote more than a year ago is terrible.
14:53 poikilotherm2 Oh that's just 1 week for me. 1 year is pretty good :-D
14:54 pdurbin :)
14:54 poikilotherm2 May I hit you with this or is now not a good time?
14:59 pdurbin I have 2 hours of meetings right now. Then I need to feed my kids lunch. You can put some questions here and I can try to answer but I'll be slow.
15:01 poikilotherm2 OK
15:01 poikilotherm2 I'll be off in ~1h so I'll be slow then, too :-D
15:01 poikilotherm2 Here comes...
15:02 poikilotherm2 1) Looking at api.Index.getSolrSchema(), I see that you fetch the fields of search.SearchFields. Looking at real world responses e. g. from Jülich DATA reveal that these fields are never included in the output...
15:03 poikilotherm2 Now I think about moving the stuff in SearchFields into a proper enum, using search.SolrField as values, feeding the data back
15:04 poikilotherm2 The background is: I need to get the fields iterable (just like you in the getSolrSchema()) to implement a validator if the active Solr schema is corresponding to the metadata blocks etc
15:07 poikilotherm2 This is a first step towards solving https://github.com/IQSS/dataverse/issues/5989, as I need to solve my chicken-n-egg problem https://github.com/gdcc/dataverse-kubernetes/issues/188
15:07 poikilotherm2 This hit me a few times in production, which is bad :-(
15:08 poikilotherm2 But instead of having a script maintaining a conversion from TSV to Solr schema, better let Dataverse manage things. Single source of truth...
15:09 poikilotherm2 Validating if the schema and Dataverse are on the same page is a good first thing to do here ;-)
15:10 poikilotherm2 2. In `SolrField` I see some (very dated) TODOs. May I remove those, as we seemed to have sticked to this by now?
15:14 poikilotherm2 3. As I am writing a validator, I'll add a new bean to encapsulate this logic. In `SolrIndexServiceBean` and `IndexServiceBean` are the checks about orphans. IMHO it would make sense to move these checks to the new bean, too, so things are more streamlined. Sounds good?
15:16 poikilotherm2 Oh and of course there are the current status checks, also looking for stuff in the database, but not inside Solr. This might be better moved to that new checking bean, too... (And applying a common async infrastructure to those)
15:22 pkiraly poikilotherm2, here you are:
15:22 pkiraly Failures:
15:22 pkiraly AccessIT.testDownloadMultip​leFiles_NonLoggedInOpen:266 expected:<200> but was:<404>
15:22 pkiraly AccessIT.testDownloadMultipleF​iles_NonLoggedInRestricted:290 expected:<200> but was:<404>
15:22 pkiraly AccessIT.testDownloadSingleFile:205 expected:<200> but was:<403>
15:22 pkiraly AccessIT.testRequestAccess:512 expected:<200> but was:<400>
15:22 pkiraly DatasetsIT.testCreateDeleteDatasetLink:1739 JSON path message doesn't match.
15:22 pkiraly DatasetsIT.testCreatePublishDestroyDataset:401 Expected status code <200> doesn't match actual status code <401>.
15:22 pkiraly DatasetsIT.testExcludeEmail:682 Expected status code <200> doesn't match actual status code <403>.
15:22 pkiraly DatasetsIT.testLinkingDatasets:2016 Expected status code <200> doesn't match actual status code <403>.
15:22 pkiraly DatasetsIT.testPrivateUrl:1051 expected:<404> but was:<200>
15:22 pkiraly DatasetsIT.testRestrictFileExportDdi:2112 Expected status code <200> doesn't match actual status code <403>.
15:22 pkiraly DownloadFilesIT.downloadAllFilesByVersion:111 Expected status code <200> doesn't match actual status code <401>.
15:22 pkiraly DownloadFilesIT.downloadAllFilesRestricted:285 Expected status code <403> doesn't match actual status code <401>.
15:22 pkiraly FilesIT.testFileMetaDataGetUpdateRoundTrip:1241 expected:<200> but was:<500>
15:22 pkiraly FilesIT.testReplaceFileBadJson:908 Expected status code <200> doesn't match actual status code <400>.
15:22 pkiraly FilesIT.test_006_ReplaceFileGood:361 JSON path message doesn't match.
15:22 pkiraly FilesIT.test_008_ReplaceFileAlreadyDeleted:851 JSON path message doesn't match.
15:22 pkiraly HarvestingServerIT.testOaiFunctionality:162 expected:<1> but was:<0>
15:22 pkiraly InReviewWorkflowIT.testCurat​orSendsCommentsToAuthor:421 JSON path data.notifications[0].type doesn't match.
15:22 pkiraly LinkIT.testLinkedDataset:88 JSON path message doesn't match.
15:22 pkiraly MoveIT.testMoveLinkedDataset:245 Expected status code <403> doesn't match actual status code <200>.
15:22 pkiraly SearchIT.testDatasetThumbnail:422 JSON path data.datasetThumbnailBase64image doesn't match.
15:22 pkiraly SearchIT.testSearchPermisions:138 JSON path data.total_count doesn't match.
15:22 pkiraly SearchIT.testSubtreePermissions:1021 JSON path data.total_count doesn't match.
15:22 pkiraly SwordIT.testCreateDatasetPublishDestroy:563 Expected status code <405> doesn't match actual status code <204>.
15:22 pkiraly SwordIT.testDeleteFiles:737 expected:<2> but was:<0>
15:23 pkiraly UsersIT.testMergeAccounts:200 expected:<200> but was:<400>
15:23 pkiraly Errors:
15:23 pkiraly FilesIT.testDataSizeInDataverse:1371 » IllegalArgument Cannot get property 'me...
15:23 pkiraly FilesIT.testForceReplaceAndUpdate:629 » IllegalArgument Cannot get property 'f...
15:23 pkiraly SearchIT.testAdditionalDatasetContent6300:264 » IllegalArgument Cannot get pro...
15:23 pkiraly Tests run: 146, Failures: 26, Errors: 3, Skipped: 8
15:23 poikilotherm2 Oh oh. That definitely does not look good regarding integration tests...
15:23 poikilotherm2 Sorry, API tests
15:24 poikilotherm2 I have to admit I do not use the docker-aio setup for this. Long standing dream of doing this with testcontainers...
15:26 poikilotherm2 You created a PR for this, didn't you? Then Jenkins should give you a report
15:26 Virgile Hello - 404 error - can't connect - firewall issue ?
15:27 poikilotherm2 Virgile he said he is running this with docker-aio. It would be more than unusual to see a firewall issue inside a container running all of the services inside it.
15:27 pkiraly poikilotherm2, I've created a PR in one of the integration tests which were not broken at the first place.
15:31 Virgile right ^^ i thought these downloads were pointing outside the container
15:32 Virgile and thought it would be the fw of the host that would cause trouble
15:32 Virgile but nvm ^^
15:42 pkiraly Virgile, Do you refer to the issues in integration test that I reported?
15:47 Virgile yes, but i think poikilotherm is right ^^
15:48 poikilotherm2 pdurbin another question:
15:48 poikilotherm2 5. I would like to make DatasetFieldType and SolrField easier coupled by refactoring and using the enum DatasetFieldType.FieldType instead of maintaining a second enum SolrField.SolrType, which creates more hassle then necessary. I hope this makes sense...
15:49 poikilotherm2 (This makes the code much more explicit in terms of how the dots get connected betwenn Solr and database)
16:01 pkiraly pdurbin, the page you suggested says: "we recommend using the “all in one” Docker configuration described in conf/docker-aio/readme.txt in the root of the repo." It mentions readme.txt twicem however it is not readme.txt, but readme.md. Should I create a ticket and PR to fix it?
16:50 pkiraly pdurbin, I've created a PR: https://github.com/IQSS/dataverse/pull/7674
16:58 dataverse-user joined #dataverse
16:59 dataverse-user left #dataverse
19:55 pdurbin thanks for making that PR, Peter
21:51 poikilotherm2 Hey pdurbin... Too many detailed questions? 🙈
21:52 pdurbin poikilotherm2: yes, and sorry, very busy today. And only 8 minutes left.
21:52 poikilotherm2 Oh oh
21:52 pdurbin did you see my comment about web.xml?
21:53 poikilotherm2 No worries. I'll just do what I think is best and create a PR to review and comment then
21:53 poikilotherm2 Yes, just responded
21:53 pdurbin just read your reply, thanks
21:54 pdurbin Sounds like we should open an issue with Payara. Do you want to do that? They know you and like you. :)
21:54 poikilotherm2 Oh. Do they? I always leave load of work for 'em... 🤠
21:54 pdurbin All I have is a Payara t-shirt from JavaOne circa 2017.
21:55 poikilotherm2 Yeah, for that DirConfigSource I would have appreciated some nice swag... They so marvelous great stuff to pose with
21:56 poikilotherm2 +have
21:58 pdurbin I got a trophy once. It's very nice.
22:01 pdurbin Anyway, I'm out. See you all tomorrow.
22:01 pdurbin left #dataverse

| Channels | #dataverse index | Today | | Search | Google Search | Plain-Text | plain, newest first | summary

Connect via chat.dataverse.org to discuss Dataverse (dataverse.org, an open source web application for sharing, citing, analyzing, and preserving research data) with users and developers.