Time
S
Nick
Message
06:45 juancorr joined #dataverse
07:03 poikilotherm joined #dataverse
07:19 jri joined #dataverse
09:38 pdurbin
poikilotherm: morning, how's the diagram coming? :)
11:22 pdurbin joined #dataverse
12:16 poikilotherm
Mornin pdurbin
12:16 poikilotherm
Had a VC session with Slava for two hours, thus being unresponsive
12:17 poikilotherm
The diagram is pretty much finished, but man - rST is not very suitable for my idea with the clickable SVG
12:17 poikilotherm
It requires hacky code in the docs, which seems a bad idea
12:17 poikilotherm
No idea yet how to get along with that
12:18 poikilotherm
Not sure if you guys want a larger portion of pure HTML in the docs :-D
12:22 poikilotherm
Currently hacking on https://github.com/IQSS/dataverse-kubernetes/blob/44-add-docs/docs/how-it-works.md
12:22 poikilotherm
Want to make Slava and Wilko at DANS/CESSDA/DataverseEU happy
12:22 poikilotherm
They are VERY interested in trying out my stuff :-D
12:30 pdurbin
Great! Wow, two hours. The "User Name" button at http://guides.dataverse.org/en/4.14/style/patterns.html#navbar is clickable, if that helps.
12:31 poikilotherm
It helps, but it is plain html :-/
12:31 poikilotherm
Which seems fine for a style guide
12:33 pdurbin
yeah
12:33 poikilotherm
I am familiar with HTML but maybe other people might dislike HTML sources... ;-)
12:33 pdurbin
I like how you can click the little triangles under the "how it works" diagrams and see the plantuml.
12:34 poikilotherm
Oh this is just a side effect...
12:34 poikilotherm
I am using Gravizo to render the images on-the-fly
12:34 pdurbin
cool, haven't heard of it
12:37 pdurbin
How do you think we could improve our technology stack?
12:37 pdurbin
You seem happy enough with Java and Postgres and Solr.
12:37 pdurbin
No objections to those building blocks, right?
12:50 poikilotherm
Depends...
12:50 poikilotherm
Solr is cool.
12:51 poikilotherm
But what be really cool is having dynamic schemas
12:51 poikilotherm
More like we do with Flyway now for database
12:51 poikilotherm
Dunno if this is possible with Solr
12:51 poikilotherm
Slava told me that Peter Kiraly is about to investigate in that field
12:52 pdurbin
Yes, one can put Solr into a schemaless mode.
12:52 poikilotherm
It would be very usefull for us
12:52 pdurbin
"Schema when you want, schemaless when you don't" https://lucene.apache.org/solr/features.html
12:53 poikilotherm
The most critical part of technology right now seems to be Glassfish
12:53 pdurbin
This pull request requires a Solr schema update: https://github.com/IQSS/dataverse/pull/5820
12:54 pdurbin
Critical in what way?
12:54 poikilotherm
Unsupported
12:54 poikilotherm
Might contain critical issues
12:55 poikilotherm
Compared to current things poor cluster support
12:56 pdurbin
Yes, there is a great need to get off of ancient Glassfish.
12:58 pdurbin
But then what? What next?
13:00 poikilotherm
You mean after a successfull transition to 5?
13:00 pdurbin
Yeah.
13:01 pdurbin
What's the most critical thing after that? Or what should be our next step in terms of our technology stack?
13:01 poikilotherm
There might be a need to transition to Jakarta EE, depending on how this will continue at Eclipse vs Oracle
13:01 pdurbin
How do we continue to evolve?
13:01 pdurbin
Right. We should move from Java EE to Jakarta EE once we can.
13:01 pdurbin
Then what?
13:02 poikilotherm
You might look into blockchain stuff
13:02 pdurbin
meh, blockchain
13:02 poikilotherm
Not so fast...
13:02 pdurbin
I was hoping you'd say MicroProfile next.
13:03 poikilotherm
:-D
13:03 poikilotherm
I didn't mention that because I fear of resistance at IQSS
13:03 poikilotherm
Going for architecture change is a huge thing
13:03 poikilotherm
Or do you want me to think big?
13:03 pdurbin
yes, please think big
13:04 poikilotherm
Ok.
13:04 poikilotherm
Dataverse needs to get more modular
13:04 pdurbin
and I am not at all resistant to MicroProfile
13:04 poikilotherm
Its a really big thing to compile, test and deploy
13:04 poikilotherm
It takes ages to load and in many places seems to suffer from maintenance hell
13:05 poikilotherm
Of course this is due to the history and it evolved
13:05 poikilotherm
Which is absolutely fine and it is great that you guys made this wonderful peace an OSS
13:05 poikilotherm
So we all can use it
13:05 poikilotherm
But the underlying technology changes and evolves
13:06 poikilotherm
Getting it more modular might be a good first thing
13:06 pdurbin
Yes, I absolutely agree. More modularity is desperately needed.
13:06 poikilotherm
You already have a cool API
13:06 poikilotherm
Which is a pretty good thing when looking into the direction of service oriented architectures
13:07 poikilotherm
It would be a great thing to get Dataverse more scalable
13:07 pdurbin
Multiple Glassfish servers, you mean? Scalable in what way?
13:08 poikilotherm
I mean distribute the work
13:08 poikilotherm
Let indexing happen on one instance, access users on another
13:08 poikilotherm
Etc
13:08 poikilotherm
Especially things like full text index is a CPU intense thing
13:08 poikilotherm
Offloading would be great
13:09 pdurbin
Yes, I totally agree.
13:09 pdurbin
What's after scalability?
13:13 poikilotherm
Hard to tell
13:13 poikilotherm
All of this stuff seems to be enough for the next 2 years
13:13 pdurbin
Heh, sure. Plenty of work to do.
13:13 pdurbin
How do you feel about the UI?
13:14 pdurbin
UI/UX
13:14 poikilotherm
I can't give an exagerated answer to that, as haven't used it very much yet :-D
13:15 poikilotherm
This should change once we have scientist using it
13:15 poikilotherm
The auth stuff should be improved though
13:19 pdurbin
How do you feel about our UI technology choices, the building blocks we use?
13:19 poikilotherm
That's a difficult question.
13:19 poikilotherm
I am not very experienced in EE UI frameworks
13:20 pdurbin
me neither
13:20 poikilotherm
I am not sure if the mix of Primefaces, Omnifaces etc is a good idea these days
13:20 poikilotherm
But I am not experienced enough to propose an alternative
13:21 pdurbin
Ok. If you gain that experience, I'd be curious to hear your thoughts.
13:21 poikilotherm
:-D
13:21 pdurbin
or even if you don't :)
13:22 poikilotherm
I am unsure if the current stack might be a blocker for the transition to Payara/Glassfish 5
13:22 poikilotherm
The errors seemed to be UI related
13:23 pdurbin
I sure hope not.
13:23 poikilotherm
Disclaimer: I haven't digged into this yet
13:23 poikilotherm
Just an impression
13:23 pdurbin
dug. and s/peace/piece/
13:24 poikilotherm
Whoops
13:25 pdurbin
ok, so the next step should be getting off ancient glassfish
13:25 pdurbin
it's hard to argue with that
13:25 poikilotherm
Argue with whom or abou what?
13:25 poikilotherm
+t
13:26 pdurbin
sorry, it's just a figure of speech, I guess
13:26 poikilotherm
Oh I missunderstood that
13:26 pdurbin
I just mean that if we had a technology roadmap...
13:26 pdurbin
which I don't think we do
13:27 pdurbin
... the next logical step is to get off ancient glassfish.
13:27 poikilotherm
Hmm. Can't remember one
13:27 poikilotherm
Maybe it would be a good idea to setup one?
13:27 pdurbin
yes, it would be a good idea
13:27 pdurbin
to complement the *feature* roadmap at https://dataverse.org/goals-roadmap-and-releases
13:28 poikilotherm
You could create another column. One for features, one for infrastructure?
13:28 pdurbin
on the feature roadmap, you mean?
13:28 poikilotherm
Yeah
13:29 poikilotherm
Right in that map you linked to
13:29 poikilotherm
And that map is pretty outdated :-D
13:29 pdurbin
yes it is
13:29 pdurbin
where else could we put a tech roadmap?
13:30 poikilotherm
You could provide a more elaborated version in the docs
13:30 pdurbin
as a markdown file or something?
13:30 poikilotherm
(Which is true for features, too)
13:30 donsizemore joined #dataverse
13:30 poikilotherm
Not necessarily
13:31 pdurbin
or rst file? in the dev guide?
13:31 poikilotherm
Yeah
13:31 poikilotherm
Why not
13:31 poikilotherm
That was what I meant
13:31 poikilotherm
This should be linked from the Install Guide, too I think
13:32 poikilotherm
As those technology stuff is pretty much linked to admins and installations
13:32 pdurbin
sure, good idea
13:32 poikilotherm
s/those/this/
13:32 pdurbin
I already wrote a roadmap for automated testing: http://guides.dataverse.org/en/4.14/developers/testing.html#future-work
13:32 poikilotherm
Oh yeah, testing is another HUGE field for improvement
13:33 poikilotherm
Maybe - just maybe, it could be justified to create a totally separate chapter
13:33 poikilotherm
Side by side with the guides
13:34 poikilotherm
Containing, feature, testing and infrastructure roadmap
13:34 poikilotherm
infrastructure/technology
13:37 pdurbin
sorry a new guide? or a new page in the dev guide?
13:48 poikilotherm
New "guide"
13:48 poikilotherm
"Toplevel"
13:59 donsizemore joined #dataverse
14:18 donsizemore
@poikilotherm agreed on solr and CPU , but can't that simply be run in a separate space now?
14:18 poikilotherm
Dunno. Currently there is only 1 index.
14:18 poikilotherm
That might get better with a Solr cluster
14:19 poikilotherm
Anyway, it will block one Dataverse instance
14:19 poikilotherm
And only a friction of Dataverse functions is needed for indexing
14:19 poikilotherm
At least no or only a very limited UI
14:19 poikilotherm
Etc
14:22 pdurbin
donsizemore: you're right, poikilotherm *could* run a dedicated Glassfish server just for reindexing. But he'd want to take a look at this first: http://guides.dataverse.org/en/4.14/installation/advanced.html#multiple-glassfish-servers
14:22 pdurbin
poikilotherm: what would the guide be called?
14:23 poikilotherm
Roadmap?
14:23 poikilotherm
The docs are named guides, but are still the docs ;-)
14:24 pdurbin
heh, I call them docs
14:24 pdurbin
it's all docs
14:25 pdurbin
poikilotherm: it has to end in "Guide" if it's a guide. Roadmap Guide? That's a little weird. Any other thoughts?
14:26 poikilotherm
To be precise: https://pasteboard.co/IeYvlic.png
14:26 pdurbin
donsizemore: how's the api test slog? Can I help? :)
14:30 jri_ joined #dataverse
14:50 donsizemore joined #dataverse
15:02 donsizemore
@pdurbin i've been dealing with MDS side-channels for the past day or so
15:05 pdurbin
huh, this? https://www.redhat.com/en/blog/understanding-mds-vulnerability-what-it-why-it-works-and-how-mitigate-it
15:09 donsizemore
yis
15:09 lubitchv joined #dataverse
15:09 donsizemore
@pdurbin also, andrey is at work extending the BagIT API endpoint to allow it to write to local filesystems
15:11 pdurbin
huh, interesting. is there an issue for the bagit stuff?
15:11 pdurbin
lubitchv: hi!
15:11 donsizemore
@pdurbin there will be when we're ready to make a PR =)
15:12 pdurbin
ok, good :)
15:37 pdurbin
wow, they must be close to ready: https://github.com/IQSS/dataverse/issues/5850 :)
16:38 pdurbin
lubitchv: hi, I saw you pushed a couple doc commits. Thanks!
16:48 pdurbin
As I just commented, I moved the issue and pull request to QA. Thanks again.
18:15 lubitchv left #dataverse
18:25 poikilotherm joined #dataverse
18:59 poikilotherm
Hey pdurbin, still around?
19:04 pdurbin
poikilotherm: back from tech hours. How can I help you? :)
19:05 poikilotherm
You could take a look at https://github.com/IQSS/dataverse-kubernetes/blob/44-add-docs/docs/how-it-works.md and tell me what you think ;-)#
19:06 pdurbin
fire at will?
19:07 poikilotherm
Yeah. You don't need to deploy the demo postgres when you want to use your own
19:07 poikilotherm
Maybe you already have a Postgres
19:08 pdurbin
Oh. Postgres is optional because maybe you already have it installed. Hmm. Ok. I would remove "fire at will" because I don't think it adds clarity.
19:08 poikilotherm
Ok, maybe the military term is misleading :-/ It was so nice having it short and simple
19:08 pdurbin
Also, who is Tini?
19:09 poikilotherm
https://github.com/krallin/tini
19:09 poikilotherm
I'll add a comment
19:09 pdurbin
huh. ok
19:14 pdurbin
otherwise it looks nice
19:14 poikilotherm
Does it make the deployment process clearer?
19:14 poikilotherm
What happens when and why?
19:17 pdurbin
yep
19:17 pdurbin
good job
19:17 pdurbin
and if it's wrong, someone can open an issue :)
19:17 poikilotherm
:-D
20:11 poikilotherm
Oi pdurbin, is https://github.com/IQSS/dataverse-kubernetes/pull/45 sufficient for now?
20:11 poikilotherm
I think about merging it...
20:12 poikilotherm
Want to clean up a bit so DANS has an easier start ;-)
20:17 pdurbin
ah, the AWS one. I haven't had a chance to test it. I'd say go ahead and merge it. Someone can open an issue if it doesn't work.
20:21 poikilotherm-2 joined #dataverse
20:21 poikilotherm-2
Alright
20:22 poikilotherm-2
Shall I keep the issue open?
20:22 pdurbin
no, I think you should close it. you addressed it. thanks!
20:30 poikilotherm-2 joined #dataverse
20:32 poikilotherm joined #dataverse
20:32 poikilotherm
Merged, commented and closed :-)
20:35 pdurbin
nice
