Time
S
Nick
Message
00:11 foobarbecue
I'm getting CreateContainerError from the schema-init container in the solr pod
08:05 juancorr joined #dataverse
11:34 donsizemore joined #dataverse
12:01 donsizemore joined #dataverse
12:31 yoh joined #dataverse
13:34 dataverse-user joined #dataverse
14:12 pdurbin joined #dataverse
14:14 pdurbin
foobarbecue: are you using dataverse-kubernetes by poikilotherm? Or dataverse-docker by 4tikhonov? Or something else?
14:18 poikilotherm
pdurbin: Looking at what images he had problems with, it looks like dataverse-kubernetes
14:18 poikilotherm
(Good morning BTW)
14:20 pdurbin
Good morning. First snow here.
14:23 poikilotherm
o rly
14:23 poikilotherm
It's cloudy here, yet we have ~10°C
14:24 pdurbin
I just signed up for https://dataversecommunity.slack.com
14:27 poikilotherm
O RLY? :-D
14:27 poikilotherm
Such a long name
14:28 pdurbin
so much typing
14:28 pdurbin
What does the Hulk yell? Too many stairs!
14:29 poikilotherm
Meh. I cannot sign up because I don't have a harvard mail address
14:29 pdurbin
If you sign up for the new metadata working group you'll get an invite.
14:30 pdurbin
see https://groups.google.com/g/dataverse-community/c/dyNNdNXNe-I/m/29Z5bgdCAQAJ
14:31 poikilotherm
http://slack.k8s.io/
14:31 poikilotherm
I like their approach to create invites :-)
14:31 poikilotherm
You can also ping them on Twitter to get an invitatio
14:34 poikilotherm
That service is based on https://github.com/rauchg/slackin
14:34 pdurbin
Yeah, Slackin is pretty common.
14:38 poikilotherm
Although there are other options than Slack, this might still be a possible solution to https://github.com/IQSS/chat.dataverse.org/issues/7
14:38 poikilotherm
We might take another look at https://github.com/IQSS/chat.dataverse.org/issues/12 , combining this slack with matrix...
14:41 pdurbin
poikilotherm: sure but as a first step, should we have you join?
14:41 poikilotherm
That would be very kind of you folks :-D
14:41 pdurbin
Are you going to sign up for that working group? That's the way to join.
14:42 poikilotherm
I did
14:42 poikilotherm
My colleague too :-)
14:43 poikilotherm
I read the announcement that not only devs are welcome, but non-devs would be much appreciated to join to broaden the discussion and don't miss any points of view.
14:45 pdurbin
yes, all are welcome, I'd say
14:46 pdurbin
poikilotherm: ok, your invite has been sent, I hear
14:49 poikilotherm
I already waved to everyone in #general ;-)
14:51 pdurbin
Cool. So are we giving up on SLOPI? I hope not.
14:51 poikilotherm
Depends.
14:51 poikilotherm
Plenty of options
14:52 poikilotherm
Matrix Bridge, Slackin, ...
14:53 pdurbin
Nobody expects Slack to be SLOPI. It would be surprising, shocking, disturbing. A SLOPI solution needs to be SLOPI from the start.
14:56 poikilotherm
I remember we had this discussion before :-D
14:58 poikilotherm
And once more I am convinced that this would need transparent communication. Like a "public" channel or "general" announced as being public.
14:59 poikilotherm
For example when we'd create channel bridged to matrix, that one would be a perfect public channel, logable etc.
14:59 pdurbin
#general already has expectations of being a normal Slack channel so #public makes more sense as the SLOPI channel.
15:00 poikilotherm
Aye
15:00 poikilotherm
Anyone wanting to join a SIG could lookup communication in the channels later on, once joined
15:01 poikilotherm
K8s people are pretty happy with that
15:01 poikilotherm
And they are a much bigger community than we are ;-)
15:01 poikilotherm
Oh sorry it's not SIG but WG
15:04 pdurbin
It sounds like you're saying with k8s you can look up history of the conversation after you join, but not before. So it's not public.
15:06 poikilotherm
Yeah. That's why I would like to see a SLOPI public channel, bridged to Matrix.
15:06 pdurbin
Ok. But you still need a logger, right?
15:06 * pdurbin
gives iqlogbot a treat
15:07 poikilotherm
Yeah
15:07 donsizemore joined #dataverse
15:07 poikilotherm
Plenty of options to do that :-)
15:08 poikilotherm
Google spits out a few options searching for slack log bot... ;-)
15:13 pdurbin
ok
15:13 pdurbin
Are we the only ones who care about SLOPI?
15:13 pdurbin
(Or am I the only one?)
15:19 poikilotherm
SLOPI is great.
15:19 poikilotherm
Yet I'm not sure anyone else really cares or just want's to get the job done
15:21 pameyer joined #dataverse
15:22 pdurbin
That's my sense too. That no one thinks SLOPI is worth the effort to get it set up, to set up a logger like iqlogbot back in the day, to set up and maintain a modern equivalent.
15:24 poikilotherm
Maybe for most people searching via Slacks history is just enough
15:25 poikilotherm
I have to commit that I barely look at the logs history
15:25 poikilotherm
Missed messages aren't a thing anymore now using this IRC via Matrix, and won't be with Slack
15:26 poikilotherm
That had be my main reason to go look at them
15:27 poikilotherm
It might be interesting to archive channels in Slack when there is no possibility to get an advanced plan for little or no money, as only 10.000 messages are archived
15:30 pdurbin
I think the public IRC log is valuable for newcomers. They can see that there are real people talking about Dataverse stuff. Then they join in the conversation. It's the same as the public mailing list. Signs of life.
15:36 pameyer
public logs are very useful
15:37 * pdurbin
fist bumps pameyer
15:38 pameyer
haven't usually thought of them in terms of "signs of life" - but it's really helpful for newcomers to see "ah, this is how people are working on this problem" when they see some error message that doesn't make sense yet
15:38 pameyer
but signs of life is another good thing :)
15:39 pdurbin
Right, if you google for an error you might find the solution in the logs. Or at least who to talk to. :)
15:40 pameyer
yup
15:41 pameyer
even if the fix is in github (or the issue's been fixed), how folks figured out the issue is less likely to be - and seeing that can be a big help sometimes
15:43 pdurbin
Yeah, the thought process.
15:43 Virgile joined #dataverse
18:32 foobarbecue9 joined #dataverse
18:33 dataverse_k8s_3 joined #dataverse
18:35 foobarbecue47 joined #dataverse
18:41 foobarbecue
pdurbin, yep I'm using dataverse-kubernetes by poikilotherm! Trying again now. I get CreateContainerError in the schema-init container in the solr pod
18:42 pdurbin
poikilotherm: we should sprinkle some CI on that thing
18:42 pdurbin
foobarbecue: first of all, I love your nick
18:43 pdurbin
I only tried dataverse-kubernetes once on a laptop a year and a half ago using minikube. It worked then!
18:44 foobarbecue
haha, thanks! So, what's a good next debugging step... I have been using k8s for a few months but pretty green. `$ kubectl logs pod/solr-6f85794c47-22kpd schema-init
18:44 foobarbecue
Error from server (BadRequest): container "schema-init" in pod "solr-6f85794c47-22kpd" is waiting to start: CreateContainerError`
18:44 pdurbin
I guess I'd suggest opening an issue, if you haven't already. It's pretty late on a Friday night for poikilotherm.
18:45 pdurbin
I mean, if we're pretty confident the code is fine (and I have no reason to believe it isn't) you could try finding some people who know kubernetes well. I'm not sure how many people like that are here.
18:45 foobarbecue
ok, sure, i'll play with it a bit more and if I still can't figure out why that container won't create I'll post an issue
18:46 pdurbin
sounds good
18:52 poikilotherm
You're lucky I have a few minutes on my mobile
18:52 poikilotherm
Foobarbecue can you take a look at kubectl describe of the failing pod?
18:54 foobarbecue
thanks! Looking now.
18:54 foobarbecue
Uploaded file: https://uploads.kiwiirc.com/files/028b132e8f20b6056cea258734a89b47/pasted.txt
18:56 foobarbecue
^ that's kubectl describe of the failing pod and here's a kubectl get all:
18:56 foobarbecue
Uploaded file: https://uploads.kiwiirc.com/files/a7d7731335e3914964a65262b4f3936e/pasted.txt
18:59 poikilotherm
Hmm. I don't even see an error in the events log
19:01 foobarbecue
I just tried `kubectl apply -k github.com/IQSS/dataverse-kubernetes/k8s/solr` and that has `Warning FailedMount 24s (x8 over 87s) kubelet, k3d-k3s-default-server-0 MountVolume.SetUp failed for volume "api-secret" : secret "dataverse-api" not found`
19:01 foobarbecue
oh wait, should I be pulling from IQSS or from https://github.com/GlobalDataverseCommunityConsortium/ ?
19:02 foobarbecue
I was following these docs which point to IQSS: http://k8s-docs.gdcc.io/en/v4.20/quickstart/k3s.html
19:03 poikilotherm
I just fired up my laptop again
19:03 poikilotherm
What version of K3d are you using?
19:04 poikilotherm
I will try with 3.1.5, although I wrote the docs back in the 1.x days (which is about 4 month ago IIRC)
19:05 poikilotherm
The repo at IQSS does not exist anymore, it has been moved to GDCC. But GH usually forwards all requests.
19:07 poikilotherm
Dear gods, why is Docker Hub so slow today.
19:08 foobarbecue
3.0.1
19:08 foobarbecue
$ k3d --version
19:08 foobarbecue
k3d version v3.0.1
19:08 foobarbecue
k3s version v1.18.6-k3s1 (default)
19:08 poikilotherm
foobarbecue this will take ages. The K3d images haven't been pulled yet...
19:10 poikilotherm
Man.... I'm pulling with maybe 50kb/s
19:10 foobarbecue
woah
19:12 poikilotherm
Did I mention that I want to move the GDCC image to Githubs registry=
19:12 pdurbin
you have so mentioned
19:12 poikilotherm
pdurbin: now that we possess github.com/gdcc, we have nice and short image names
19:12 pdurbin
I do love me some short names.
19:16 poikilotherm
WOW. It took ~10 minutes to pull those ~100 MB
19:17 pdurbin
I can lend you my old 28.8 modem if you think it'll help.
19:20 poikilotherm
Yeah I'll create a trunk :-D
19:22 poikilotherm
OK, deployed. I removed the local path provisioner from the kustomization.yaml because since IIRC 3.0, K3d comes with the provisioner out of the box
19:22 poikilotherm
Now image pulling. Will take another century, I guess
19:28 foobarbecue
maybe they can mail it to you on a floppy disk
19:28 foobarbecue
btw to create my k3d cluster I used `k3d cluster create --port 8080:80
19:29 poikilotherm
GNRF. Still pulling.
19:38 poikilotherm
UN. BE. LIEV. ABLE.
19:38 poikilotherm
This hasn't happened in the past. It's bad.
19:38 poikilotherm
I mean, it's just docker images. Nothing special.
19:39 poikilotherm
Folks I'm going for a walk to the supermarket. BRB, letting the pull complete.
19:42 pameyer
foobarbecue: might be something you've already ruled out, but do you have enough free disk space?
19:50 foobarbecue
Uploaded file: https://uploads.kiwiirc.com/files/6ae182f64c451a135e2abc857b1b56b1/pasted.txt
19:50 foobarbecue
thanks pameyer, I have 378GB free... I am running k3d on Windows Subsystem for Linux... here's df:
19:58 pameyer
great - I mentioned it because docker on mac would sometimes have weird issues when the VM filled up, but the host OS visible stuff had plenty of space
19:59 foobarbecue
ah shoot pameyer, I bet that's exactly what it is
20:00 foobarbecue
at least maybe. it's using Docker Desktop which is probably on a hyper-v VM and that has a space limit. I'll check
20:01 poikilotherm
foobarbecue I got similar error messages about "rror: failed to create containerd container: error unpacking image: failed to extract layer" because of the pull failing.
20:01 foobarbecue
oh nevermind, I don't think it's space now
20:01 poikilotherm
I pull manually and import to K3s from my local docker daemon
20:01 foobarbecue
cool yeah I had pull problems at first but once I got the images down locally I didn't have any more trouble with that
20:02 pameyer
good thing to rule out before diving into other troubleshooting :) would be better if the errors were cleaner, but sometimes they're not
20:02 poikilotherm
Now Solr is starting without flaws
20:02 poikilotherm
It took ages to pull the images :-(
20:03 poikilotherm
I blame Docker Hub for your bad UX
20:04 poikilotherm
This is a nightmare
20:09 pdurbin
scarier than most of the decorations near me
20:13 pameyer
with my default browser profile, docker hub looks remarkably like a blank page
20:15 foobarbecue
Uploaded file: https://uploads.kiwiirc.com/files/e429a6c7075e7d35d77e696092a955dd/pasted.txt
20:15 foobarbecue
so I guess it's having trouble creating the api-secret volume for solr:
20:16 foobarbecue
or, rather, mounting the secret
20:19 foobarbecue
this is when running `kubectl apply -k github.com/GlobalDataverseCommunityConsortium/dataverse-kubernetes/k8s/solr` but ah I suppose I need to create the secrets
20:19 poikilotherm
I'm pushing the current images to Github right now
20:19 poikilotherm
We might try to pull from there
20:22 pdurbin
poikilotherm: have you been holding back the good stuff? 😄
20:27 foobarbecue41 joined #dataverse
20:29 poikilotherm
pdurbin: what?
20:29 poikilotherm
foobarbecue: this should be done for you by the deployment
20:29 poikilotherm
But it depends on what you are deploying
20:30 poikilotherm
It went with the k3s persona
20:30 poikilotherm
s/It/I/
20:30 poikilotherm
That persona pulls in the secrets from the demo persona
20:33 poikilotherm
OK Github Registry is much faster
20:33 pdurbin
poikilotherm: I though you were saying you have newer, better images, somewhere. I guess not. Sorry. 😁
20:38 poikilotherm
Alright, so deploying via kubectl apply -k personas/demo-k3 was successful and doesn't take centuries after changing some bits in the kustomization.yaml
20:38 poikilotherm
(And pushing the images to Github)
20:38 poikilotherm
LOL no
20:39 poikilotherm
Still v4.20
20:39 poikilotherm
v5.x still waiting for things like Postgres support
20:39 poikilotherm
And of course merging the Maven build chain ;-)
20:40 poikilotherm
Maybe I can have a change to earn that shirt for Hacktoberfest by solving the pull troubles...
20:40 pdurbin
:)
20:47 poikilotherm
foobarbecue I changed the personas/demo-k3s/kustomization.yaml like I posted it to https://github.com/GlobalDataverseCommunityConsortium/dataverse-kubernetes/issues/211
20:47 poikilotherm
I documented everything, but I wont work in this today... It's late ;-)
20:47 poikilotherm
s/in/on/
20:48 pdurbin
Ha! "Since unknown time, pulls from Docker Hub take centuries."
20:49 foobarbecue41
ok, thanks. BTW I just tried unmodified personas/demo-k3s on an actual linux machine and it gets past the solr part. So, it was something with my k3d setup on WSL on that windows machine
20:50 foobarbecue41
(now it's having PullErr on a different image... so I'll try issues/211
20:50 foobarbecue41
)
20:50 poikilotherm
Oh. Yeah, that might be a problem. I'm running this on Linux all the time
20:50 poikilotherm
It would be cool to investigate
20:50 poikilotherm
NASA running Dataverse on K8s... \o/
20:51 poikilotherm
foobarbecue feel free to hit me with issues
20:52 poikilotherm
(And PRs if you feel like it ;-) )
20:53 poikilotherm
Yeah, solr-k8s and dataverse-k8s aren't small images...
20:53 poikilotherm
So better pull from Github. Still taking some time, but way faster than Docker Hub
20:54 poikilotherm
Have a good night everyone plus a fabulous weekend
20:54 * poikilotherm
waves at everyone
20:54 pdurbin
eat some candy
20:54 foobarbecue
thanks poikilotherm!
21:00 pdurbin
Ok, folks. Happy Halloween. Have a great weekend. See you Monday.
21:00 pdurbin left #dataverse
21:06 foobarbecue41
btw the demo-k3s is working great now with poikilotherm's github version of the kustomization
21:15 foobarbecue joined #dataverse
21:28 nightowl313 joined #dataverse
21:33 nightowl313
hi everyone! happy friday! wondering if someone could confirm what is happening with apache proxy used with glassfish .. not sure I completely understand ... traffic is coming in on ports 80 and 443 and apache is just passing it off to glassfish, right (onto port 8080 and 8081) or port 8009 if ssl is configured?
21:36 nightowl313
and, for SSL is the ajp connector necessary or could you just use https
21:37 nightowl313
proxy_http i mean
22:09 pameyer
it depends on how apache's configured
22:10 pameyer
I believe that the default way is to send traffic on 80 and 443 through the ajp connector to glassfish/payara 8080
22:10 pameyer
mod_ajp is different from a straight http proxy, but I'm not fully clear on the details
22:11 pameyer
if apache and glassfish are both on the same system, SSL is probably un-necessary
22:40 poikilotherm
Hi nightowl313, usually things get easier to handle when Apache takes care of port 80 and 443, as Java on priviledged ports (<1024) is always a bit... chaotic.
22:41 poikilotherm
You don't have to use proxy_ajp when you are not using mod_shib
22:41 poikilotherm
For example I'm using nginx on K8s as the reverse proxy fronting the application server
22:42 poikilotherm
We don't use HTTPS for proxying the requests to the glassfish app server, as this seems to be a waste of resources for traffic only flowing inside the cluster, running at the same location.
22:43 poikilotherm
Plus HTTPS with Glassfish / Payara is always a bit... chaotic.
23:14 nightowl313 joined #dataverse
23:44 pameyer
yeah, glassfish and certs....
