Time
S
Nick
Message
07:12 Virgile joined #dataverse
07:33 VJ joined #dataverse
08:14 dataverse_k8s_64 joined #dataverse
08:15 dataverse_k8s_64
good morning
08:16 dataverse_k8s_64
i am considering the option of installing a dataverse using k8s
08:16 dataverse_k8s_64
but i have some questions
08:16 dataverse_k8s_64
first: it's community driven, and i am afraid the project can be uncontinued
08:17 dataverse_k8s_64
second: which are the pros of k8s comparing to tipical installation (e.g. server for db, server for webapp,...)
10:11 Virgile joined #dataverse
11:05 juancorr joined #dataverse
14:08 Virgile joined #dataverse
15:11 pdurbin joined #dataverse
15:12 pdurbin
shoot, missed those questions from dataverse_k8s_64 but at least there's a related post: https://groups.google.com/g/dataverse-community/c/EZEQKw3gj-k/m/E_fzl5y-AAAJ
15:12 pdurbin
poikilotherm: any interest in replying? :)
15:13 poikilotherm
Dunno
18:38 nightowl313 joined #dataverse
18:40 nightowl313
hi all ... in trying to get an external store set up in our test dataverse, I have messed something up royally. I am getting an error in the log "[#|2021-05-05T18:29:32.746+0000|SEVERE|Payara 5.2020.6|javax.enterprise.resource.webcontainer.jsf.context|_ThreadID=85;_ThreadName=http-thread-pool::jk-connector(2);_TimeMillis=1620239372746;_LevelValue=1000;| javax.faces.application.ViewExpiredException: viewId:/dataset.xhtml - View /dataset.xhtm
18:40 nightowl313
this error is repeating over and over, filling up the log in seconds
18:41 nightowl313
any ideas how to stop it? I've deleted all of the files that I uploaded in testing, and deaccessioned the dataset that I was working on
18:42 nightowl313
removed the extra storage drivers and other options
18:45 pdurbin
nightowl313: have you tried stopping and starting payara?
18:45 pdurbin
That's very strange behavior.
18:46 nightowl313
yes, several times and have restarted the server several times
18:46 pdurbin
Hmm. Let me ask in dv-tech in our Slack.
18:48 nightowl313
when testing the store in dataverse, if i tried to upload a file, the page would just hang, and I would have to hit the back button to get out of it ... wonder if it is just stuck trying to complete the save or redisplay the page
18:48 nightowl313
per jim, that error happens when a user is sitting on the page for a period of time
18:49 donsizemore joined #dataverse
18:49 Jim56 joined #dataverse
18:49 pdurbin
I've definitely seen ViewExpiredException before. I think that's right. If you leave a page up and go to lunch and then try to click something you might see that. Not sure why or when exactly.
18:50 donsizemore
usually I see "view could not be restored" or some such
18:50 Jim56
Any page with buttons that are expired will try to call them and get the view expired.
18:50 Jim56
yeah - but ajax calls don't display that
18:51 Jim56
they just silently fail
18:51 Jim56
refreshing the whole page should fix it
18:51 Jim56
(would be nice to fix that silent fail someday...)
18:52 nightowl313
it is just the same error over and over in the log, and it creates a new log every 20 seconds
18:52 nightowl313
even if no one is on the site
18:53 Jim56
Hmm - are you sure its you, and not a bot repeatedly trying simething?
18:55 nightowl313
could be? I've also been working on customization of the header, footer, and have added a bunch of styling ... so I removed those as well
18:55 donsizemore
if it's behind an apache proxy, what's in ssl_access_log?
18:55 donsizemore
or, more importantly, whomst?
18:55 nightowl313
but this just started yesterday after working on the store
18:56 donsizemore
something like `cat ssl_access_log |awk '{print $1}' |sort |uniq -c |sort -n` should ferret out the busiest clients
18:56 nightowl313
oh yea something going on there
18:56 Jim56
The view is all about the session for a page someone or something is looking at, so its hard to see how its store related. If someone has a window open, I could imagine a bug where a direct upload gone wrong is repeatedly pinging the server, but that would stop with closing the page.
18:57 Jim56
get that IP address!
18:57 nightowl313
this repeated: " [05/May/2021:18:56:38 +0000] "POST /dataset.xhtml?persistentId=doi:10.5072/FK2/CX17WP HTTP /1.1" 200 249"
18:57 nightowl313
it is mostly the ip of the server itself
18:58 pdurbin
Maybe turn off your header and footer customizations, just in case.
18:59 nightowl313
nooo it's not
19:00 nightowl313
they are off for usre
19:00 nightowl313
sure
19:00 donsizemore
@nightowl313 did you remove the client IP from the log entry above, or is that what my cat command spat out?
19:02 nightowl313
i removed it ... sorry thought it was our server ip
19:03 nightowl313
it is an aws private ip, so trying to determine if it is one of ours
19:03 donsizemore
add <RequireAll> Require all granted Require not ip <aws_ip> </RequireAll> to your apache proxy config and see who complains =)
19:05 donsizemore
IRC won't let me format à la Slackque, but those four lines in your Apache SSL vhost should block the ipv4. until they get a new one.
19:08 nightowl313
oh.my.gosh ... it is our nessus scanner!
19:08 poikilotherm
You're running on AWS? How is AWS dealing with outside connections? Is there a reverse proxy in the middle and you will only ever see private IPs connecting like in most K8s clouds?
19:08 nightowl313
the dev site is being vulnerability scanned!
19:08 poikilotherm
Oh. Good you figured that out
19:08 nightowl313
a surprise scan!
19:08 nightowl313
surprise!
19:08 poikilotherm
Tada
19:09 nightowl313
oh my gosh .. i'm so sorry to be a pain here ... i had no idea what was going on and thought our site was exploding or someone was hacking it!
19:09 poikilotherm
The access logs don't contain the user agent string by default, right?
19:10 poikilotherm
Maybe it would be a good idea to add those. Likely that Nessus is setting a special agent string
19:10 Jim56
FWIW: If you see scanners causing stack traces that really fill the log, I started an issue to collect cases where we shouldn't print the trace.
19:11 donsizemore
@Jim56 I missed this issue. I could probably fill IT with traces =)
19:11 Jim56
https://github.com/IQSS/dataverse/issues/7706
19:11 nightowl313
i'm not sure ... i see a lot of lines like the one above and then a lot of cryptic things that look like hacking
19:11 Jim56
Yep - that's what scanners do
19:13 nightowl313
yea, they have done a good job of alerting us to potential issues
19:13 Jim56
QDR set up to report bad DOIs that are requested (so we can see if someone hasn't published but gave out the doi/link, etc.) and we had to ban UTDorkbot's ip because it was asking for DOIs ending in /etc/passwd, etc. -
19:13 Jim56
and filling those reports enough to obscure the relevant bad DOIs
19:14 nightowl313
eeks, i deaccessioned the dataset that they were hacking ... wonder what that will show? =D
19:14 nightowl313
Dorkbot .. so fun that it is named that!
19:15 nightowl313
i will add to your list above!
19:15 nightowl313
thanks to you all for your assistance ... sorry I didn't check that first ... I'm out of my league, I know .. trying to fix that
19:16 Jim56
thanks!
19:17 donsizemore
@nightowl313 during a meeting, have you mistakenly sent IQSS the wrong branch in the wrong repo... this afternoon?
19:17 donsizemore
@nightowl313 'cause I did.
19:17 nightowl31385 joined #dataverse
19:18 nightowl31385
sorry, got kicked out and had to come back in
19:18 nightowl31385
@donsizemore what do you mean?
19:19 donsizemore
you're being sheepish, and i'm telling you that, during a meeting this afternoon, i was talking about a co-worker's work on ingest and managed to paste the wrong branch from the wrong repo into the zoom chat
19:19 nightowl31385
ohhhh hahaha ... i thought you meant i did that, and it is entirely possible that I did ... lol
19:20 donsizemore
oh, no I did it. and Jim caught me so at least I could fix it
19:21 nightowl31385
glad other peole do goofy things ... sorry i'm really feeling like an dummy lately ... so much to learn, so little time ... appreciate all the assistance from you and jim
19:21 nightowl31385
don't think asu dataverse would be a thing without your help!
19:22 poikilotherm
No worries. Just keep the questions coming.
19:22 nightowl31385
haha you can count on it!
19:22 nightowl31385
=D
19:22 donsizemore
Dataverse has a lot of moving parts
19:23 * pdurbin
ran a nessus server in a previous life
19:25 nightowl31385
and we are trying to implement everything at once .. the cloud, storage, backups, DR, while also just learning dataverse .. was really good to hear the conversation yesterday in the meeting with other orgs dealing with config and other things
19:25 nightowl31385
oh sorry, just realized it is qualys doing the scanning, but we do have a nessus scanner as well
19:36 nightowl31385
thanks again to you all for all the help .. this is the best support community i've ever worked with!
19:44 donsizemore
Qualys makes us bizzy but i've never noticed it to throw Dataverse for quite the loop your scanner appliance seemed to
19:48 nightowl31385
yea, it would be interesting to see if there are ways to keep all of those logs from being generated ... it literally was creating a new log every 20 seconds
19:54 pdurbin
Sounds like a denial of service attack in the making.
19:54 pdurbin
nightowl31385: you are welcome to comment on https://github.com/IQSS/dataverse/issues/7706
19:55 pdurbin
And you could also set up log rotation so your disk doesn't fill up.
19:58 nightowl31385
yes, i will add to that for sure ... i believe we have log rotation set up but i think i need to tweak it! =D
19:59 pdurbin
:)
20:39 pdurbin left #dataverse
22:26 nils`` joined #dataverse
