Time
S
Nick
Message
00:17 danmcp joined #dataverse
01:23 danmcp joined #dataverse
03:23 tyrel joined #dataverse
03:36 tyrel joined #dataverse
04:22 tyrel joined #dataverse
04:51 tyrel joined #dataverse
05:53 tyrel joined #dataverse
08:06 jri joined #dataverse
12:06 pdurbin joined #dataverse
12:37 jri joined #dataverse
12:45 pdurbin
tyrel: awesome that you got `vagrant up` working.
12:48 jri_ joined #dataverse
13:03 danmcp joined #dataverse
13:08 jri joined #dataverse
13:10 danmcp joined #dataverse
14:01 danmcp joined #dataverse
15:05 pameyer joined #dataverse
15:07 pameyer
pdurbin: did you see http://irclog.iq.harvard.edu/dataverse/2018-03-02#i_63996 ?
15:13 pdurbin_m joined #dataverse
15:14 pdurbin_m
pameyer: yes! I tried to give you a shout out at http://irclog.iq.harvard.edu/dataverse/2018-03-03#i_64098
15:15 pdurbin_m
I need to get in touch with aculich, obviously.
15:18 pameyer
it sounds like he'll be back online (probably with students) earlier Friday - but it might be good to connect before then (if possible) for some more precise scheduling
15:18 pameyer
glad you saw it!
15:20 pdurbin_m
Ah, "with the students there is a time next friday earlier in the day." I glossed over that. Thanks.
16:30 danmcp joined #dataverse
16:38 jri joined #dataverse
16:40 jri joined #dataverse
17:36 pdurbin
pameyer: do you have any thoughts on https://github.com/IQSS/dataverse/issues/4481 ? "As a sysadmin, I'd like to know how much usage (hits, etc.) my Dataverse installation is getting". Maybe this is a question for andrewSC bricas and knikolla as well.
17:37 pameyer
pdurbin: the awstats one?
17:38 pdurbin
Yeah, as a starting point I documented `yum install awstats` etc at https://github.com/IQSS/dataverse/commit/a56a3ef and I stood up the awstats web interface at https://dev1.dataverse.org/awstats/awstats.pl?config=localhost.localdomain (blocked by IP but anyone here can PM me if they want to see it).
17:40 pameyer
I've seen the awstats interface before - anything non-standard that would be worth unblocking an IP ?
17:42 pdurbin
Probably not. It's awstats 7.0 from EPEL. Very standard.
17:43 pameyer
https://goaccess.io/ might be worth a look - but it seems essential the same concept as awstats
17:44 pameyer
I think awstats would be a useful tool, but probably not top of my list
17:48 pdurbin
awstats is just something we happened to use two jobs ago so I'm vaguely familiar with it. GoAccess doesn't seem to support Glassfish out of the box according to https://goaccess.io/man#custom-log but that's not especially surprising.
17:48 pdurbin
"Predefined options include, Common Log Format (CLF), Combined Log Format (XLF/ELF), including virtual host, W3C format (IIS) and Amazon CloudFront (Download Distribution)."
17:48 pameyer
ah - I'd thought you were going with apache and X-Forwarded-For
17:48 pameyer
missed the bit about glassfish logs
17:50 pdurbin
Right now I'm using awstats to process Apache's /var/log/httpd/ssl_access_log (and rotated logs) as-is. I haven't re-configured Apache to show the X-Forwarded-For header in the logs.
17:50 pameyer
google analytics / piwik might also be useful; but complementary to log analysis. Can't get page load times from server side log analysis; but ga/piwik would probably be uninformative for scripts/apis/my normal browsing profile
17:52 pdurbin
Right, Google Analytics and Piwiki rely on cookies and I think I'll walk down the hall and see if I can get access to our GA data but I'm thinking that a server-side log analytics solution would probably give a more comprehensive picture of what's going on because it wouldn't rely on cookies.
17:54 pameyer
cookies / javascript / hidden "pixel" images / other annoyances - browsers with cookies disabled would still show up
17:54 pameyer
but if I'm remembering correctly, dataverse with cookies disabled is already unhappy
17:55 pameyer
…. that translates in to me agreeing with you about log analysis here
17:55 pdurbin
Yeah, the jsessionid goes in a cooke. What about the ELK stack? (Elasticsearch, Logstash, and Kibana.) Is it still popular? Is it worth learning and using? I see bencomp wrote a "grok pattern" for Glassfish server.log and access.log at https://gist.github.com/bencomp/9b99cee2720d6233c8a1
17:57 pameyer
haven't gotten into ELK, so I don't have a useful opinion there
18:00 pdurbin
Ok, no worries. If someone else here has an opinion on ELK, please let me know.
18:05 pdurbin
pameyer: have you used GoAccess?
18:29 dimitri joined #dataverse
18:38 andrewSC
pdurbin: for something fairly comprehensive i'd recommend datadoghq.com (if you're able to get funding/pay for it)
18:38 andrewSC
I've used it personally at Adobe for a handful of projects and it was pretty nice
18:39 andrewSC
where i work now we don't have it and we honestly haven't had any discussions around analytics yet
18:39 andrewSC
we probably should though....
18:39 andrewSC
I know i'm using GA to track a couple projects i maintain here but that's only for my own personal info
18:40 andrewSC
I'm not entirely sure what the open source variant of datadog would be?
18:42 pdurbin
I think people in my building are starting to use Datadog for monitoring. Different group than the one that hosts our production installation though. Thanks, andrewSC.
18:43 andrewSC
mmmmmm nice
18:43 andrewSC
and for sure :)
18:44 pdurbin
This post is from November so I have the impression that analyzing logs is a relatively new feature of Datadog: https://www.datadoghq.com/blog/announcing-logs/
18:44 pameyer
pdurbin: did a little investigation into goaccess, but it hasn't been high priority enough to get it (or awstats) converged
18:45 andrewSC
you know what, i think we used something else for parsing logs
18:45 pdurbin
pameyer: ok, thanks
18:45 andrewSC
i can't remember the name though
18:45 andrewSC
lemme ask a buddy of mine
18:45 pdurbin
andrewSC: ok but no worries
18:46 pdurbin
Whatever we come up with for making sense of access logs, I think it would be nice to write up in the Dataverse Admin Guide as a suggested approach.
18:47 pdurbin
There are 32 production installations of Dataverse around the world and I wonder if anyone else already has a server-side solution in place. Something other than Google Analytics or Piwik, I mean.
18:47 andrewSC
mmmmm
18:47 andrewSC
pretty sure it was this https://www.sumologic.com
18:48 pdurbin
andrewSC: I haven't heard of it but thanks!
18:48 andrewSC
npnp
18:50 pameyer
andrewSC: for some reason, that made me think of P =?= NP
18:50 andrewSC
hahah
19:57 andrewSC
do you guys use another service or tool to handle other types of submissions?
19:57 andrewSC
i know that sounds overly vague lol
19:57 andrewSC
basically i'm trying to figure out if there's something out there that's open source that is a submission management system?
19:57 andrewSC
or if i need to build something in house
19:59 pdurbin
What do you mean by submission?
20:04 andrewSC
so for example outside of publications we also have a couple larger planning/process? sprints throughout the year around gathering research proposals for the coming year, people submitting research posters for the symposium we host every year, project leads submitting the final output of their project's etc
20:05 andrewSC
some of these things end up in dataverse while others are more ephemeral? in nature
20:05 pdurbin
People submitting talk ideas for the next Dataverse Community Meeting?
20:05 andrewSC
hmmmmm that could be something similar
20:05 andrewSC
yeah
20:06 andrewSC
how do you guys handle that now? just a google form or is there some tooling or is it via email or?
20:07 pdurbin
I want to say we used a Google Form but I can't remember or tell by looking at https://groups.google.com/d/msg/dataverse-community/S7lV0-QmgLY/Y6q8YtwTAwAJ
20:08 pdurbin
Probably either Google Forms or Qualtrics. It sounds like you were looking for an open source solution, though.
20:08 pameyer
andrewSC: it might be worth looking at conference management software
20:09 pameyer
I don't have any recommendations, but it sounds like that might be a name for useful searching
20:10 pdurbin
andrewSC: "an easy submission process" at https://osf.io/meetings/ ... "A free poster and presentation sharing service for academic meetings and conferences"
20:11 andrewSC
nicee
20:11 andrewSC
thanks for the suggestions :)
20:13 pdurbin
sure
22:28 pameyer
pdurbin: do you have strong opinions about APIs "returning what you sent" and binary data?
22:54 pdurbin
pameyer: are you testing a Dataverse API ?
22:54 pdurbin
This might be a good question for #rest here on freenode.
22:55 pameyer
every time I run integration tests I get a bell in my terminal because the thumbnail APIs return the thumbnail
22:56 pdurbin
Oh. I wrote that API endpoint. I wanted to be able to test that the thumbnail was correct. :/
22:58 pdurbin
In this case, the binary data is probably a PNG, which doesn't strike me as weird to return.
22:58 pameyer
np
22:58 pameyer
thanks
23:08 pdurbin
sorry about the bells
23:08 pdurbin
:)
