Centralised logging

1

What do people here use for centralised logging?

I’d like to collect syslog, the systemd journal, /var/log/*.log, and various other structured (e.g. JSON) and unstructured (e.g. .log files) log sources across all my machines and have a nice web UI for viewing them / searching through them. Ideally looking for something self-hosted.

I know of ELK (Elasticsearch, Logstash, and Kibana), fluentd, NXLog, Graylog, and of course the old-school approach of logging everything to syslog then use a remote syslog server, but I’m wondering if anyone has any experience with and would recommend any of these?

I’m already using Sentry for some of my web apps, but it’s more for app-specific error logging rather than general server logs.

Thanks!

2

In one of my abandoned projects I used rsync to copy logs from every server, every few minutes, to unique directories for each server. Then I had written a python script to run through each of those logs and pull data. I learned a lesson that day that sometimes building your own solution instead of using the existing ones is not always a good thing.

Now I just run commands to the whole fleet at once to grep logs or whatever else, example:

root@gw:~# darun exim -bpc
pixel.mxrouting.net: 85
sunfire.mxrouting.net: 57
shadow.mxrouting.net: 172
safari.mxrouting.net: 62
taylor.mxrouting.net: 82
echo.mxrouting.net: 73
lucy.mxrouting.net: 77
blizzard.mxrouting.net: 109
london.mxroute.com: 20
tuesday.mxrouting.net: 70
eagle.mxlogin.com: 83
arrow.mxrouting.net: 101
longhorn.mxrouting.net: 19
moose.mxrouting.net: 49

That’s:

root@gw:~# cat bin/darun
pdsh -w ^/root/hosts/dahosts -R ssh "$@"
1 Like
3

I’m using Elasticsearch for many customers, and works really well. If you want to avoid setting it up yourself is there hosted version, not too bad either. They even have elastic agents (as a replacement for filebeat) now that allows you to remote manage your agents.

1 Like
4

I consider self-hosting for centralised logging much too painful, unless you have some sort of compliance requirement that forbids you to send logs to 3rd parties.

In this particular case, my team chose Grafana Loki instead of Elasticsearch, and I have yet to hear anyone complain about it.

If on the other hand you don’t have any particular compliance requirement and are OK with spending a few dollars to save some time, my preferences are

  • Datadog Logs if you have lots of money to spend
  • LogDNA if you don’t

I’ve been using LogDNA for all my personal servers for at least a couple of years, I spend between $1 and $2 per month, and whilst I had zero expectations I have to say that their support has been nothing short of excellent (always quick and helpful replies).

1 Like