MacOS, Beats and Graylog. Learning for better logging.

Background

Until recently I’ve had to dump the entire syslog to the syslog server, now trying to  begin using Filebeat collector for macOS  and Graylog Elastic Beats Input Plugin which one can send a specific log or set of logs to a syslog server.

How I was doing it:

Edit the syslog conf at /etc/syslog.conf

*.*                                       @serverip:port

Redirect Logs To A Syslog Server In OS X | Krypted.com

The caveat of this method is it dumps the entire of the syslog to the syslog server. I dislike the chattiness of  syslog and would prefer to send only a particular log or set of logs that I am interested in, hence this post.

The server I was particularly interested in was averaging about 250 or so various entries and hour. A bit too much for my liking.

Sometimes it felt like the logs could easily get out of hand…

logs

Just found this log on my server. Should I be worried?

 

The pieces 

I was lucky enough to inherit a preconfigured infrastructure of Graylog, but assuming nothing I set up my own and tested this from scratch… if have log server already setup that you can skip the configuration of server…

I am sure this well documented somewhere else too, this process was mostly for me to better understand 1.) Logging service in general, 2.) MacOS logging practices and 3.) Assessing the plausibility of using Beats or similar for a backend to forward logs in a package able, deployable fashion.

Before you start:

Mac OS VM for testing, I use VMware either local or a remote server (ESXi) for my MacOS testing.

Graylog preconfigured OVA (Download)

Graylog Elastic Beats Input Plugin (Included in v2 of Graylog, may not need this)

Filebeat collector for macOS

Generally the flow of information will look something like this:

 

  1. Log is written by .app or service
  2. File collector then forwards files to Beats input on Graylog server
  3. Beats input plug allows for any beats File collector source to be treated as any TCP/UDP log dump.

filebeat-overview-001

Testing: 

Graylog Server

This is very well documented in Graylog’s docs-

  1. Setting up from an OVA
    1. Download and run OVA in whatever virtual appliance host you’d like
    2. Make changes to defaults as needed.
  2. Install Beats plugin (If needed)
    1. Get Beats plugin for graylog
    2. mv to /opt/graylog/plugin/
    3. restart graylog
      • graylog-ctl restart
  3. Setup input
    1. See Graylog documentation here

Client (macOS)

  1. Downloaded filebeat-5.0.2-darwin-x86_64.tar (or current)
    1. Unzip
    2. Modify lines in yaml file:

 

- input_type: log #uncomment

paths:

- /var/log/install.log #uncomment I changed to install.log for specific log testing, but you could set it to whatever you'd like. 

output.elasticsearch: #uncomment

hosts: ["URL:PORT"] #change to server ip and port make sure it aligns to you input configuration

Once changes are made you can start the forwarder, by:

  • sudo ./filebeat -e -c filebeat.yml

More considerations & To Dos

  • Automated start/stop of forwarder
    • I’d like to figure out (or find someone who has) how to auto-start the filebeat service
    • As well as bundling in a deployable pkg yo distribute to a large number of clients
  • Further granularity/ filtering at the Graylog Level

Reference:

Filebeat Reference

Grayling Documentation “Sending in log data”

Graylog Elastic Beats Input Plugin

Elastic Filebeat Collector  (Mac | Win | Linux)

https://www.reddit.com/r/funny/comments/5ft0hi/just_found_this_log_on_my_server_should_i_be/

One thought on “MacOS, Beats and Graylog. Learning for better logging.

  1. Pingback: Packaging Filebeat on macOS | Lucas J. Hall

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s