Background
Until recently I’ve had to dump the entire syslog to the syslog server, now trying to begin using Filebeat collector for macOS and Graylog Elastic Beats Input Plugin which one can send a specific log or set of logs to a syslog server.
How I was doing it:
Edit the syslog conf at /etc/syslog.conf
*.* @serverip:port
Redirect Logs To A Syslog Server In OS X | Krypted.com
The caveat of this method is it dumps the entire of the syslog to the syslog server. I dislike the chattiness of syslog and would prefer to send only a particular log or set of logs that I am interested in, hence this post.
The server I was particularly interested in was averaging about 250 or so various entries and hour. A bit too much for my liking.
Sometimes it felt like the logs could easily get out of hand…
The pieces
I was lucky enough to inherit a preconfigured infrastructure of Graylog, but assuming nothing I set up my own and tested this from scratch… if have log server already setup that you can skip the configuration of server…
I am sure this well documented somewhere else too, this process was mostly for me to better understand 1.) Logging service in general, 2.) MacOS logging practices and 3.) Assessing the plausibility of using Beats or similar for a backend to forward logs in a package able, deployable fashion.
Before you start:
Mac OS VM for testing, I use VMware either local or a remote server (ESXi) for my MacOS testing.
Graylog preconfigured OVA (Download)
Graylog Elastic Beats Input Plugin (Included in v2 of Graylog, may not need this)
Generally the flow of information will look something like this:
- Log is written by .app or service
- File collector then forwards files to Beats input on Graylog server
- Beats input plug allows for any beats File collector source to be treated as any TCP/UDP log dump.
Testing:
Graylog Server
This is very well documented in Graylog’s docs-
- Setting up from an OVA
- Download and run OVA in whatever virtual appliance host you’d like
- Make changes to defaults as needed.
- Install Beats plugin (If needed)
- Get Beats plugin for graylog
-
mv to /opt/graylog/plugin/
- restart graylog
-
graylog-ctl restart
-
- Setup input
Client (macOS)
- Downloaded filebeat-5.0.2-darwin-x86_64.tar (or current)
- Unzip
- Modify lines in yaml file:
- input_type: log #uncomment paths: - /var/log/install.log #uncomment I changed to install.log for specific log testing, but you could set it to whatever you'd like. output.elasticsearch: #uncomment hosts: ["URL:PORT"] #change to server ip and port make sure it aligns to you input configuration
Once changes are made you can start the forwarder, by:
-
sudo ./filebeat -e -c filebeat.yml
More considerations & To Dos
- Automated start/stop of forwarder
- I’d like to figure out (or find someone who has) how to auto-start the filebeat service
- As well as bundling in a deployable pkg yo distribute to a large number of clients
- Further granularity/ filtering at the Graylog Level
Reference:
Grayling Documentation “Sending in log data”
Graylog Elastic Beats Input Plugin
Elastic Filebeat Collector (Mac | Win | Linux)
https://www.reddit.com/r/funny/comments/5ft0hi/just_found_this_log_on_my_server_should_i_be/
[…] In my previous post I explained why I set out to dig more into logging and how I got a proof of concept of how to deploy a system to forward particular log files to to a syslog server. […]
Hi Lucas, thank yuo for your post, I am wondering what port do you set it up on filebeat.yum because I tried graylog(ip):5044 and I have nothing comming up..can you please let me know? I have another imput with all logs from my macbook pro comming.
Thanks in advance
Alex- thanks for the read-
Firstly, make sure you have an input setup on graylog for this input type, (http://docs.graylog.org/en/2.4/pages/sending_data.html) that “input” will determine the port you need to send to via the client. These inputs are the first layer of organizing logs from sources.
Secondly, during the setup of the input process, note the tls settings you changed (if any).
Lastly, make sure you have no firewall rules in your env prevent communication on whatever port is specified in your config.
My output section in the yaml was no more than:
output.logstash:
hosts: [“IP:5044”]
More here(https://www.elastic.co/guide/en/beats/filebeat/current/configuring-howto-filebeat.html)
You can find me, and a ton or other great admins on slack as well – macadmins.org
Thanks Lucas Again, what i am trying to do is to get the logs from an Expecific app on Mac the path from logs are located in /var/log/Defendpoint/audit.log, I have my Graylog install and works good even I have already all system logs from this Laptop comming but they are a lot, i just need from this app (Avecto) this is a copy from my filebeat.yum: (I will appreciate it if you take a look)
###################### Filebeat Configuration #########################
filebeat.prospectors:
– input_type: log
paths:
– /var/log/Defendpoint/audit.log
# The regexp Pattern that has to be matched. The example pattern matches all lines starting with [
multiline.pattern: ^\[
# Defines if the pattern set under pattern should be negated or not. Default is false.
multiline.negate: true
# Match can be set to “after” or “before”. It is used to define if lines should be append to a pattern
# that was (not) matched before or after or as long as a pattern is not matched based on negate.
multiline.match: after
#============================= Filebeat modules ===============================
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 3
#================================ Outputs =====================================
#————————– Elasticsearch output ——————————
output.elasticsearch:
# Array of hosts to connect to.
hosts: [“{IP}:9200”]
#—————————– Logstash output ——————————–
#output.logstash:
# The Logstash hosts
hosts: [“{IP}}:5044”]
Alex, first- I cleaned up your comment a bit, there was some sensitive (maybe?) info included, I omitted any #commented lines as well for readability.
An initial config example would be along the lines of this: filebeat-example.yml (https://gist.github.com/LucasjHall/37792ab98b96c46d416975b87a176fd6#file-filebeat-example-yml)
I’d start simple, and then add more tuning from there. I suggest using the logging.* options for visibility during troubleshooting.
You can see more in the following post as well. https://lucasjhall.com/2017/01/24/packaging-filebeat-macos/
Thanks Lucas, it works..I really apreciate–Great Blog…i will reccomend it. 🙂