In my previous post I explained why I set out to dig more into logging and how I got a proof of concept of how to deploy a system to forward particular log files to to a syslog server.
This post is more about bundling it all up in a way I could easily deploy (.pkg).
Edit: I didn’t explicitly state this was for testing, I do plan on moving/bundling and placing in a place that it better for an environment that say would interact with an end user, thats not this! Just what I need to get it onto some machines for testing.
I am not going to get into the ins and outs of creating packages. There are many other people who’ve wrote far more elegant.
- I use Packages from WhiteBox
- Some good videos:
- Packaging Mac Apps with The Luggage – Jeremy Reichman (video)
- Practical Packaging – Matt Willmore (video)
- Also helpful is Apple’s Developer Docs on Creating Launch Jobs
Getting the pieces
In my last post and via Beats documentation the extent of launching Beats is
sudo ./filebeat -e -c filebeat.yml
but I left the “-e” off this when I transitioned it into a .plist which
-e Log to stderr and disable syslog/file output
so the .plist which we will call co.elastic.filebeat.plist, ends up looking like this:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>co.elastic.filebeat</string> <key>ProgramArguments</key> <array> <string>/Applications/Filebeat/filebeat</string> <string>-c</string> <string>/Applications/Filebeat/filebeat.yml</string> </array> <key>KeepAlive</key> <true/> </dict> </plist>
Before I continue I did want to make a note that I did some more research to the configuration filebeat.yaml and found out a couple neat items, you can array log files and you can specify multiple prospectors. But wait aren’t those the same things? Look at this example filebeat.yaml:
filebeat.prospectors: - input_type: log paths: - /var/log/install.log - /var/log/accountpolicy.log - input_type: log paths: - /var/log/system.log include_lines: ["sshd","screensharingd"] output.logstash: hosts: ["hostip:port"]
What this does is it sends all entries from install.log and accountpolicy.log to the syslog server.
AND then watches the syslog for any messages containing sshd and screensharingd.
Pretty nifty, the documentation on configuring prospectors has a lot of neat features, even regex options that I may explore later on…
Assembly
So I started with the contents of “filebeat-5.1.1-darwin-x86_64” which I downloaded from Elastic’s site.
Get the pieces
- filebeat-5.1.1-darwin-x86_64/
- (custom) filebeat.yaml
- Which I placed into the filebeat-5.1.1-darwin-x86_64 directory
- And then I renamed ‘Filebeat’
- (custom) .plist (I called mine: co.elastic.filebeat.plist`
- I also downloaded a ‘B’ icon that I found by scanning through the Elastic site to use as an icon. Just to put a little polish on the folder.
Put the pieces into Place
filebeat-5.1.1-darwin-x86_64 directory
- I renamed “filebeat-5.1.1-darwin-x86_64” to “Filebeat”
- I placed the folder into the /Applications/ Directory and made sure it had the proper permissions
- I then found the afore mentioned B.png
- Opened in preview
- cmd+a (select all), cmd+c (copy)
- Then I do a get info on the Filebeat folder, cmd+i,
- Then cmd+v (paste)
- Gives us a little more polished folder icon.
- Opened in preview
So at this point we have a “app,” well a folder that hosts the exec needed, next thing to do is to place our config, filebeat.yaml, into the /Applications/Filebeat/ directory. (Or modify the existing one.)
Next we will place the Launchd .plist we created earlier, co.elastic.filebeat.plist, in /Library/LaunchDaemons/ but wait, theres more. If you’ve never done much with Launchd I encourage you to rtm. To actually get this to load with out a restart one would need to:
launchctl load /Library/LaunchDaemons/co.elastic.filebeat.plist
Also make sure this has the proper permissions:
-rw-r--r-- root:wheel
And feel free to load it, this is a great point to test the setup. I am not going to touch on Graylog or the Beats Input, as I looked at it in my previous post, I will say in Graylog 2.1.2, the .ova you can download to test, Beats input is included, so no additional loading of a .jar file is needed.
Packaging
Brief review
- Filebeat folder, with custom filebeat.yml config is in place
- co.elastic.filebeat.plist Launchd is in place.
Packages (How I did it), start a new “Raw Distribution”
- Project
- Name, path and exclusions
- Settings
- ID and version for your development reference
- Payload
- The afore mentioned items in their locations
- Scripts
- This is a point where we can change schools of thoughts, you have two options.
- Include a script to load the launchd here
- Don’t include said script, and have it run by a pkg management client you may use.
- This is a point where we can change schools of thoughts, you have two options.
- Comments
- I leave my self reminders in the comments during development
Build! Build! Build!
I used Suspicious Package here to show you what it looks like after the build…
So there it is… plenty more to test around with as time permits… but a good start.
What’s next?
Testing.
- Whats the impact/implications on…
- Machine
- Network
- Do I need everything in that /Filebeat/ directoy?
More Testing.
- Update just the yaml for future versions?
Even more testing.
- What logs do Ii really want?
- Of those logs do I want to exclude or include any more items!?
- Do YOU know? I haven’t a clue.
Uninstall
I also made this handy uninstall script for testing as well:
#!/bin/bash #filebeat testing quick cleanup #unload launchd if its running /bin/launchctl unload /Library/LaunchDaemons/co.elastic.filebeat.plist #remove app folder /bin/rm -rf /Applications/Filebeat #remove the launchd /bin/rm /Library/LaunchDaemons/co.elastic.filebeat.plist #remove receipts, I don't use in production if I can avoid it /bin/rm /private/var/db/receipts/com.yourinstitution.pkg.Filebeat.bom /bin/rm /private/var/db/receipts/com.yourinstitution.pkg.Filebeat.plist
UPDATEs
2017-03-17
So a few edits I’ve made since I was working on this a few months ago.
- Install location
- I ended up putting the application into /Library/Filebeat for a cleaner, unobtrusive install
- Folder GFX
- Point 1 means I no longer need to make it pretty, so I dropped the folder graphic
- Launchd Auto Load
- I deployed this to a small # of machines and manually installed the pkg, and then loaded the launch daemon manually as well.
- This also allowed me to test the config locally before adding it to load at launch., I had some firewall rules and other items I needed to ensure weren’t conflicting so it ended up not being quite as “set it and forget it” as I once set out for it to be-
- 2 months later
- Works great. Planning on a followup, specifically about the graylog input, notification and extractors side of things.
Referenced materials
Deploying Filebeat on MacOS X | Elastic Forums
Any updates on this. Are you still using this approach? I am finding the Homebrew install just does not work very well.
This is really helpful. Thank you. Not much online about macOS and Filebeat. I have this working and sending logs to GrayLog but with one issue, it does not survive a reboot. I have my .plist in Library/LaunchAgents and set to root:wheel but i have to manuall start filebeat after a boot.