Packaging Filebeat on macOS

In a previous post I explained why I set out to dig more into logging and how I got a proof of concept of how to deploy a system to forward particular log files to to a syslog server.

This post is more about bundling it all up in a way I could easily deploy (.pkg).

Edit: I didn't explicitly state this was for testing, I do plan on moving/bundling and placing in a place that it better for an environment that say would interact with an end user, thats not this! Just what I need to get it onto some machines for testing.

I am not going to get into the ins and outs of creating packages. There are many other people who've wrote far more elegant.

Getting the pieces

In my last post and via Beats documentation the extent of launching Beats is

sudo ./filebeat -e -c filebeat.yml

but I left the "-e" off this when I transitioned it into a .plist which

-e     Log to stderr and disable syslog/file output

so the .plist which we will call co.elastic.filebeat.plist, ends up looking like this:

<?xml version="1.0" encoding="UTF-8"?>
 <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
 <plist version="1.0">
 <dict>
 <key>Label</key>
 <string>co.elastic.filebeat</string>
 <key>ProgramArguments</key>
 <array>
 <string>/Applications/Filebeat/filebeat</string>
 <string>-c</string>
 <string>/Applications/Filebeat/filebeat.yml</string>
 </array>
 <key>KeepAlive</key>
 <true/>
 </dict>
 </plist>

Before I continue I did want to make a note that I did some more research to the configuration filebeat.yaml and found out a couple neat items, you can array log files and you can specify multiple prospectors. But wait aren't those the same things? Look at this example filebeat.yaml:

filebeat.prospectors:

- input_type: log
 paths:
 - /var/log/install.log
 - /var/log/accountpolicy.log
 - input_type: log
 paths:
 - /var/log/system.log
 include_lines: ["sshd","screensharingd"]

output.logstash:
 hosts: ["hostip:port"]

What this does is it sends [all]{style=”text-decoration:underline;”} entries from install.log and accountpolicy.log to the syslog server.

AND then watches the syslog for any messages [containing]{style=”text-decoration:underline;”} sshd and screensharingd.

Pretty nifty, the documentation on configuring prospectors has a lot of neat features, even regex options that I may explore later on...

Assembly

So I started with the contents of "filebeat-5.1.1-darwin-x86_64" which I downloaded from Elastic's site.

Get the pieces

  • filebeat-5.1.1-darwin-x86_64/
  • (custom) filebeat.yaml
    • Which I placed into the filebeat-5.1.1-darwin-x86_64 directory
    • And then I renamed 'Filebeat'
  • (custom) .plist (I called mine: co.elastic.filebeat.plist`
  • I also downloaded a 'B' icon that I found by scanning through the Elastic site to use as an icon. Just to put a little polish on the folder.

Put the pieces into Place

filebeat-5.1.1-darwin-x86_64 directory

  1. I renamed "filebeat-5.1.1-darwin-x86_64" to "Filebeat"
  2. I placed the folder into the /Applications/ Directory and made sure it had the proper permissions
  3. I then found the afore mentioned B.png
    1. Opened in preview
      • b png screenshot.png{.alignnone .wp-image-1771 width=”311” height=”319”}
    2. cmd+a (select all), cmd+c (copy)
    3. Then I do a get info on the Filebeat folder, cmd+i,
      • get-info{.alignnone .size-full .wp-image-1779 width=”98” height=”48”}
    4. Then cmd+v (paste)
      • paste{.alignnone .size-full .wp-image-1780 width=”95” height=”44”}
    5. Gives us a little more polished folder icon.

So at this point we have a "app," well a folder that hosts the exec needed, next thing to do is to place our config, filebeat.yaml, into the /Applications/Filebeat/ directory. (Or modify the existing one.)

Next we will place the Launchd .plist we created earlier, co.elastic.filebeat.plist, in /Library/LaunchDaemons/ but wait, theres more. If you've never done much with Launchd I encourage you to rtm. To actually get this to load with out a restart one would need to:

launchctl load /Library/LaunchDaemons/co.elastic.filebeat.plist

Also make sure this has the proper permissions:

-rw-r--r--   root:wheel

 

And feel free to load it, this is a great point to test the setup. I am not going to touch on Graylog or the Beats Input, as I looked at it in my previous post, I will say in Graylog 2.1.2, the .ova you can download to test, Beats input is included, so no additional loading of a .jar file is needed.

Packaging

Brief review

  • Filebeat folder, with custom filebeat.yml config is in place
  • co.elastic.filebeat.plist Launchd is in place.

Packages (How I did it), start a new "Raw Distribution"

  1. Project
    • Name, path and exclusions
    • packages-1-project{.alignnone .size-full .wp-image-1834 width=”1138” height=”747”}
  2. Settings
    • ID and version for your development reference
    • packages-2-settings{.alignnone .size-full .wp-image-1835 width=”1138” height=”747”}
  3. Payload
    • The afore mentioned items in their locations
    • packages-3-payload{.alignnone .size-full .wp-image-1836 width=”1138” height=”747”}
  4. Scripts
    • This is a point where we can change schools of thoughts, you have two options.
      1. Include a script to load the launchd here
      2. Don't include said script, and have it run by a pkg management client you may use.
    • packages-4-scripts{.alignnone .size-full .wp-image-1837 width=”1138” height=”747”}
  5. Comments
    • I leave my self reminders in the comments during development
    • packages-5-comments{.alignnone .size-full .wp-image-1838 width=”1138” height=”747”}

Build! Build! Build!

filebeat-build{.alignnone .size-full .wp-image-1852 width=”990” height=”851”}

I used Suspicious Package here to show you what it looks like after the build...

So there it is... plenty more to test around with as time permits... but a good start.

What's next?

Testing.

  • Whats the impact/implications on...
    • Machine
    • Network
  • Do I need everything in that /Filebeat/ directoy?

More Testing.

  • Update just the yaml for future versions?

Even more testing.

  • What logs do Ii really want?
    • Of those logs do I want to exclude or include any more items!?
    • Do YOU know? I haven't a clue.

###

Uninstall

I also  made this handy uninstall script for testing as well:

#!/bin/bash
#filebeat testing quick cleanup

#unload launchd if its running
/bin/launchctl unload /Library/LaunchDaemons/co.elastic.filebeat.plist

#remove app folder
/bin/rm -rf /Applications/Filebeat
#remove the launchd
/bin/rm /Library/LaunchDaemons/co.elastic.filebeat.plist
#remove receipts, I don't use in production if I can avoid it 
/bin/rm /private/var/db/receipts/com.yourinstitution.pkg.Filebeat.bom
/bin/rm /private/var/db/receipts/com.yourinstitution.pkg.Filebeat.plist

UPDATEs

2017-03-17

So a few edits I've made since I was working on this a few months ago.

  1. Install location
    1. I ended up putting the application into /Library/Filebeat  for a cleaner, unobtrusive install
  2. Folder GFX
    1. Point 1 means I no longer need to make it pretty, so I dropped the folder graphic
  3. Launchd Auto Load
    1. I deployed this to a small # of machines and manually installed the pkg, and then loaded the launch daemon manually as well.
    2. This also allowed me to test the config locally before adding it to load at launch., I had some firewall rules and other items I needed to ensure weren't conflicting so it ended up not being quite as "set it and forget it" as I once set out for it to be-
  4. 2 months later
    1. Works great. Planning on a followup, specifically about the graylog input, notification and extractors side of things.

Referenced materials

Installing Filebeat | Elastic 

Deploying Filebeat on MacOS X | Elastic Forums 

Creating Launchd Jobs | Apple Developer

Packages | Whitebox

Suspicious Package | Mothers Ruin