Event Driven Client Management

Tue, 18 May 2021 19:11:11 +0000

A consideration on a code based, scalable, event driven client management framework…

What are we trying to accomplish here?

I investigate the situation where we, as system administrators, find ourselves in a vastly SaaS driven world which derives many points of truth from across many different systems. How do we manage those things centrally? How do we shift from an actively managed posture, to a passive (or event driven posture), and is there a way to meet in the middle? Is there “one tool to rule them all” or are we siloed by platform? I think to pull myself back, I am just going to outline this idea, conclusion and how I arrived at it in terms of macOS and how we can embrace the inevitable, and build systems around what is currently the “only way” to manage macOS devices. And to do that we must…

Understand the macOS management ecosystem and its current trajectory

Apple has set the direction with managing macOS devices. What I try to reiterate to folks in conversation is that the MDM spec is the MDM spec. Meaning Apple, like everywhere else in their ecosystem is defining the “rules of the game” to interact with their devices in a managed way. When setting out to “look for an mdm” I think it’s good to constantly remind yourself that it’s the same set of API calls from the biggest to the smallest vendor, Apple only gives you a defined amount of tooling, anything else is fluff. Not saying fluff isn’t helpful, but fluff is fluff and not core to device management implications.

If we step back and look for an MDM vendor that 1.) works well, especially at the scale you’re operating and 2.) has the ability to be extensible to your desire. That path of discovery will look a lot different for a lot of people, and that’s okay, “live your truth”. I make the argument that being extensible is having an actively under development and functioning API. A good API is, in my opinion, the best feature a vendor can offer.

Why is that the best feature? I am approaching this from a perspective where we treat not only “infrastructure as code” but client “configuration as code” as well. Something increasingly difficult to do when Apple is gating particular controls that can be managed of devices via an MDM. Having an API allows for the possibility of settings being managed in a systematic way, via some system. I realize this isn’t something everyone wants, and a small business managing a few dozen machines is completely doable in a good web interface. Where this is very beneficial is when you start approaching thousands of clients and the idea of client “configuration as code” piece really begins to have benefits.

As Apple shifts this control and requirements to MDM only, this means for those items in place we are reliant on our MDM first and foremost, which is then in turn reliant on Apple and the MDM framework. The reliability and visibility into anything past what your MDM offers is minimal, to any and all vendors, and that’s just a reality of the current ecosystem in place. How do we function within those constraints while also having the ability to add robustness and logic driven workflows? In short, how do we build a flexible glue between our systems to both adhere to this new paradigm as well as have the benefits of a code driven environment? That’s what I want to know as well, and what led me on this thought experiment.

What if client management wasn’t linear?

I remember watching some videos on AWS’s EventBridge after it got announced and thinking “huh, that could be cool”, as most of my bad ideas start out. If you’re not familiar with AWS Event Bridge you can learn more here. In short it’s a central flow of json “events” (logs) that can adhere to a predefined schema, this schema you can then trigger other AWS services is a pattern match in that schema is found. The idea is that you can scale out actions horizontally, independently and quickly to other AWS services. A successor to a more linear SQS or SNS queue, arguably fulfilling an entirely different role, but I digress.

This “bad idea” was in fact, “What if client management wasn’t linear?” What if we shifted from treating the management hierarchy as config -> apply -> remediate and looked at clients as nodes within a greater ecosystem, going through various stages of their own life cycle, and allowing events that these nodes “emote” determine other actions that happen within the management ecosystem?

So it’s just a big if this then that?

Yes! No! Well sort of… isn’t all code? It would be more like a “if this then that and that and that or that and if… and if…” An oversimplification of this idea is a single stream of events propagated from any system in regards to a device, or node. From that single stream of events, we can delve into the events, in near real time, pick and choose what actions we want to take, all while archiving these node specific events for evaluating trends. This seems huge, adn overwhelming, and it was/is! The benefit of this design is you can start with the smallest process or flow and add and grow the interactions in this framework as needed, approaching it at a micro, rather than a macro, scale.

I think the idea in general is not limited to AWS or EventBridge, conceptually you could extend this idea, or framework, to any system that can parse incoming data, and then perform actions based on said triggers, such as initiating webhooks via a logging tool. I will keep exploring this framework through the lens of AWS and it’s available tooling, but in no way is that the only path to implementing this idea/framework.

To begin, we start in the middle.

The core idea of EventBridge is that there is a defined JSON schema. AWS EventBridge Schema and Registry options are quite robust, so for sake of illustration we will define our schema with some actions we wish to perform. In the life of our node we can expect lots of things to happen, but at the very core of what happens to a node, what do we want? We want to know about the event, we want to perform an action because of the event, we want to keep record of the event historically and we want to be able to look at a high level of events looking for trends.

Hey listen!

By default, any event in an Event Bridge does nothing. It effectively goes to /dev/null unless a rule is triggered. So getting started the best way to have some visibility is to set a blanket rule for any, or a subset meeting an event pattern, to send the events to Cloudwatch. You can configure a target of a specific Cloudwatch log group to receive any/all of the events. A good way to ensure the events are hitting the event bus at all.

By each extension, each trigger could have its own target log group, or you could have a blanket catch-all log group for any event passing through an event bus, that design choice is really up to you and the complexity of the event based triggers, and the desired level of visibility. You could also set a lambda target that transforms the event payload to GELF or some other logging specific format and then ships it to a unified logging service you may have at your utilization. Or perhaps just setting the TTL on cloudwatch to be a few weeks but also sending the logs to an s3 bucket for long term retention? All possibilities.

Do the thing-

What thing? All the things? AWS Event Bridge logic has an ever expanding list of available targets:

API destination
API Gateway
AWS Batch job queue
CloudWatch Logs group
CodeBuild project
CodePipeline
Amazon ECS tasks
Event bus in a different account or Region
Firehose delivery stream (Kinesis Data Firehose)
Inspector assessment template (Amazon Inspector)
Kinesis stream (Kinesis Data Streams)
AWS Lambda function
…

Wowza, that’s a lot of services (and only some of the supported ones), and it makes my head spin with the possibilities. From a system administration perspective we can utilize our event bus as just that, a single place for passing and triggering off events we care about. Will it be something for passive analysis? No. But the ability to integrate with Kinesis opens a lot of doors in that realm.

So what could this look like?

Starting off will be an illustration of a basic scenario, a device enrolls. Assuming your MDM of choice has the ability to support webhooks, you could receive that payload via API Gateway, process via Lambda, and turn around and invoke change, nothing new and a fair amount of tooling and processes exist around this idea.

In this thought experiment I want to have a less direct invocation, but the ability to process this event and then have multiple subsets of services fan out doing the needed actions for this event. So rather than a single Lambda function that is responsible for all the processing and logic of an event.

So something to this ☝🏻 design can illustrate the beginning of what I am imagining. A MDM endpoint will send a POST event to an API Gateway endpoint where it will send to the Lambda function to process the event specific to that data source. Those processors generate an event that contains all the needed information for the Event Bridge to process, here’s an over simplified example of an output of the MDM Processor to an event:

{
    "scope": "device",
    "action": "enroll",
    "device": {
        "uuid": "someSerial",
    },
    "communicate": true,
    "log": true
}

From this event we can trigger each of these individual Lambda processors that can systematically interact with the upstream services. Looking at the hypothetical triggers:

Device Enroll is triggered as scope and action match
- This process then knows to use some of the information to:
  - Update the AssetsDB with device information, such as enrollment state and time
  - Update the MDM with any Asset specific notes or device name from the AssetsDB from above
User Deprovision is not triggered
Comms Trigger is triggered as communicate is true
- This payload can then be translated to the needed block kit syntax and sent to Slack via a Lambda function that contains all the needed information to interact with Slack via the SDK, Bolt Python etc
Log Trigger is triggered as log is true
- Send the log to a specific s3 bucket as described in the trigger
- Send the log to stats processor that could translate the basic event, device enrolled to a time series data point then sent off to AWS Timestream for later analysis via the managed TSDB (time series database)

Wait, MDM Processor is on that illustration twice? Yes. With this in theory you can have as specific services as you’d like to handle this processes. I envision each processor emoting its own events, if for nothing else then centralized logging and statistics around these duties they are performing, such as:

{
    "scope": "communicate",
    "action": "slack_post",
    "info": {
        "channel": "someChanel",
        "body": "someText",
        "triggering_event": "someEvent",
    },
    "log": true
}

Now we will systematically aggregate logs in a logical fashion as well as be able to create time series data specific to the usage of any of these micro services as they are consumed. cool 😎

User Deactivation

Identity Provider processor as well! So something like:

{
    "scope": "user",
    "action": "deactivated",
    "log": true
}

This event could trigger an MDM Processor event to lookup the user’s assigned devices and automatically deactivate or lock (depending on your use case) when a user becomes deactivated.

SaaS Partners

I included an ambiguous item on the diagram “SaaS Support” simple because Event Bridge supports SaaS integrations, such as (but not limited too):

SalesForce via AppFlow
Auth0
Datadog
New Relic
Zendesk

Which allows for the skipping of the event ingestion and allows for these SaaS Partners to send information directly to your event bus. Not super helpful in the world of system administration, but time will lend to seeing if the adaptation of this integration continues.

Conclusion

As services become dispersed, information becomes harder to aggregate, system administrators are seeing those sources of truth in various locations, a single database with all the things would be amazing, but a far cry from reality.

I think an event driven client endpoint ecosystem allows for the inflexibility of these dispersed services to be aggregated by events initially, triggering downstream events and actions. An event driven architecture is something I’ve thought about in the past, triggering scripts and items from syslog servers, but it was always strenuous with a large amount of work and overhead. Not that an Event Bridge based system isn’t, it just feels more accessible as this, to some degree, is what it was designed to do… for sysadmins? probably not. But our services, infrastructure and systems need to follow suit with the direction of the industry if we want to stay flexible and scalable.

In Casablanca there is a conversation between Major Strasser (an Officer of the Luftwaffe) and Captain Renault (captain of police in Casablanca) that goes like this:

Strasser: I’m not entirely sure which side you’re on.
Renault: I have no conviction, if that’s what you mean. I blow with the wind, and the prevailing wind happens to be from Vichy.
Strasser: And if it should change?
Renault: Well, surely the Reich doesn’t permit that possibility.

I wouldn’t say I have no conviction, but I resonate with Renault’s sentiment here, in my little town, I only have the control over the systems I am permitted, and I have to do what I can to better that environment within the bounds of the “prevailing wind”, be it trends in SaaS services, changes to the MDM Specification or shifting sands of internal tooling and processes.

Thus concludes this ramble under the guise of a thought experiment.

Some References

Apple MDM
AWS EventBridge
AWS Lambda
AWS CloudWatch
AWS S3
JSON Schema

pyThings

Wed, 31 Mar 2021 20:00:00 +0000

pyThings is a Python library used to create and call x-callback-urls with the assistance of xcall.

Here is an example of creating some Projects (Epics) and Tasks (Stories) via another bit of logic (dry details of the items withheld):

This was created in haste to include in some other unnecessary automation of daily tasks and time tracking. This library supports most of the URL Schema published by Cultured Code on their documentation page here.

Add/Update a Task:

>>> from pyThings.tasks import AddTask
>>> AddTask(title='A New Task')
>>> from pyThings.tasks import UpdateTask
>>> UpdateTask(auth_token=auth_token, task_id='ID_STRING', completed=True)

Add/Update a Project:

>>> from pyThings.projects import AddProject
>>> AddProject(title='My New Project')
>>> from pyThings.projects import UpdateProject
>>> UpdateProject(auth_token=auth_token, task_id='ID_STRING', completed=True)

As I mentioned in the README, while Things Callback URL Scheme is robust, it is very write oriented. It is not currently possible to get anything from Things as far as I could find, aside from an associated x-things-id upon creation, which is needed to update any Task or Project.

A few things I am thinking about for future release are moving to something like robwalton/python-xcall which is a client for bi-directional communication with x-callback-url. As well as removing the execution of the callback url upon class creation, this was dne just to simplify the library in my context, though I realize that may not be a wide use case. The idea of creating Projects and Issues as classes is retaining the information in a diffable format, that can then be cached locally, outside of Things, to be used again (when using this library in syncing from say a project management suite via another tool).

I went as far as digging through the sqlite db Things keeps on disk, but…

TimingPy

Fri, 19 Mar 2021 08:00:00 +0000

TimingPy is a Python library to wrap the web API for Timing.app.

Timing.app is a paid macOS app for time tracking. The goals of this quick project were two fold,

1) to continue learning Python and 2) creating some tooling to bridge all of the various tools/service I use throughout the day, more on this someday (maybe).

The repository is available on Github here: lucasjhall/TimingPy

TimingPy supports all of the calls available via Web API Documentation as of 2021-02-27 but currently has no data validation. Here is a quick illustration to create a task:

>>> task = TimingPy.Task.create_new_task(
    title="New Task",
    start_date="2021-03-16T16:00:00+0000",
    end_date="2021-03-16T17:00:00+0000")

Returning task a requst obj that shold read <Response [201]> via print(). Supporing the usual request attributes. Reading details about the task created:

>>> print(task.text) 

{"data":{"self":"\/time-entries\/$ID_OF_EVENT","start_date":"2021-03-16T16:00:00.000000+00:00","end_date":"2021-03-16T17:00:00.000000+00:00","duration":3600,"project":null,"title":"New Task","notes":null,"is_running":false,"creator_name":"$YOUR_ACCOUNT_EMAIL"}}

Both start_date and end_date need to be formated in ISO8601 which this tool doesn’t account for, as of yet.

You can read more on Timing here, the Web API Documentation and Timing for Teams.

Into the Stream Deck

Sat, 27 Feb 2021 11:11:11 +0000

The Elgato Stream Deck is a nifty device aimed for customizable interface for streamers and broadcasters, and I didn’t use it for any of that… one word: macros.

tl;dr

Elgato’s Stream Deck and it’s software capabilities
100% not using it for streaming, but mostly just macros and daily automation
Basic app/script launch
Intro to Stream Deck’s profiles and application aware profiles

A little on the hardware

If you’d like you can read more about it on Elgato’s page here. It comes in three sizes, Mini (6 keys), Standard (15 keys) and XL (32 keys), it even offers a mobile version that runs on iOS and Android.

Though this devices is designed for one use, my immediate thought was macros. I have been using Keyboard Maestro from sometime and often use a key combo to initiate a function, but a deck with pictures? Maybe even gifs to trigger macros, sign me up! I wont get into my Keyboard Maestro setup as that could, and may be, a blog post all of its own.

(A lot of the following examples are synthesized from something I am currently doing, or have done, and not specific to a singular device. All that to say I don’t every recommend accessing or using personally associated apps on a work device.)

Getting Set Up

Get a Stream Deck, whatever size suits your style/desk. There is effectively a possible infinite folder structure with later Stream Deck firmware/os updates so I went with the “Standard” model as 15 keys seem a happy medium.

Snag the software for your OS and get going.

Making a plan… some basic items for illustration:

Launch Slack, possibly even users or specific channels
Launch user specific Chrome browsers (work/personal)
Some basic tooling around apps I use on the daily
Zoom (Call controls)
- Code (Documentation, text manipulation, basic git wrappers)
- VMware Fusion (VM Controls)

Down the rabbit hole

Lets take a look at some examples within the apps I mentioned above…

A note on my shorthand, when I reference something like System > Website, I am saying within Stream Deck’s configuration (sidebar) under the System header I used Website function, so on and so forth.

Slack

Like a lot a folks I live in Slack during the work day, and the quick search/channel jumping is fine, but I wanted to see if there was a way to make some shortcuts to channels/DMs I spend the most time… and it was actually pretty easy on macOS. I used an application url callback for Slack, and that looks something like this:

slack://channel?team=$TEAM_ID&id=$CHANNEL_ID

The quickest way to find these values out is load the slack team/channel you’d like to access in the browser:

https://app.slack.com/client/T04QVKUQG/C0533K6HC

The first T04QVKUQG is the team id, the second is the channel or user C0533K6HC so swap those out with the callback URL above:

slack://channel?team=T04QVKUQG&id=C0533K6HC and voila

Add that to a System > Website as the url, on clicking you should launch directly to your desired channel or user. A team id won’t change per “Slack” instance (team) so you can note this and then while in slack, right click > Copy Link on a channel name and you’ll get a url such as https://$team.slack.com/archives/C0533K6HC where the last part as we see, is the same channel identifier.

Chrome Profiles

This is another simple one, that just needs a slight bit of digging. If you use multiple Chrome profiles, you just need to determine what those profiles are and how you can reference them. In Chrome (while in the profile you seek to identify) open chrome://version/ and look for Profile Path:

Profile Path: /Users/$user/Library/Application Support/Google/Chrome/Profile 1

That is the name of the Chrome profile (Profile 1) we care about, you can test this by running in the terminal:

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --args --profile-directory='Profile 1'

and if its the first profile created, its most likely:

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --args --profile-directory='Default'

A word of warning, running --profile-directory='NonexistentProfile' Chrome will just generate a profile if one does not exist, easily remediated via the “Other People” UI box launched via your user icon.

I pipe these commands through to Keyboard Maestro, which is for sure over kill, but you could easily call them in various ways such as the configuration System > Open > App / File where this is the script you wish to run.

Application Awareness

So far we’ve talked about triggering some basic functionality with the Stream Deck, launching apps, or specific parts of apps/urls. I have a “base” profile that the stream deck goes to by default and some sub “folders” that look something like this (slimmed down for illustration):

default_profile
├── fusion
├── personal_chrome
├── slack_personal
│   ├── channel_a
│   ├── channel_b
│   ├── channel_c
│   ├── user_1
│   └── user_2
├── slack_work
│   ├── channel_a
│   ├── channel_b
│   ├── channel_c
│   ├── user_1
│   └── user_2
├── spotify
├── system_controls
│   ├── lock
│   ├── screen_brightness_preset_a
│   ├── screen_brightness_preset_b
│   ├── screen_saver
│   └── toggle_audio_out
├── work_chrome
└── zoom

Any apps such as Zoom, VMware Fusion, or Spotify I didn’t create a static sub folder of items for these apps, instead I used Stream Deck’s ability to switch to a specific profile when an app is launched, “Smart Profiles”.

I see two big benefits here:

any time an app is opened/in the foreground Stream Deck displays that profile
your not constantly navigating and in and our of sub folders trying to access different actions

Zoom & VMware Fusion

These profiles are application aware, my base profile simply launches these apps respectively, the app coming to the fore ground causes stream deck to load the profile for each. These applications I created “Hotkeys” for common keyboard commands and shortcuts for these app I find myself in a lot. In Stream Deck, System > Hotkey, you can define, name and give an icon to a hotkey trigger.

Zoom Examples

Toggle Mute ⌘ ⇧ A
Controls ^\
Start/Stop Share ⌘ ⇧ S
Chat Panel ⌘ ⇧ H
Toggle View ⌘ ⇧ W

VMware Fusion Examples

Take Snapshot ⌘ ⇧ S
Show Snapshots ⌘ S
Settings ⌘ E
Get Info ⌘ I
Show Machine Library ⇧ ⌘ L

VSCode

For VSCode I have a few shortcuts, and some script invocations, but I also want to illustrate the ability to go to a specific profile via a Stream Deck configuration. While when VSCode is in the foreground, its profile will be on the Stream Deck, but you can override this by adding a link to another profile, Stream Deck > Switch Profile. So I have created a git specific profile for a lot of wrapping around git and gh. So while in my VSCode profile I can quickly switch to Git items, and then switch back (via the same configuration, just VSCode profile) on the fly as needed. This is handy as using VSCode’s internal terminal wouldn’t trigger Stream Deck in the same way as say iTerm or Terminal.app. To visualize it as above, it would look something like this:

default_profile
├── fusion(launch app)
├── vscode(launch app)
└── zoom(launch app)

vscode
├── snapshot
├── library
└── git(switch to profile)

git
├── branch_create
├── sync_main
├── pr
├── stage
└── vscode(switch to profile)

There is also the ability to Stream Deck > Switch Profile and set Next Profile and this button will take you to the next profile as configured in your Stream Deck software.

There is also a VSCode “Action” pack available via More Actions... that includes some functionality like Create Terminal, Open Folder and Insert Snippet.

“What a wondrous boat ride…”

This is just the surface of what Stream Deck’s software itself can do, I haven’t even touched on all of the built in features, the additional “Actions” available, or the community created plugins (Unofficial Stream Deck Plugins) implemented via the SDK.

As for me, while it has some basic macro functionality, it mostly acts as a trigger for far more complex automation in underlying Python scripts or Keyboard Maestro workflows. Who knows, maybe I will dive into that in another post… no promises there.

And any icon you want to use, can even be a gif. chef kiss

Welcome to Github!

Mon, 25 Jan 2021 10:10:00 +0000

No more scroll back? I am working on determining what, if anything, will be brought to this site from my original blog on wordpress.com.

The future is static, and markdown.

WiPi - A RaspberryPi Hotspot To Go

Sun, 09 Aug 2020 00:20:10 +0000

tl;dr: A Raspberry Pi with a cellular HAT to act as a private internet access point and portable local network.

Taking paternity leave for the month of July to introduce our newborn to extended family introduced a lot of hurdles. With all going on in the world not many could be solved with some DIY tech. Being on the road for a month I thought it’d be rad to have a WiFi hotspot to connect to with a bit more configuration than just an iPhone.

In short the goals for the project were:

Portable WiFi Hotspot
Standalone internet connectivity
Bring our WiFi devices on the road and not have to setup or count on other folks / hotel / public WiFi access

Here's the gist...

The Build

Hardware

Raspberry Pi 4B 2019
- I chose a Raspberry Pi 4B for the base board for no other reason that I wanted to get an 4GB model for tinkering previously and had it to repurpose for this build.
Raspberry Pi 4G/LTE HAT Kit
- Bundled from SixFab
  - Raspberry Pi 3G/4G & LTE Base HAT1
  - Quectel: EC25 Mini PCle 4G/LTE Module
  - 1LTE Full Band PCB Antenna – u.FL Plug – 100mm
  - 1LTE - GNSS Dual u.FL Antenna - 100mm
Pelican 1200 w/foam
Anker PowerPortCube USB Cube
Journeyman Pro AC Pass through
TP Link AC750
Wyze Cam 1080p HD Indoor
- Custom Firmware Supporting RTSP

The Design

Over all pretty simple idea, a local wifi / lan with its own independent internet connection.

{.wp-image-2333}

Few items of note here

I have never done anything with cellular providers / HATS etc and a Pi platform
I wrongly assumed most cameras have rtsp support out of box
I wrongly assumed modern applications are designed to be data conscious

Setting Up the Pi

OS

Flashed and installed Raspberry Pi OS Latest (formerly known as Raspbian)

Cellular HAT

SixFabHas Links to "Complete Documentation" on the Cellular Modem Kit's Page. I will say I was impressed that at the documentation SixFab Provides. Depending on your IoT cellular provider, Sixfab ships with a SIM from Twilio, you can follow the PPP/QMI tutorials respectively.

I was kind of like, "Wait thats it?" the Twilio SIM being QMI with some inherent trust didn't need anything past the QMI tutorials Sixfab provided.

WiFi

RaspAP, billz/raspap-webgui

RaspAP lets you quickly get a WiFi access point up and running to share the connectivity of many popular Debian-based devices, including the Raspberry Pi. Our popular Quick installer creates a known-good default configuration that “just works” on all current Raspberry Pis with onboard wireless. A responsive interface gives you control over the relevant services and networking options. Advanced DHCP settings, OpenVPN client support, SSL, security audits, themes and multilingual options are included.

https://raspap.com/

Again, not a lot to say here, another great tool that "just works".

Nebula

The Pi hosting the WiFi / Cellular connections is also enrolled in my nebula network, I talk a bit more about nebula in a previous post. This allows for this pi to reach other nodes that are also connected to the network, it does not treat the local lan as an extension of the nebula network. Once enrolled I was able to ping/ssh into the Pi via its cellular network, as well as reach any other resources as allowed on the nebula network. Very slick.

Extending WiFi via AC750

The TP-Link AC750 (TL-WR902AC) has built in functionality for various Wifi modes. Setting up to extend an existing SSID, then moving it away from the Pi's location would allow for just that, an extended WiFi signal.

Assembly

::: {.wp-block-jetpack-slideshow .aligncenter effect=”slide”} ::: {.wp-block-jetpack-slideshow_container .swiper-container}

{.wp-block-jetpack-slideshow_image .wp-image-2356}
{.wp-block-jetpack-slideshow_image .wp-image-2352}
{.wp-block-jetpack-slideshow_image .wp-image-2349}
{.wp-block-jetpack-slideshow_image .wp-image-2350}
{.wp-block-jetpack-slideshow_image .wp-image-2351}
{.wp-block-jetpack-slideshow_image .wp-image-2353}

[]{.wp-block-jetpack-slideshow_button-prev .swiper-button-prev .swiper-button-white}[]{.wp-block-jetpack-slideshow_button-next .swiper-button-next .swiper-button-white}[]{.wp-block-jetpack-slideshow_button-pause}

::: {.wp-block-jetpack-slideshow_pagination .swiper-pagination .swiper-pagination-white} ::: ::: :::

Thoughts

All in all this was a fun project, not much use on the day to to day, but as a travel companion its a rad little setup. It was a fun project that will more than likely get dismantled into other projects!

Nebula - Getting Started

Sun, 16 Feb 2020 23:10:00 +0000

Open source mesh networking- cool!

Disclaimer

I had no part in the development or implementation of this tool. This blog post explores the tool from the perspective of the f/oss tool as it appears on Github for a home lab scenario, and what I've learned from getting it up and running in a said scenario.

This guide will use Ubuntu, UFW, so if those aren't your favorites, sorry but you'll still get the gist.

What Is Nebula

Most likely you know, or you wouldn't found your way here. But in the developers words:

Nebula is a scalable overlay networking tool with a focus on performance, simplicity and security. It lets you seamlessly connect computers anywhere in the world. Nebula is portable, and runs on Linux, OSX, and Windows. (Also: keep this quiet, but we have an early prototype running on iOS). It can be used to connect a small number of computers, but is also able to connect tens of thousands of computers.

Nebula incorporates a number of existing concepts like encryption, security groups, certificates, and tunneling, and each of those individual pieces existed before Nebula in various forms. What makes Nebula different to existing offerings is that it brings all of these ideas together, resulting in a sum that is greater than its individual parts.

The rest of this post will explore getting a few nodes enrolled and talking to one another, based off the public documentation "Getting Started (quickly)".

Getting Started, less quickly, with some commentary and opinions

Planning

Lighthouse

You need a Lighthouse node, something that is reachable via the public internet and will point nodes to one another.

[Nebula lighthouses allow nodes to find each other, anywhere in the world. A lighthouse is the only node in a Nebula network whose IP should not change.]{style=”color:#24292e;font-family:-apple-system, BlinkMacSystemFont,;”}[[ ]{.Apple-converted-space}]{style=”color:#24292e;font-family:-apple-system, BlinkMacSystemFont,;”}

[[The ]{.Apple-converted-space}]{style=”color:#24292e;font-family:-apple-system, BlinkMacSystemFont,;”}recommendation[[ on the Github page is:]{.Apple-converted-space}]{style=”color:#24292e;font-family:-apple-system, BlinkMacSystemFont,;”}

[Nebula lighthouses allow nodes to find each other, anywhere in the world. A lighthouse is the only node in a Nebula network whose IP should not change.]{style=”color:#24292e;font-family:-apple-system, BlinkMacSystemFont,;”}[[ ]{.Apple-converted-space}]{style=”color:#24292e;font-family:-apple-system, BlinkMacSystemFont,;”}

Which is[ sound, and very beneficial if you will have multiple nodes ]{style=”color:#24292e;font-family:-apple-system, BlinkMacSystemFont,;”}across[ locations or cloud ]{style=”color:#24292e;font-family:-apple-system, BlinkMacSystemFont,;”}providers[. ]{style=”color:#24292e;font-family:-apple-system, BlinkMacSystemFont,;”}

[This node will need:]{style=”color:#24292e;font-family:-apple-system, BlinkMacSystemFont,;”}

Reachable via the internet
Have the nebula port exposed, default udp/4242
Console/ssh access for configuration etc

Nodes

Any device that will connect to the Nebula network will be a node, including the Lighthouse, however the lighthouse's config differs slightly and will be touch on separately.

These nodes can be any supported OS, the Nebula binaries are available via the github to download or compile.

For sake illustration one node and lighthouse is required, though I'd recommend two nodes to better demonstrate the peer to peer capabilities.

Documentation & Planning

Node Organization

A good ole spreadsheet of the the IP range you choose will be handy for this- the default network in the example config is 192.168.100.1, which could work for you but it could be anything. I made a spreadsheet with 192.168.100.10.24 in column A, hostname in column B and C for noting any group associations. For sake of a home setup this was a solid base document to help me keep track of what device is assigned what IP etc. Knowing these values and associations are handy as you look to reference your infrastructure in the future, especially if its decentralized and all not directly accessible on your LAN.

Groups can be used to limit in and outbound traffic for nodes. Outbound traffic defaults to any, and inbound to specific ports and protocols. In our examples we will be looking at a few different services, SSH, Netdata and Filebeat in a later post. Each of these have a particular port / protocol associated with them, and we could make specific rules around each. For illustrations on this post we will be sticking-with icmp. An example of making an unbound rule allowing icmp from any nebula host would look something like:

[gist https://gist.github.com/lucasjhall/0e5baf2abfc0c106430677c407127a89 /]

You can look at the example config as well for some other ideas.

Enough Chat, Let's Go

Okay, so we will end up with the following:

Lighthouse1 (Ubuntu, cloud based node, routable ip)
Node1 (Ubuntu, hardware based node, internal LAN, in my case a Pi 3B +)
CA (its you, not specifically a device, as this is a mutual trust environment)
working area, specific to mgmt1 and not (at this point) enrolled in nebula

We will be taking all of our actions from mgmt1 but will not be installing Nebula on this device for this example.

So that looks like:

[gist https://gist.github.com/lucasjhall/af232532d7609e1e76d23a8a3c7e911a ]

Working Area

This will all be on mgmt1 a place to create and manage all the items for the nodes you're enrolling, also where all the sensitive stuff will live, until you deploy and remove it, ephemeralness of this stuff is up to you-

I made a directory structure that was similar to:

[gist https://gist.github.com/lucasjhall/77e4c3dbfa9a958a3c5de85d6b81459c /]

Let's Setup Our CA

I got the nebula-cert binary that was needed for the specific OS of my mgmt1 and placed it in nebula/binaries/, if all the devices are Ubuntu with the same architecture then this is all pretty easy.

./nebula-cert ca -name "Myorganization, Inc"

This will create files named[ ]{.Apple-converted-space}ca.key[ ]{.Apple-converted-space}and[ ]{.Apple-converted-space}ca.cert[ ]{.Apple-converted-space}in the current directory. The[ ]{.Apple-converted-space}ca.key[ ]{.Apple-converted-space}file is the most sensitive file you'll create, because it is the key used to sign the certificates for individual nebula nodes/hosts. Please store this file somewhere safe, preferably with strong encryption.

The ca.key is the signing key for the CA, so use this when you're creating nodes, but otherwise store it somewhere else, a vault or a password manager that supports encrypting attached files.

Once the ca.key is created you can proceed with generating nodes needed credentials. During this command you'll reference your planned naming schema afore mentioned in the spreadsheet.

Lighthouse (on mgmt1)

The docs have this example:

./nebula-cert sign -name "lighthouse1" -ip "192.168.100.1/24"

This is telling nebula-cert to create an identity and sign it where the name is "lighthouse1" and the up will be "192.168.100.1" and it will, by default be in no groups. As lighthouses can be used to for simply pointing nodes to one another this can be beneficial.

Or you can treat a cloud based lighthouse as a regular node and run additional service and tag it with any groups that may be applicable.

I then placed all the needed pieces into the lighthouse1 directory:

cp the ca.cert . (generated in the CA step)
lighthouse1.crt
lighthouse1.key
cp example config.yaml . (from the public or cloned repo)
nebula binary (from github / compiled)
nebula.service (from github)

As an aside I was initially naming config.yml to {host}.yml, this was short sighted as you start automate the starting of the process via launchctl or systemd this makes the template of the config.yml more cumbersome, changing the nebula.service for every host, NOPE. (Unless you're using a config management tool, Chef, Puppet, Salt et al, but I digress).

Configure the Lighthouse1 Nebula config.yaml per host config notes, with special attention to:

On the lighthouse node, you'll need to

ensure[ ]{.Apple-converted-space}am_lighthouse: true[ ]{.Apple-converted-space}is set.

####

Lighthouse Host Configuration (on lighthouse1)

With your choice of service provider get a machine up and patched. I chose Ubuntu 18.04 Server because it was what I was most familiar with- you do you.

I accessed my machine via the console, patched, etc. (You may want SSH or some means to transport files to the device)

I then setup my UFW to basically deny all inbound, then add a rule to except for 4242, our nebula port. Via a ufw allow 4242/udp. If you're not familiar with UFW you can read more here. Or choose the firewall of your choice. Heck have at iptables yourself, it's your life.

You can now move the entirety of that nebula/lighthouse1/ to your lighthouse1 machine, an scp -r to ~/ or whatever your preference.

From here you could try and start nebula per the docs, but remember the config is pointing to /etc/nebula/ for pki items and the like. So lets just put everything where it needs to be- And this is where we want all the pieces to end up here:

[gist https://gist.github.com/lucasjhall/b75399d35e4951e10a2d75e73e273ab7 /]

Run the binary pointed to the proper config and if no errors are reported the service has stated. So at this point our lighthouse is up and running, we can now configure our client.

Node1 (on mgmt1)

./nebula-cert sign -name "node1" -ip "192.168.100.2/24"

Configure the node Nebula config.yaml per host config notes, with special attention to:

[am_lighthouse]{.pl-ent}: [false]{.pl-c1}
[static_host_map]{.pl-ent}:, where IP is the nebula interface ip followed by real, and reachable IP of the lighthouse, [["]{.pl-pds}[192.168.100.1]{.pl-ent}["]{.pl-pds}]{.pl-s}: [["100.64.22.11:4242"]]{.pl-s}
[hosts]{.pl-ent}: where we list ip of the lighthouse on the nebula network, 192.168.100.1
reference the cloned config for in line notes on the above

As before this is telling nebula-cert to create an identity and sign it where the name is "node1" and the up will be "192.168.100.2" and it will, by default be in no groups.

I then placed all the needed pieces into the node1 directory:

cp the ca.cert . (generated in the CA step)
node1.crt
node1.key
cp example config.yaml . (from the public or cloned repo)
nebula binary (from github / compiled) (ensure its for your architecture)
nebula.service (from github)

Node1 Configuration (on node1)

Rinse and repeat with the steps in the lighthouse1 section.

I accessed my machine via the console, patched, etc. (You may want SSH or some means to transport files to the device.

You can now move the entirety of that nebula/node1/ to your lighthouse1 machine, an scp -r to ~/ or whatever your preference.

Run the binary pointed to the proper config and if no errors are reported the service has stated. So at this point our node1 is up and running, and we should see some handshakes showing we have now connected our node1/lighthouse.

Testing

At this point if you have no errors you can ping one host via the nebula interface from another, this demonstrates the connectivity over the neb0 or whatever configure interface.

Future Posts

Explore groups and Example Services
Nebula's built in sshd

Questions

Re: Changes

Fri, 20 Dec 2019 20:21:47 +0000

Last year I picked up and left Bellingham to take a CPE role at Slack. I worked out of the Denver office and bit- and now I am back in Bellingham. Slack allowed me to go full time remote (rad) and my wife and I moved back to be close to friends during our next phase of life.

Hopefully I will have more desire to post soon. Some cool stuff I'd love to share, perhaps for the impending conference season... ✌🏻

Ch... ch... changes

Tue, 01 Jan 2019 19:10:23 +0000

I am excited to announce that I have accepted a Client Platform Engineer position at Slack{.profileLink} . My family will be relocating to Denver CO at the end of January. We are deeply saddened to leave the Pacific Northwest, Bellingham, and the incredible community here, the #Cascadia Macadmins community, and the Apple Admins of Seattle and the Great Northwest.{.groupHomeHeader-groupNameLink} On the flip side we are so excited to see what awaits us in Denver and Colorful Colorado holds...

{.alignnone .wp-image-2276 width=”304” height=”182”} {.alignnone .wp-image-2277 width=”300” height=”180”}

"Touchless" macOS deployment utilizing JAMF Connect and InstallApplications

Wed, 21 Nov 2018 17:30:11 +0000

This post will take a high level view of using and MDM (SimpleMDM), JAMF Connect and InstallApplications for "touchless" deployment of macOS. This deployment method uses Munki as the primary tool for package management, and has been in place since the "NoMAD Days."

Big assumptions. You are familiar with MDM and the process surrounding macOS, and have some knowledge of InstallApplications. You are using Okta as an IdP, have the needed services and are paying/testing JAMF Connect.

The workflow:

{.alignnone .size-full .wp-image-2258 width=”980” height=”3065”}

Whats happening here...

This was born out of the following scope:

"Touchless" Deployment of macOS
Manage user accounts and provisioning
Desire for 2FA for login/SSO
Users are "managed"

User gets a machine

They power on and... Assuming your MDM workflow is setup correctly and you've configured Install Applications for your needs...

Munki (and Middleware)
Jamf Connect (Login+ Okta, LaunchAgent and Sync)
Images/Assets for branding
DEPNotify

After the initial setup process, the machine will load those aforementioned tools and assets prior to the login screen, giving your user a branded JAMF Connect Login screen. I've completely disabled the ability to create an account on the machine via the MDM.

User Account Creation

Account setup in Okta

This is controlled through JAMF Connects OIDC groups as outlined in the NoMAD Login+ Admin Guide.

Using OpenID Connect Applications to Manage Authentication NoMAD Login+Okta can use native applications created in the Okta admin dashboard to shape the login process. This allows for live management of who can authenticate at the loginwindow from the Okta admin dashboard without having to update configurations on the local machine. Also this makes it easy to work NoMAD Login+ into your existing Okta account creation workflow.

Woot! So even if a user account exists, it can only login if it is in the OIDCAccessClientID group. This is also extended by OIDCAdminClientID group which allows the user to be admin when they are logged in, and the OIDCSecondaryLoginClientID which is used to determine who can create local accounts after an initial user has been created.

JAMF Connect Also allows for the managed setting of DenyLocal which will force a login through Okta (there are LocalFallback options which won't be explored here).

As well... the Login Window mechanism knows if a user is flagged for a change of password and will redirect to an Okta captive portal.

This forces a user to change their password and setup their multifactor because the login window redirects to predefined Okta url where the Okta account settings are enforced.

Once the user has setup their MFA and reset their password via Okta, the user can then login to their machine, whichis forced to auth through Okta and use a MFA, for example it could look like:

{.alignnone .wp-image-2268 width=”260” height=”185”}

After authenticating with their username/password then verifying their mfa JAMF Connect creates the local user as prescribed by the OIDC groups as assigned to the user in Okta when the user logins to the machine for the first time.

Machine Configuration

At this point with InstallApplications already having run before login and running any additional items in the user space as needed, I have Munki run in the background to start downloading packages, but I held the actual notifications via DEPNotify until the user logged in with the JAMF Connect Sync item via its SignInCommand configuration.

{.alignnone .size-full .wp-image-2273 width=”285” height=”117”}

Which then steps the user through a customized welcome via DEPNotify...

Items of Note

Yubikey Provisioning

Yubikey's must be provisioned and added into Okta before the user logs into Okta for the first time. Using YubiKey Authentication in Okta
This restriction/limitation doesn't come into place with the Okta Push or a TOTP code

Temp password

User is provided with a temp password in on boarding materials with an assigned hardware token

DEPNotify Slack Alerts

May be over kill, but you can integrate this process into a webhooks into you DEPNotify to notify when this process is started/finished on clients
{.alignnone .size-full .wp-image-2261 width=”546” height=”72”}
{.alignnone .size-full .wp-image-2260 width=”554” height=”78”}
This will not scale well.. but if you want a little more insight into a smaller shop this is handy.
As well as if you have an optional install into the JAMF Connect provisioning...
{.alignnone .size-full .wp-image-2262 width=”550” height=”75”}

MDM Slack Alerts

Again, probably doesn't scale well... and depending on your MDM...
SimpleMDM allows for webhooks, then you can use something like this to receive notifications on device enrollments/unenrollmcnts as well
{.alignnone .wp-image-2264 width=”444” height=”55”}

Other items not explored

Filevault
- Escrowing Keys
- Securetoken Limitations
JAMF Connect "Demobilization" & AD
- This assumes you are using Okta as your IdP and machines are not bound to AD
- If you have Okta syncing from or mastered by your directory service, its not a stretch to see the potential for management here.…
InstallApplicatoins Package Delivery
- You'll need to host these items publicly for the initial enrollment to work