NOTE: This is not a HOW TO, more of a rant, the
how to that may or may not come later. This is written from the perspective of someone who manages (for the most part) their infrastructure from end to end, vm config, service management, monitoring and patching. When it comes to scale, and complexity your milage may very.
- there are some caveats this route
- its worth it
- git isn’t a one way street, you can use it as little or as much as you want
- Gitlab as a tool sets you up for more in the CI/CD realm
This post is just some gernal process / lessons learned from migrating a Munki repo to a git tracked repository
fwiw, “git” and “git-lfs” don’t need Gitlab, its just my life, and these are my thoughts.
- a version control system was huge, the benefit to see what I changed when I changed was really rad
- though I didn’t have that word for it at the time, some flow of dev/test/deploy of a repo
- multiple users
- though this still isn’t prod, I wanted the ability for multiple users to make changes, and said changes be tracked
- no more smb/sfp
- all the transport security can be done with pub/priv keys, etc
- I could keep it locally
- OR use the cloud, privately too
- The CE edition is rad (and I will do my best to link exclusively to the Community Edition docs)
- and free
- It has CI/CD built in
- (though I didn’t realize how great this was ’till later)
There were some kinks that had to be worked out, the first and maybe most obvious was managing large files with git. Gitlab has LFS support built in, which is rad, but still comes with its own nuances.
Git Large File Storage (LFS) replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server like GitHub.com or GitHub Enterprise.
Heres a handy tutorial: Getting started with Git LFS
Git LFS on Gitlab
Sounds great, and can be enabled in Gitlab easily.
LFS on Gitlab also requires you use
HTTPS for auth and transport, rather than ssh. Digging into the Gitlab administration docs, we can see the docs straight list some pretty big limitations:
Support for removing unreferenced LFS objects was added in 8.14 onwards.
LFS authentications via SSH was added with GitLab 8.12
Only compatible with the GitLFS client versions 1.1.0 and up, or 1.0.2.
The storage statistics currently count each LFS object multiple times for every project linking to it
But some stuff they don’t tell you that I found out the hard way-
LFS caches (
What I found was when Gitlab was receiving LFS file sit could cache them in a
/cache location, then move them to the configured storage location.
This was noticed when my OS disk on my VM filled. womp. Modifying the
gitlab.rb you can change the LFS storage location.
Looking at gitlab.rb.template, we have these LFS options:
### Git LFS # gitlab_rails['lfs_enabled'] = true # gitlab_rails['lfs_storage_path'] = "/var/opt/gitlab/gitlab-rails/shared/lfs-objects" # gitlab_rails['lfs_object_store_enabled'] = false # EE only # gitlab_rails['lfs_object_store_direct_upload'] = false # gitlab_rails['lfs_object_store_background_upload'] = true # gitlab_rails['lfs_object_store_proxy_download'] = false # gitlab_rails['lfs_object_store_remote_directory'] = "lfs-objects" ...
Oh nice! a
gitlab_rails['lfs_storage_path'] option, sweet, so you can store your repo on /dev/sd(whatever), this is good to know- so say- your os disk doesn’t fill…
And what about your client? Most people have Munki running with autopkg or munki admin on a macOS box. So you have to have Git-LFS on your mac, and little less supported.
LFS on macOS
You can do this via the installer git provides (macOS), which is basically the command line extension and a shell script. Or you could use brew, loads of people have loads of opinions about
brew, so don’t use it and just keep that to yourself.
Regardless you will need to initialize it-
# Update global git config $ git lfs install # Update system git config $ git lfs install
Which is like HEY GIT we going to use LFS now. But for what? and when?, touché git.
Looking into Configuring Git Large File Storage you can, once you’re in a tracked dir,
$ git lfs track "*.psd" Adding path *.psd
Which is cool, and it gets added to your
.gitattributes file, but most admins know what they’re big files in their munki repo are… so somehting like this in your
.gitattributes file may be more applicable:
*.pkg filter=lfs diff=lfs merge=lfs -text *.mpkg filter=lfs diff=lfs merge=lfs -text *.dmg filter=lfs diff=lfs merge=lfs -text
It still CACHES
Lets take a look at our mcOS git env…
$ git lfs env Endpoint=https://your.gitlab.com/group/munki_repo.git/info/lfs LocalGitDir=/your/munki_repo/.git LocalGitStorageDir=/your/munki_repo/.git LocalMediaDir=/your/munki_repo/.git/lfs/objects LocalReferenceDir= TempDir=/your/munki_repo/.git/lfs/tmp ... LfsStorageDir=/your/munki_repo/.git/lfs ...
That effectively means your local repo folder can be double its actual size. So plan accordingly.
Once its tracked it is as simple as a
git push to get those files and changes to the Gitlab server.
I know it sounds pretty negative up until this point, the benefit though, of the hump of getting it setup is git. And you can “git” as much or as little as you want.
Munki & Git
GitFileRepo – a proof-of-concept/demonstration plugin. It inherits the behavior of FileRepo and does git commits for file changes in the repo (which must be already configured/initialized as a git repo)
This is rad because once your repo is tracked it does git commits for file changes in the repo. Rad.
master Branch All Day
Once you have all your stuff in git, you can choose how you’re going to use it- I have seen a lot of benefit of a tracked repo that just simply commits changes to
prod are all contained in “munki logic” commit any changes to
master allows you to track any changes made to
Which this works, and is totally legit, and will get you a load of good info from tracked files.
Let’s Face(book) It, we need more
Or maybe you don’t but Facebook’s CPE team has a really rad option…
This is an AutoPkg wrapper script creates a separate git feature branch and puts up a commit for each item that is imported into a git-managed Munki repo.
They have a Getting Started Guide if this sounds like more of what you’re looking for-
Stuff I didn’t touch on, that I could
- catalogs, not tracking them, making them, making them with a runner
- git-fat as an option?
I personally won’t go back, there was a little tweaking to get it all sorted, but the information and tracking git provides to the munki repo is well worth it.
- I am going to try and sanitize some of the helpful CI/CD stuff I got rolling in Gitlab and talk about it here.
- Munki in the cloud stuff
- macOS monitoring?
- maybe a similar rant on osquery and fleet.io