Introduction

The mcfg crate and command-line tool, implements a simple Machine Configurator or meta-package manager to keep desktop environments the same across machines and wherever possible across operating systems. The tool makes use of existing package managers such as homebrew, apt, or yum. It allows for packages to be grouped into package sets which are the units of management and then package sets into groups for simple organization.

The tool keeps all of it's package sets organized in a repository which just happens to be a Git repo and so can be versioned and easily shared between machines. It allows for the specification of different installer tools that will be used to do actual package management, so the user doesn't need to remember specific command-lines or other details. This repo can also include any additional scripts or tools the user needs, and the execution of the package set includes a set of environment variables to allow scripts to run without knowing any O/S or machine specific paths or other details.

Goals and non-goals

The intent is to provide a light-weigh way to describe the installation of a lot of related packages, scripts, and customizations that comprise a machine environment. Specifically this was developed for keeping developer desktops as close as possible between different laptop or desktop machines and between Linux and macOS systems.

The intent is also to not:

Be a package manager itself; package sets are simply logical higher-level grouping of packages but rely solely on the underlying package manager.
Be a sync mechanism; the CLI does not sync automatically, refreshing the package repository and running update and install actions are still manual steps.
Be atomic; there is no roll-back mechanism on failure.

Current status

Basic operations working, not yet ready for more use than that.

Getting Started

To use the mcfg tool you need a package repository and an installer registry. Your package repository is a directory containing the configuration you want to use for your machine, the installer registry provides details of the tools used to actually install individual packages. The tool expects that the package repository is also a Git repository, this being the mechanism to share your repository across machines. For first use, the tool has an init command that we will demonstrate below.

Install

TBD

Initialize your repository

To initialize a new repository, on a new machine, using the system defaults for configuration and data paths the following is all that is necessary.

$ mcfg init
1. Creating local directory for repository
2. Initializing Git repository
3. Creating repository '.config' directory
4. Creating repository '.local' directory
5. Creating '00-installers/homebrew' package set
6. Creating '00-installers/homebrew-services' package set
7. Creating 'example/hello world' package set
8. Creating standard installer registry file
9. Creating package install log file
Done.

Step number 2 is important, after creating the repository directory it will perform the equivalent of a git init command. This sets up the versioning for the repository but obviously as this repository has no upstream origin we can't push changes until we make that connection.

Alternatively you can provide the URL to an existing Git repository which will be cloned into the package repository directory.

$ mcfg init -r https://github.com/simon/mcfg-repo.git
1. Creating local directory for repository
2. Cloning <https://github.com/simon/mcfg-repo.git> into repository
3. Creating repository '.config' directory
4. Creating repository '.local' directory
5. Creating '00-installers/homebrew' package set
6. Creating '00-installers/homebrew-services' package set
7. Creating 'example/hello world' package set
8. Creating standard installer registry file
9. Creating package install log file

Finally, if you would rather keep your actual Git repository in a known place outside the standard directory structure you can specify a local directory (with the -l argument) and a symlink will be created from the standard repository location to your local Git directory.

$ mcfg init -r https://github.com/fakeuser/mcfg-repo.git -l $HOME/mcfg-repo

Paths

To show all the paths that the tool uses, the paths command will list them all.

$ mcfg paths
Package Repository path:
	"/Users/simon/Library/Application Support/mcfg/repository"
Package Repository symlinked to:
	"/Users/simon/dotfiles-2"
Package Repository config file path:
	"/Users/simon/Library/Application Support/mcfg/repository/.config"
Package Repository local file path:
	"/Users/simon/Library/Application Support/mcfg/repository/.local"
Installer Registry path:
	"/Users/simon/Library/Application Support/mcfg/installers.yml"
Package Installer log file path:
	"/Users/simon/Library/Logs/mcfg/install-log.sql"

Add Package Sets

The command add <group> <package-set> will create a new group if one doesn't already exist, a directory for the package set name and finally a package-set.yml file within this. If the -a/--as-file flag is set the tool will create a new group if one doesn't already exist, and a file name + .yml. After creating the package set the tool will execute an editor to edit the template package set file it created.

Alternatively the shell command can be used to start an interactive shell in the repository document so you can create groups and package set files more manually.

Install!

Once you

$ mcfg install

The CLI tool

$ mcfg help
mcfg 0.1.0
Machine configurator.

USAGE:
    mcfg [FLAGS] <SUBCOMMAND>

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information
    -v, --verbose    The level of logging to perform; from off to trace

SUBCOMMANDS:
    add            Add a new package-set to the local repository
    edit           Add an existing package-set in the local repository
    help           Prints this message or the help of the given subcommand(s)
    history        Show a history of install actions on the local machine
    init           Initialize a repository to manage package-set installs
    install        Install package-sets as described in the local repository
    installers     Edit the current installer registry file
    link-files     Link any files specified in package-sets as described in the local repository
    list           List package-sets in the local repository
    paths          Show current path locations
    refresh        Refresh the current repository
    remove         Remove an existing package-set from the local repository
    shell          Run a shell in the repository directory, with a basic script environment
    uninstall      Uninstall package-sets as described in the local repository
    update         Update package-sets as described in the local repository
    update-self    Show the current configuration

These can be grouped into those that 1) act on the package repository, 2) those that act on package sets, and 3) those that act on the installer registry.

Package repository commands

add a new package set to the repository; either creating a directory for the package set with a single file named package-set.yml, or if the -a/--as-file flag is set the file will be named for the package-set in the group directory.

edit an existing package set in the repository, this will look for a file in the following order:

{{group}}/{{package_set}}/package-set.yml
{{group}}/{{package_set}}.yml

init-ialize the repository, creating the repository, adding the default installer registry, and log file.

list the repository contents, as a hierarchy with groups and package sets. By default it will list all groups, the -g/--group argument can be set to list only the contents of the named group.

show the configured paths for the current package repository, installer registry, and log file.

remove an existing package set from the repository.

refresh the Git repository.

Run a shell within the package repository directory, with the default set of script environment variables set. This is useful for testing scripts and doing repository edit/Git actions.

Package set commands

All the following commands take both -g/--group and -p/--package-set arguments, resulting in the following behavior:

If neither is set the tool attempts to act on all groups, and all required package sets in each group.
If only the group is specified the tool attempts to act on all required package sets in thee specified group.
If both are specified, the tool attempts to act on the specified package set in the specified group and will also act even if the package set is marked as optional.

install the package set(s); this will attempt to install even if previously installed, and the behavior of such is dependent on the installer.

link-files specified in the package set(s), performing no other actions - specifically neither the run-before or run-after script strings are run.

uninstall package set(s) from the repository; the behavior of this if the package is not previously installed is dependent on the installer.

update package set(s) to their latest version; the behavior of this if the package is not previously installed is dependent on the installer.

Installer commands

Show a history of all package install actions. The -l/--limit argument can be used to return only a number of most recent entries from the log.

Edit the installers in the registry file.

Ask all installers in the registry to update-self.

The library API

TBD

The Package Set Repository

$HOME/
├─ .config/
│  └─ mcfg/
│     └─ installers.yml
└─ .local/
   └─ share/
      └─ mcfg/
         ├─ logs/
         │  └─ install-log.sql
         └─ repository/
            ├─ .config/
            ├─ .git/  
            └─ .local/

$HOME/
├─ .config/
│  └─ mcfg/
│     └─ installers.yml
├─ .local/
│  └─ share/
│     └─ mcfg/
│        ├─ logs/
│        │  └─ install-log.sql
│        └─ repository/  ->  $HOME/mcfg-repo-simon/
└─ mcfg-repo-simon/
   ├─ .config/
   ├─ .git/  
   └─ .local/

$HOME/
└─ Library/
   ├─ Application Support/
   │  └─ mcfg/
   │     ├─ installers.yml
   │     └─ repository/
   │        ├─ .config/
   │        ├─ .git/  
   │        └─ .local/
   └─ Logs/
      └─ mcfg/
         └─ install-log.sql

Example PackageRepository API


#![allow(unused_variables)]
fn main() {
use mcfg::shared::PackageRepository;

let package_repository = PackageRepository::open().unwrap();
}

Packages

Packages are described within a package set, they have the following p properties:

A name.
An optional platform specification.
An optional package kind specification.

Platforms

The platform value, typed as Option<mcfg::shared::Platform>, specifies whether a package is only applicable for one of the supported operating system and where None implies no restriction, it should be installed for all.

Package Kinds

The package kind value, typed as mcfg::shared::PackageKind, specifies the kind of installer to use for this package.

application
default
language
script

Packages Sets

A package set is described in a YAML file, usually named package-set.yml and which contains the following properties.

A name, and optional description.
A flag denoting whether the package set is optional.
An optional script line to run before any other action.
Either:
- A list of packages to be installed by their respective installers.
- A set of keyed scripts executed during different installer actions.
An optional name for an env file to link into the user's configuration space.
An optional map of files to be symbolically linked into the user's file system.
An optional script line to run after all other actions.

A number of examples are described in Example package sets appendix.

Example Package Set file

name: lux
env-file: sample.env
actions:
  packages:
    - name: lux
      kind:
        language: python
link-files:
  set-lux: "{{local-bin}}/set-lux"

Env variables

Package actions

Script actions

Run-before and run-after script strings

Env files

Link files

Package Set Groups

Naming

^([0-9]+\-)?(.*)$

A set of package-set groups, these are basically all the directories under the repository directory. Certain directories, such as .git, will be ignored.
A set of package-sets, these are within the group directories and are of one of two forms:
1. a directory containing a file with the name package-set.yml, or
2. a file in the group directory with the .yml extension.

$HOME/
└─ .local/
   └─ share/
      └─ mcfg/
         └─ repository/
            ├─ .config/
            ├─ .git/  
            ├─ .local/
            ├─ 01-operating-system/
            ├─ 02-developer-stuff/
            ├─ 03-productivity-stuff/
            └─ 04-work-stuff/

The Installer Registry

Example Installer Registry file

- name: homebrew
  platform: macos
  kind: default
  commands:
    install: "brew install {{package}}"
    uninstall: "brew uninstall {{package}}"
    update-self: "brew upgrade"
    update: "brew update {{package}}"

- name: homebrew apps
  platform: macos
  kind: application
  commands:
    install: "brew cask install {{package}}"
    uninstall: "brew cask uninstall {{package}}"
    update-self: "brew upgrade"
    update: "brew cask update {{package}}"

- name: cargo
  kind:
    language: rust
  commands:
    install: "cargo install {{package}}"
    uninstall: "cargo uninstall {{package}}"

Example InstallerRegistry API


#![allow(unused_variables)]
fn main() {
use mcfg::shared::InstallerRegistry;

let installer_registry = InstallerRegistry::open().unwrap();
}

Installing installers

TBD

Script strings

Package installers, and package sets, have a number of properties that are intended to execute commands. These are simple YAML strings, but some parsing has to be done to ensure correct handling of a number of possibilities.

A simple self-contained command, no shell interpretation or variable substitution.

- name: homebrew
  commands:
    update-self: "brew upgrade"

A simple command, but parameterized using one or more of the standard script variables.

- name: homebrew
  commands:
    install: "brew install {{package}}"

Usage in installers

Usage in package sets

Variables

All of these variables are also set as environment variables to be used inside any running script. Each variable name is upper-cased and prefixed with "MCFG_", so command_action becomes MCFG_COMMAND_ACTION.

Default variables

home - the current user's home directory, usually equivalent to $HOME.
command_log_level - the name of the current log level, if a command wishes to do any logging of it's own.
command_shell - the name of the command shell used to execute script strings.
local_download_path - the name of the user's local download directory.
platform - the value of the Platform enum.
platform_family - the operating system family, defined by Rust.
platform_os - the operating system ID, defined by Rust.
platform_arch - the system architecture ID, defined by Rust.
repo_config_path - the path within the package repository for config files.
repo_local_path - the path within the package repository for local files, including the bin directory.

Action variables

command_action - the kind of action being performed; one of install, link-files, update, or uninstall.

Package set variables

package_set_name - the name of the package set being actioned.
package_set_file - the name of the package set file, this is within package_set_path
package_set_path - the directory containing the package set file.

Package variables

package_name - the name of the package being actioned.
package_config_path - the current user's local configuration path for this package.
package_data_local_path - the current user's local data path for this package.
package_log_path - the full path to the installer log file.

User-defined variables

The mcfg Tool

Actions

An action is a unit of work executed by a command in the CLI. It is simply a rust type that implements the following trait.


#![allow(unused_variables)]
fn main() {
pub trait Action: Debug {
    /// Run this action, this assumes all information was passed to the action during creation.
    fn run(&self) -> Result<()>;
}
}

Using existing actions

HistoryAction
InitAction
InstallAction
EditInstallersAction
ListAction
ManageAction
ShowPathsAction
RefreshAction
ShellAction
UpdateSelfAction

Example calling InstallAction


#![allow(unused_variables)]
fn main() {
use mcfg::actions::InstallAction;
use mcfg::shared::Environment;

fn wrapper() {
let env = Environment::default();

let action = InstallAction::install(
    env,
    Some("work-tools".to_string()),
    Some("productivity".to_string())).unwrap();

action.run().unwrap();
}
}

Adding new actions

The following is an example Action implementation that does very little.


#![allow(unused_variables)]
fn main() {
use mcfg::actions::Action;
use mcfg::error::Result;
use mcfg::shared::Environment;

#[derive(Debug)]
pub struct ExampleAction {
    env: Environment,
}

impl Action for ExampleAction {
    fn run(&self) -> Result<()> {
        println!("ListAction::run {:?}", self);
        Ok(())
    }
}
impl ExampleAction {
    pub fn new(env: Environment) -> Result<Box<dyn Action>> {
        Ok(Box::from(ExampleAction { env }))
    }
}
}

Appendix: Example package sets

The following are package-sets that can be useful starting places.

Example: installing homebrew via curl

---
name: homebrew
platform: macos
description: macOS homebrew package manager
actions:
  scripts:
    install: "curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh | bash"

Example: setting macOS experience defaults

---
name: macos defaults
platform: macos
actions:
  scripts:
    install: >-
      defaults write com.apple.dashboard devmode YES &&
      defaults write com.apple.finder _FXShowPosixPathInTitle -bool YES &&
      defaults write com.apple.Dock showhidden -bool YES

Example: a long list of packages

name: fonts
description: all those missing fonts!
actions:
  packages:
    - name: homebrew/cask-fonts/font-fira-code
      platform: macos
      kind: application
    - name: homebrew/cask-fonts/font-fira-code-nerd-font
      platform: macos
      kind: application
    - name: font-meslo-lg
      platform: macos
      kind: application
    - name: font-meslo-lg-nerd-font
      platform: macos
      kind: application
    - name: font-linux-libertine
      platform: macos
      kind: application
    - name: fonts-powerline
      platform: linux

Example: linking files and run-after

---
name: zsh
description: the Z shell
actions:
  packages:
    - name: zsh
    - name: zsh-completions
    - name: zsh-navigation-tools
      platform: macos
link-files:
  dot-zlogin: "{{home}}/.zlogin"
  dot-zshenv: "{{home}}/.zshenv"
  dot-zshrc: "{{home}}/.zshrc"
run-after: "{{package_set_path}}/run-after"

Example: custom variables

---
name: gpg
description: Gnu Privacy Guard
env-vars:
  gpg_home: "{{home}}/.gnupg"
actions:
  packages:
    - name: gpg
    - name: pinentry-gnome3
      platform: linux
    - name: pinentry-mac
      platform: macos
link-files:
  gpg.conf: "{{gpg_home}}/gpg.conf"
  "gpg-agent-{{platform_os}}.conf": "{{gpg_home}}/gpg-agent.conf"
run-after: gpg --list-keys

Appendix: Schema for YAML

Schema for installer registry

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://github/schema/installers.json",
  "title": "Installer Registry",
  "description": "The Installer Registry file for mcfg",
  "definitions": {
    "name": {
      "$id": "#name",
      "type": "string",
      "pattern": "^[a-zA-Z0-9\\-+.@_/]+$"
    },
    "platform": {
      "$id": "#platform-kind",
      "type": "string",
      "enum": ["linux", "macos"]
    },
    "kind": {
      "$id": "#package-kind",
      "oneOf": [
        {
          "type": "string",
          "enum": [
            "application",
            "default",
            "language"
          ]
        },
        {
          "type": "object",
          "properties": {
            "language": {
              "type": "string",
              "pattern": "^[a-zA-Z0-9\\-+.@_/]+$"
            }
          },
          "required": [
            "language"
          ]
        }
      ]
    }
  },
  "type": "array",
  "items": {
    "type": "object",
    "properties": {
      "name": {
        "$ref": "#name"
      },
      "platform": {
        "$ref": "#platform-kind"
      },
      "kind": {
        "$ref": "#package-kind"
      },
      "if_exist": {
        "type": "string"
      },
      "commands": {
        "type": "object",
        "properties": {
          "install": {
            "type": "string"
          },
          "link-files": {
            "type": "string"
          },
          "uninstall": {
            "type": "string"
          },
          "update": {
            "type": "string"
          }
        }
      },
      "update-self": {
        "type": "string"
      }
    },
    "required": ["name"]
  }
}

Schema for package sets

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://github/schema/package-set.json",
  "title": "Package Set",
  "description": "A Package Set for mcfg",
  "definitions": {
    "name": {
      "$id": "#name",
      "type": "string",
      "pattern": "^[a-zA-Z0-9\\-+.@_/]+$"
    },
    "platform": {
      "$id": "#platform-kind",
      "type": "string",
      "enum": ["linux", "macos"]
    },
    "kind": {
      "$id": "#package-kind",
      "oneOf": [
        {
          "type": "string",
          "enum": [
            "application",
            "default",
            "language"
          ]
        },
        {
          "type": "object",
          "properties": {
            "language": {
              "type": "string",
              "pattern": "^[a-zA-Z0-9\\-+.@_/]+$"
            }
          },
          "required": [
            "language"
          ]
        }
      ]
    },
    "packages": {
      "$id": "#package-action",
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "name": {
            "$ref": "#name"
          },
          "platform": {
            "$ref": "#platform-kind"
          },
          "kind": {
            "$ref": "#package-kind"
          }
        },
        "required": [
          "name"
        ]
      }
    },
    "scripts": {
      "$id": "#script-action",
      "type": "object",
      "properties": {
        "install": {
          "type": "string"
        },
        "link-files": {
          "type": "string"
        },
        "uninstall": {
          "type": "string"
        },
        "update": {
          "type": "string"
        }
      }
    }
  },
  "type": "object",
  "properties": {
    "name": {
      "$ref": "#name"
    },
    "description": { "type":  "string" },
    "platform": { "$ref":  "#platform-kind" },
    "optional": { "type": "boolean" },
    "env-vars": { "type": "object" },
    "run-before": { "type":  "string" },
    "run-after": { "type":  "string" },
    "link-files": { "type": "object" },
    "env-file": { "type":  "string" },
    "actions": {
      "type": "object",
      "oneOf": [
        {
          "$ref": "#package-action"
        },
        { "$ref":  "#script-action"}
      ]

    }
  },
  "required": ["name"]
}

Appendix: Schema for install log

Installed package table

CREATE TABLE installed (
    date_time         DATETIME NOT NULL,
    package_set_group TEXT     NOT NULL,
    package_set       TEXT     NOT NULL,
    package           TEXT     NOT NULL,
    installer         TEXT     NOT NULL
);

The mcfg Book