Salt Formulas¶

Adding a Formula as a GitFS remote¶

One design goal of Salt's GitFS fileserver backend was to facilitate reusable States. GitFS is a quick and natural way to use Formulas.

1. Install any necessary dependencies and configure GitFS.

2. Add one or more Formula repository URLs as remotes in the gitfs_remotes list in the Salt Master configuration file:

   gitfs_remotes:
     - https://github.com/saltstack-formulas/apache-formula
     - https://github.com/saltstack-formulas/memcached-formula
   

   We strongly recommend forking a formula repository into your own GitHub account to avoid unexpected changes to your infrastructure.

   Many Salt Formulas are highly active repositories so pull new changes with care. Plus any additions you make to your fork can be easily sent back upstream with a quick pull request!

3. Restart the Salt master.

Beginning with the 2018.3.0 release, using formulas with GitFS is now much more convenient for deployments which use many different fileserver environments (i.e. saltenvs). Using the all_saltenvs parameter, files from a single git branch/tag will appear in all environments. See here for more information on this feature.

Adding a Formula directory manually¶

Formulas are simply directories that can be copied onto the local file system by using Git to clone the repository or by downloading and expanding a tarball or zip file of the repository. The directory structure is designed to work with file_roots in the Salt master configuration.

1. Clone or download the repository into a directory:

   mkdir -p /srv/formulas
   cd /srv/formulas
   git clone https://github.com/saltstack-formulas/apache-formula.git
   
   # or
   
   mkdir -p /srv/formulas
   cd /srv/formulas
   wget https://github.com/saltstack-formulas/apache-formula/archive/master.tar.gz
   tar xf apache-formula-master.tar.gz
   

2. Add the new directory to file_roots:

   file_roots:
     base:
       - /srv/salt
       - /srv/formulas/apache-formula
   

3. Restart the Salt Master.

Including a Formula in an existing State tree¶

Formula may be included in an existing sls file. This is often useful when a state you are writing needs to require or extend a state defined in the formula.

Here is an example of a state that uses the epel-formula in a require declaration which directs Salt to not install the python26 package until after the EPEL repository has also been installed:

include:
  - epel

python26:
  pkg.installed:
    - require:
      - pkg: epel

Including a Formula from a Top File¶

Some Formula perform completely standalone installations that are not referenced from other state files. It is usually cleanest to include these Formula directly from a Top File.

For example the easiest way to set up an OpenStack deployment on a single machine is to include the openstack-standalone-formula directly from a top.sls file:

base:
  'myopenstackmaster':
    - openstack

Quickly deploying OpenStack across several dedicated machines could also be done directly from a Top File and may look something like this:

base:
  'controller':
    - openstack.horizon
    - openstack.keystone
  'hyper-*':
    - openstack.nova
    - openstack.glance
  'storage-*':
    - openstack.swift

Configuring Formula using Pillar¶

Salt Formulas are designed to work out of the box with no additional configuration. However, many Formula support additional configuration and customization through Pillar. Examples of available options can be found in a file named pillar.example in the root directory of each Formula repository.

Using Formula with your own states¶

Remember that Formula are regular Salt States and can be used with all Salt's normal state mechanisms. Formula can be required from other States with require declarations, they can be modified using extend, they can made to watch other states with The _in version of requisites.

The following example uses the stock apache-formula alongside a custom state to create a vhost on a Debian/Ubuntu system and to reload the Apache service whenever the vhost is changed.

# Include the stock, upstream apache formula.
include:
  - apache

# Use the watch_in requisite to cause the apache service state to reload
# apache whenever the my-example-com-vhost state changes.
my-example-com-vhost:
  file:
    - managed
    - name: /etc/apache2/sites-available/my-example-com
    - watch_in:
      - service: apache

Don't be shy to read through the source for each Formula!

Reporting problems & making additions¶

Each Formula is a separate repository on GitHub. If you encounter a bug with a Formula please file an issue in the respective repository! Send fixes and additions as a pull request. Add tips and tricks to the repository wiki.

Get involved creating new Formulas¶

The best way to create new Formula repositories for now is to create a repository in your own account on GitHub and notify a SaltStack employee when it is ready. We will add you to the Contributors team on the saltstack-formulas organization and help you transfer the repository over. Ping a SaltStack employee on IRC (#salt on Freenode), join the #formulas channel on the salt-slack (bridged to #saltstack-formulas on Freenode) or send an email to the salt-users mailing list. Note that the IRC logs are available at https://freenode.logbot.info/salt and https://freenode.logbot.info/saltstack-formulas respectively.

There are a lot of repositories in that organization! Team members can manage which repositories they are subscribed to on GitHub's watching page: https://github.com/watching.

Members of the Contributors team are welcome to participate in reviewing pull requests across the Organization. Some repositories will have regular contributors and some repositories will not. As you get involved in a repository be sure to communicate with any other contributors there on pull requests that are large or have breaking changes.

In general it is best to have another Contributor review and merge any pull requests that you open. Feel free to at-mention other regular contributors to a repository and request a review. However, there are a lot of formula repositories so if a repository does not yet have regular contributors or if your pull request has stayed open for more than a couple days feel free to "selfie-merge" your own pull request.

Style¶

Maintainability, readability, and reusability are all marks of a good Salt sls file. This section contains several suggestions and examples.

# Deploy the stable master branch unless version overridden by passing
# Pillar at the CLI or via the Reactor.

deploy_myapp:
  git.latest:
    - name: git@github.com/myco/myapp.git
    - version: {{ salt.pillar.get('myapp:version', 'master') }}

# Do
apache:
  pkg.installed:
    - name: httpd

# Don't
apache:
  pkg:
    - installed
    - name: httpd

# Start with a high-level description of the current sls file.
# Explain the scope of what it will do or manage.

# Comment individual states as necessary.
update_a_config_file:
  # Provide details on why an unusual choice was made. For example:
  #
  # This template is fetched from a third-party and does not fit our
  # company norm of using Jinja. This must be processed using Mako.
  file.managed:
    - name: /path/to/file.cfg
    - source: salt://path/to/file.cfg.template
    - template: mako

  # Provide a description or explanation that did not fit within the state
  # ID. For example:
  #
  # Update the application's last-deployed timestamp.
  # This is a workaround until Bob configures Jenkins to automate RPM
  # builds of the app.
  cmd.run:
    # FIXME: Joe needs this to run on Windows by next quarter. Switch these
    # from shell commands to Salt's file.managed and file.replace state
    # modules.
    - name: |
        touch /path/to/file_last_updated
        sed -e 's/foo/bar/g' /path/to/file_environment
    - onchanges:
      - file: a_config_file

# BAD EXAMPLE
# The Jinja in this YAML comment is still executed!
# {% set apache_is_installed = 'apache' in salt.pkg.list_pkgs() %}

# GOOD EXAMPLE
# The Jinja in this Jinja comment will not be executed.
{# {% set apache_is_installed = 'apache' in salt.pkg.list_pkgs() %} #}

Easy on the Jinja!¶

Jinja templating provides vast flexibility and power when building Salt sls files. It can also create an unmaintainable tangle of logic and data. Speaking broadly, Jinja is best used when kept apart from the states (as much as is possible).

Below are guidelines and examples of how Jinja can be used effectively.

Jinja -> YAML -> Highstate -> low state -> execution

{% set is_selinux_enabled = salt.cmd.run('sestatus') == '1' %}

{% set some_data = salt.pillar.get('some_data', {'sane default': True}) %}

{# or #}

{% import_yaml 'path/to/file.yaml' as some_data %}

{# or #}

{% import_json 'path/to/file.json' as some_data %}

{# or #}

{% import_text 'path/to/ssh_key.pub' as ssh_pub_key %}

{# or #}

{% from 'path/to/other_file.jinja' import some_data with context %}

{% for user in salt.pillar.get('list_of_users', []) %}

{# Ensure unique state IDs when looping. #}
{{ user.name }}-{{ loop.index }}:
  user.present:
    - name: {{ user.name }}
    - shell: {{ user.shell }}

{% endfor %}

{# ---- Bad example ---- #}

apache:
  pkg.installed:
    {% if grains.os_family == 'RedHat' %}
    - name: httpd
    {% elif grains.os_family == 'Debian' %}
    - name: apache2
    {% endif %}

{# ---- Better example ---- #}

{% if grains.os_family == 'RedHat' %}
{% set name = 'httpd' %}
{% elif grains.os_family == 'Debian' %}
{% set name = 'apache2' %}
{% endif %}

 apache:
  pkg.installed:
    - name: {{ name }}

{# ---- Good example ---- #}

{% set name = {
    'RedHat': 'httpd',
    'Debian': 'apache2',
}.get(grains.os_family) %}

 apache:
  pkg.installed:
    - name: {{ name }}

{# ---- Bad example ---- #}

haproxy_conf:
  file.managed:
    - name: /etc/haproxy/haproxy.cfg
    - template: jinja
    {% if 'external_loadbalancer' in grains.roles %}
    - source: salt://haproxy/external_haproxy.cfg
    {% elif 'internal_loadbalancer' in grains.roles %}
    - source: salt://haproxy/internal_haproxy.cfg
    {% endif %}
    - context:
        {% if 'external_loadbalancer' in grains.roles %}
        ssl_termination: True
        {% elif 'internal_loadbalancer' in grains.roles %}
        ssl_termination: False
        {% endif %}

{# ---- Better example ---- #}

{% load_yaml as haproxy_defaults %}
common_settings:
  bind_port: 80

internal_loadbalancer:
  source: salt://haproxy/internal_haproxy.cfg
  settings:
    bind_port: 8080
    ssl_termination: False

external_loadbalancer:
  source: salt://haproxy/external_haproxy.cfg
  settings:
    ssl_termination: True
{% endload %}

{% if 'external_loadbalancer' in grains.roles %}
{% set haproxy = haproxy_defaults['external_loadbalancer'] %}
{% elif 'internal_loadbalancer' in grains.roles %}
{% set haproxy = haproxy_defaults['internal_loadbalancer'] %}
{% endif %}

{% do haproxy.settings.update(haproxy_defaults.common_settings) %}

haproxy_conf:
  file.managed:
    - name: /etc/haproxy/haproxy.cfg
    - template: jinja
    - source: {{ haproxy.source }}
    - context: {{ haproxy.settings | yaml() }}

base:
  '*':
    - common_configuration
    - roles_configuration

common_users:
  user.present:
    - names:
      - larry
      - curly
      - moe

#!py
def run():
    list_of_roles = set()

    # This example has the minion id in the form 'web-03-dev'.
    # Easily access the grains dictionary:
    try:
        app, instance_number, environment = __grains__["id"].split("-")
        instance_number = int(instance_number)
    except ValueError:
        app, instance_number, environment = ["Unknown", 0, "dev"]

    list_of_roles.add(app)

    if app == "web" and environment == "dev":
        list_of_roles.add("primary")
        list_of_roles.add("secondary")
    elif app == "web" and environment == "staging":
        if instance_number == 0:
            list_of_roles.add("primary")
        else:
            list_of_roles.add("secondary")

    # Easily cross-call Salt execution modules:
    if __salt__["myutils.query_valid_ec2_instance"]():
        list_of_roles.add("is_ec2_instance")

    return {
        "set_roles_grains": {
            "grains.present": [{"name": "roles"}, {"value": list(list_of_roles)},],
        },
    }

{% macro user_state(state_id, user_name, shell='/bin/bash', groups=[]) %}
{{ state_id }}:
  user.present:
    - name: {{ user_name }}
    - shell: {{ shell }}
    - groups: {{ groups | json() }}
{% endmacro %}

{% for user_info in salt.pillar.get('my_users', []) %}
{{ user_state('user_number_' ~ loop.index, **user_info) }}
{% endfor %}

php_ini:
  file.managed:
    - name: /etc/php.ini
    - source: salt://php.ini.tmpl
    - template: jinja
    - context:
        php_ini_settings: {{ salt.pillar.get('php_ini', {}) | json() }}

php_ini:
  PHP:
    engine: 'On'
    short_open_tag: 'Off'
    error_reporting: 'E_ALL & ~E_DEPRECATED & ~E_STRICT'

{% macro php_ini_serializer(data) %}
{% for section_name, name_val_pairs in data.items() %}
[{{ section_name }}]
{% for name, val in name_val_pairs.items() -%}
{{ name }} = "{{ val }}"
{% endfor %}
{% endfor %}
{% endmacro %}

; File managed by Salt at <{{ source }}>.
; Your changes will be overwritten.

{{ php_ini_serializer(php_ini_settings) }}

Abstracting static defaults into a lookup table¶

Separate data that a state uses from the state itself to increases the flexibility and reusability of a state.

An obvious and common example of this is platform-specific package names and file system paths. Another example is sane defaults for an application, or common settings within a company or organization. Organizing such data as a dictionary (aka hash map, lookup table, associative array) often provides a lightweight namespacing and allows for quick and easy lookups. In addition, using a dictionary allows for easily merging and overriding static values within a lookup table with dynamic values fetched from Pillar.

A strong convention in Salt Formulas is to place platform-specific data, such as package names and file system paths, into a file named map.jinja that is placed alongside the state files.

The following is an example from the MySQL Formula. The grains.filter_by function performs a lookup on that table using the os_family grain (by default).

The result is that the mysql variable is assigned to a subset of the lookup table for the current platform. This allows states to reference, for example, the name of a package without worrying about the underlying OS. The syntax for referencing a value is a normal dictionary lookup in Jinja, such as {{ mysql['service'] }} or the shorthand {{ mysql.service }}.

map.jinja:

{% set mysql = salt['grains.filter_by']({
    'Debian': {
        'server': 'mysql-server',
        'client': 'mysql-client',
        'service': 'mysql',
        'config': '/etc/mysql/my.cnf',
        'python': 'python-mysqldb',
    },
    'RedHat': {
        'server': 'mysql-server',
        'client': 'mysql',
        'service': 'mysqld',
        'config': '/etc/my.cnf',
        'python': 'MySQL-python',
    },
    'Gentoo': {
        'server': 'dev-db/mysql',
        'client': 'dev-db/mysql',
        'service': 'mysql',
        'config': '/etc/mysql/my.cnf',
        'python': 'dev-python/mysql-python',
    },
}, merge=salt['pillar.get']('mysql:lookup')) %}

Values defined in the map file can be fetched for the current platform in any state file using the following syntax:

{% from "mysql/map.jinja" import mysql with context %}

mysql-server:
  pkg.installed:
    - name: {{ mysql.server }}
  service.running:
    - name: {{ mysql.service }}

mysql:
  lookup:
    version: 5.7.11

{% set mysql = salt['grains.filter_by']({
    'default': {
        'server': 'mysql-server',
        'client': 'mysql-client',
        'service': 'mysql',
        'config': '/etc/mysql/my.cnf',
        'python': 'python-mysqldb',
    },
    'Debian': {
    },
    'RedHat': {
        'client': 'mysql',
        'service': 'mysqld',
        'config': '/etc/my.cnf',
        'python': 'MySQL-python',
    },
    'Gentoo': {
        'server': 'dev-db/mysql',
        'client': 'dev-db/mysql',
        'python': 'dev-python/mysql-python',
    },
},
merge=salt['pillar.get']('mysql:lookup'), base='default') %}

mysql:
  lookup:
    config: /usr/local/etc/mysql/my.cnf

{%- set foo = 7.7 %}
{%- set bar = none %}
{%- set baz = true %}
{%- set zap = 'The word of the day is "salty".' %}
{%- set zip = '"The quick brown fox . . ."' %}

foo: {{ foo|yaml_encode }}
bar: {{ bar|yaml_encode }}
baz: {{ baz|yaml_encode }}
zap: {{ zap|yaml_encode }}
zip: {{ zip|yaml_dquote }}

foo: 7.7
bar: null
baz: true
zap: "The word of the day is \"salty\"."
zip: "\"The quick brown fox . . .\""

{% set lookup_table = {...} %}
{% do lookup_table.update(salt.pillar.get('my:custom:data')) %}

Server:
  '@port': '8005'
  '@shutdown': SHUTDOWN
  GlobalNamingResources:
    Resource:
      '@auth': Container
      '@description': User database that can be updated and saved
      '@factory': org.apache.catalina.users.MemoryUserDatabaseFactory
      '@name': UserDatabase
      '@pathname': conf/tomcat-users.xml
      '@type': org.apache.catalina.UserDatabase
  # <...snip...>

appX:
  server_xml_overrides:
    Server:
      Service:
        '@name': Catalina
        Connector:
          '@port': '8009'
          '@protocol': AJP/1.3
          '@redirectPort': '8443'
          # <...snip...>

{% import_yaml 'tomcat/defaults.yaml' as server_xml_defaults %}
{% set server_xml_final_values = salt.pillar.get(
    'appX:server_xml_overrides',
    default=server_xml_defaults,
    merge=True)
%}

appX_server_xml:
  file.serialize:
    - name: /etc/tomcat/server.xml
    - dataset: {{ server_xml_final_values | json() }}
    - formatter: xml_badgerfish

{# Load the map file. #}
{% import_yaml 'app/defaults.yaml' as app_defaults %}

{# Extract the relevant subset for the app configured on the current
   machine (configured via a grain in this example). #}
{% app = app_defaults.get(salt.grains.get('role')) %}

{# Allow values from Pillar to (optionally) update values from the lookup
   table. #}
{% do app_defaults.update(salt.pillar.get('myapp', {})) %}

deploy_application:
  git.latest:
    - name: {{ app.repo_url }}
    - version: {{ app.version }}
    - target: {{ app.deploy_dir }}

myco/myapp/deployed:
  event.send:
    - data:
        version: {{ app.version }}
    - onchanges:
      - git: deploy_application

appX:
  repo_url: git@github.com/myco/appX.git
  target: /var/www/appX
  version: master
appY:
  repo_url: git@github.com/myco/appY.git
  target: /var/www/appY
  version: v1.2.3.4

Single-purpose SLS files¶

Each sls file in a Formula should strive to do a single thing. This increases the reusability of this file by keeping unrelated tasks from getting coupled together.

As an example, the base Apache formula should only install the Apache httpd server and start the httpd service. This is the basic, expected behavior when installing Apache. It should not perform additional changes such as set the Apache configuration file or create vhosts.

If a formula is single-purpose as in the example above, other formulas, and also other states can include and use that formula with Requisites and Other Global State Arguments without also including undesirable or unintended side-effects.

The following is a best-practice example for a reusable Apache formula. (This skips platform-specific options for brevity. See the full apache-formula for more.)

# apache/init.sls
apache:
  pkg.installed:
    [...]
  service.running:
    [...]

# apache/mod_wsgi.sls
include:
  - apache

mod_wsgi:
  pkg.installed:
    [...]
    - require:
      - pkg: apache

# apache/conf.sls
include:
  - apache

apache_conf:
  file.managed:
    [...]
    - watch_in:
      - service: apache

To illustrate a bad example, say the above Apache formula installed Apache and also created a default vhost. The mod_wsgi state would not be able to include the Apache formula to create that dependency tree without also installing the unneeded default vhost.

Formulas should be reusable. Avoid coupling unrelated actions together.

Parameterization¶

Parameterization is a key feature of Salt Formulas and also for Salt States. Parameterization allows a single Formula to be reused across many operating systems; to be reused across production, development, or staging environments; and to be reused by many people all with varying goals.

Writing states, specifying ordering and dependencies is the part that takes the longest to write and to test. Filling those states out with data such as users or package names or file locations is the easy part. How many users, what those users are named, or where the files live are all implementation details that should be parameterized. This separation between a state and the data that populates a state creates a reusable formula.

In the example below the data that populates the state can come from anywhere -- it can be hard-coded at the top of the state, it can come from an external file, it can come from Pillar, it can come from an execution function call, or it can come from a database query. The state itself doesn't change regardless of where the data comes from. Production data will vary from development data will vary from data from one company to another, however the state itself stays the same.

{% set user_list = [
    {'name': 'larry', 'shell': 'bash'},
    {'name': 'curly', 'shell': 'bash'},
    {'name': 'moe', 'shell': 'zsh'},
] %}

{# or #}

{% set user_list = salt['pillar.get']('user_list') %}

{# or #}

{% load_json "default_users.json" as user_list %}

{# or #}

{% set user_list = salt['acme_utils.get_user_list']() %}

{% for user in list_list %}
{{ user.name }}:
  user.present:
    - name: {{ user.name }}
    - shell: {{ user.shell }}
{% endfor %}

Configuration¶

Formulas should strive to use the defaults of the underlying platform, followed by defaults from the upstream project, followed by sane defaults for the formula itself.

As an example, a formula to install Apache should not change the default Apache configuration file installed by the OS package. However, the Apache formula should include a state to change or override the default configuration file.

Pillar overrides¶

Pillar lookups must use the safe get() and must provide a default value. Create local variables using the Jinja set construct to increase readability and to avoid potentially hundreds or thousands of function calls across a large state tree.

{% from "apache/map.jinja" import apache with context %}
{% set settings = salt['pillar.get']('apache', {}) %}

mod_status:
  file.managed:
    - name: {{ apache.conf_dir }}
    - source: {{ settings.get('mod_status_conf', 'salt://apache/mod_status.conf') }}
    - template: {{ settings.get('template_engine', 'jinja') }}

Any default values used in the Formula must also be documented in the pillar.example file in the root of the repository. Comments should be used liberally to explain the intent of each configuration value. In addition, users should be able copy-and-paste the contents of this file into their own Pillar to make any desired changes.

Scripting¶

Remember that both State files and Pillar files can easily call out to Salt execution modules and have access to all the system grains as well.

{% if '/storage' in salt['mount.active']() %}
/usr/local/etc/myfile.conf:
  file:
    - symlink
    - target: /storage/myfile.conf
{% endif %}

Jinja macros to encapsulate logic or conditionals are discouraged in favor of writing custom execution modules in Python.

README.rst¶

The README should detail each available .sls file by explaining what it does, whether it has any dependencies on other formulas, whether it has a target platform, and any other installation or usage instructions or tips.

A sample skeleton for the README.rst file:

===
foo
===

Install and configure the FOO service.

**NOTE**

See the full `Salt Formulas installation and usage instructions
<https://docs.saltstack.com/en/latest/topics/development/conventions/formulas.html>`_.

Available states
================

.. contents::
    :local:

``foo``
-------

Install the ``foo`` package and enable the service.

``foo.bar``
-----------

Install the ``bar`` package.

CHANGELOG.rst¶

The CHANGELOG.rst file should detail the individual versions, their release date and a set of bullet points for each version highlighting the overall changes in a given version of the formula.

A sample skeleton for the CHANGELOG.rst file:

CHANGELOG.rst:

foo formula
===========

0.0.2 (2013-01-01)

- Re-organized formula file layout
- Fixed filename used for upstart logger template
- Allow for pillar message to have default if none specified

Versioning¶

Formula are versioned according to Semantic Versioning, https://semver.org/.

Formula versions are tracked using Git tags as well as the VERSION file in the formula repository. The VERSION file should contain the currently released version of the particular formula.