salt.pillar.git_pillar¶

Use a git repository as a Pillar source¶

This external pillar allows for a Pillar top file and Pillar SLS files to be sourced from a git repository.

However, since git_pillar does not have an equivalent to the pillar_roots parameter, configuration is slightly different. A Pillar top file is required to be in the git repository and must still contain the relevant environment, like so:

base:
  '*':
    - foo

The branch/tag which maps to that environment must then be specified along with the repo's URL. Configuration details can be found below.

Important

Each branch/tag used for git_pillar must have its own top file. This is different from how the top file works when configuring States. The reason for this is that each git_pillar branch/tag is processed separately from the rest. Therefore, if the qa branch is to be used for git_pillar, it would need to have its own top file, with the qa environment defined within it, like this:

qa:
  'dev-*':
    - bar

Additionally, while git_pillar allows for the branch/tag to be overridden (see here), keep in mind that the top file must reference the actual environment name. It is common practice to make the environment in a git_pillar top file match the branch/tag name, but when remapping, the environment of course no longer matches the branch/tag, and the top file needs to be adjusted accordingly. When expected Pillar values configured in git_pillar are missing, this is a common misconfiguration that may be to blame, and is a good first step in troubleshooting.

Configuring git_pillar for Salt¶

Beginning with Salt version 2015.8.0, pygit2 is now supported in addition to GitPython. The requirements for GitPython and pygit2 are the same as for GitFS, as described here.

Important

git_pillar has its own set of global configuration parameters. While it may seem intuitive to use the global gitfs configuration parameters (gitfs_base, etc.) to manage git_pillar, this will not work. The main difference for this is the fact that the different components which use Salt's git backend code do not all function identically. For instance, in git_pillar it is necessary to specify which branch/tag to be used for git_pillar remotes. This is the reverse behavior from gitfs, where branches/tags make up your environments.

See here for documentation on the git_pillar configuration options and their usage.

Here is an example git_pillar configuration:

ext_pillar:
  - git:
    # Use 'prod' instead of the branch name 'production' as the environment
    - production https://gitserver/git-pillar.git:
      - env: prod
    # Use 'dev' instead of the branch name 'develop' as the environment
    - develop https://gitserver/git-pillar.git:
      - env: dev
    # No per-remote config parameters (and no trailing colon), 'qa' will
    # be used as the environment
    - qa https://gitserver/git-pillar.git
    # SSH key authentication
    - master git@other-git-server:pillardata-ssh.git:
      # Pillar SLS files will be read from the 'pillar' subdirectory in
      # this repository
      - root: pillar
      - privkey: /path/to/key
      - pubkey: /path/to/key.pub
      - passphrase: CorrectHorseBatteryStaple
    # HTTPS authentication
    - master https://other-git-server/pillardata-https.git:
      - user: git
      - password: CorrectHorseBatteryStaple

The main difference between this and the old way of configuring git_pillar is that multiple remotes can be configured under one git section under ext_pillar. More than one git section can be used, but it is not necessary. Remotes will be evaluated sequentially.

Per-remote configuration parameters are supported (similar to gitfs), and global versions of the git_pillar configuration parameters can also be set.

To remap a specific branch to a specific Pillar environment, use the env per-remote parameter:

ext_pillar:
  - git:
    - production https://gitserver/git-pillar.git:
      - env: prod

If __env__ is specified as the branch name, then git_pillar will decide which branch to use based on the following criteria:

If the minion has a pillarenv configured, it will use that pillar environment. (2016.11.2 and later)
Otherwise, if the minion has an environment configured, it will use that environment.
Otherwise, the master's git_pillar_base will be used.

Note

The use of environment to choose the pillar environment dates from a time before the pillarenv parameter was added. In a future release, it will be ignored and either the minion's pillarenv or the master's git_pillar_base will be used.

Here's an example of using __env__ as the git_pillar environment:

ext_pillar:
  - git:
    - __env__ https://gitserver/git-pillar.git:
      - root: pillar

The corresponding Pillar top file would look like this:

"{{saltenv}}":
  '*':
    - bar

With the addition of pygit2 support, git_pillar can now interact with authenticated remotes. Authentication works just like in gitfs (as outlined in the Git Fileserver Backend Walkthrough), only with the global authentication parameter names prefixed with git_pillar instead of gitfs (e.g. git_pillar_pubkey, git_pillar_privkey, git_pillar_passphrase, etc.).

Note

The name parameter can be used to further differentiate between two remotes with the same URL and branch. When using two remotes with the same URL, the name option is required.

How Multiple Remotes Are Handled¶

As noted above, multiple remotes can be included in the same git ext_pillar configuration. Consider the following:

my_etcd_config:
  etcd.host: 127.0.0.1
  etcd.port: 4001

ext_pillar:
  - etcd: my_etcd_config
  - git:
    - master https://mydomain.tld/foo.git:
      - root: pillar
    - master https://mydomain.tld/bar.git
    - master https://mydomain.tld/baz.git
    - dev https://mydomain.tld/qux.git
  - git:
    - master https://mydomain.tld/abc.git
    - dev https://mydomain.tld/123.git

To understand how pillar data from these repos will be compiled, it's important to know how Salt will process them. The following points should be kept in mind:

Each ext_pillar is called separately from the others. So, in the above example, the etcd ext_pillar will be evaluated first, with the first group of git_pillar remotes evaluated next (and merged into the etcd pillar data). Lastly, the second group of git_pillar remotes will be evaluated, and then merged into the ext_pillar data evaluated before it.
Within a single group of git_pillar remotes, each remote will be evaluated in order, with results merged together as each remote is evaluated.

Note

Prior to the 2017.7.0 release, remotes would be evaluated in a non-deterministic order.
By default, when a repo is evaluated, other remotes' which share its pillar environment will have their files made available to the remote being processed.

The first point should be straightforward enough, but the second and third could use some additional clarification.

First, point #2. In the first group of git_pillar remotes, the top file and pillar SLS files in the foo remote will be evaluated first. The bar remote will be evaluated next, and its results will be merged into the pillar data compiled when the foo remote was evaluated. As the subsequent remotes are evaluated, their data will be merged in the same fashion.

But wait, don't these repositories belong to more than one pillar environments? Well, yes. The default method of generating pillar data compiles pillar data from all environments. This behavior can be overridden using a pillarenv. Setting a pillarenv in the minion config file will make that minion tell the master to ignore any pillar data from environments which don't match that pillarenv. A pillarenv can also be specified for a given minion or set of minions when running states, by using the pillarenv argument. The CLI pillarenv will override one set in the minion config file. So, assuming that a pillarenv of base was set for a minion, it would not get any of the pillar variables configured in the qux remote, since that remote is assigned to the dev environment. The only way to get its pillar data would be to specify a pillarenv of dev, which would mean that it would then ignore any items from the base pillarenv. A more detailed explanation of pillar environments can be found here.

Moving on to point #3, and looking at the example ext_pillar configuration, as the foo remote is evaluated, it will also have access to the files from the bar and baz remotes, since all three are assigned to the base pillar environment. So, if an SLS file referenced by the foo remotes's top file does not exist in the foo remote, it will be searched for in the bar remote, followed by the baz remote. When it comes time to evaluate the bar remote, SLS files referenced by the bar remote's top file will first be looked for in the bar remote, followed by foo, and baz, and when the baz remote is processed, SLS files will be looked for in baz, followed by foo and bar. This "failover" logic is called a directory overlay, and it is also used by file_roots and :conf_minion`pillar_roots`. The ordering of which remote is checked for SLS files is determined by the order they are listed. First the remote being processed is checked, then the others that share the same environment are checked. However, before the 2017.7.0 release, since evaluation was unordered, the remote being processed would be checked, followed in no specific order by the other repos which share the same environment.

Beginning with the 2017.7.0 release, this behavior of git_pillar remotes having access to files in other repos which share the same environment can be disabled by setting git_pillar_includes to False. If this is done, then all git_pillar remotes will only have access to their own SLS files. Another way of ensuring that a git_pillar remote will not have access to SLS files from other git_pillar remotes which share the same pillar environment is to put them in a separate git section under ext_pillar. Look again at the example configuration above. In the second group of git_pillar remotes, the abc remote would not have access to the SLS files from the foo, bar, and baz remotes, and vice-versa.

Mountpoints¶

New in version 2017.7.0.

Assume the following pillar top file:

base:
  'web*':
    - common
    - web.server.nginx
    - web.server.appdata

Now, assume that you would like to configure the web.server.nginx and web.server.appdata SLS files in separate repos. This could be done using the following ext_pillar configuration (assuming that git_pillar_includes has not been set to False):

ext_pillar:
  - git:
    - master https://mydomain.tld/pillar-common.git
    - master https://mydomain.tld/pillar-nginx.git
    - master https://mydomain.tld/pillar-appdata.git

However, in order to get the files in the second and third git_pillar remotes to work, you would need to first create the directory structure underneath it (i.e. place them underneath web/server/ in the repository). This also makes it tedious to reorganize the configuration, as changing web.server.nginx to web.nginx in the top file would require you to also move the SLS files in the pillar-nginx up a directory level.

For these reasons, much like gitfs, git_pillar now supports a "mountpoint" feature. Using the following ext_pillar configuration, the SLS files in the second and third git_pillar remotes can be placed in the root of the git repository:

ext_pillar:
  - git:
    - master https://mydomain.tld/pillar-common.git
    - master https://mydomain.tld/pillar-nginx.git:
      - mountpoint: web/server/
    - master https://mydomain.tld/pillar-appdata.git:
      - mountpoint: web/server/

Now, if the top file changed the SLS target from web.server.nginx, instead of reorganizing the git repository, you would just need to adjust the mountpoint to web/ (and restart the salt-master daemon).

Note

Leading and trailing slashes on the mountpoints are optional.
Use of the mountpoint feature requires that git_pillar_includes is not disabled.
Content from mounted git_pillar repos can only be referenced by a top file in the same pillar environment.
Salt versions prior to 2018.3.4 ignore the root parameter when mountpoint is set.

all_saltenvs¶

New in version 2018.3.4.

When __env__ is specified as the branch name, all_saltenvs per-remote configuration parameter overrides the logic Salt uses to map branches/tags to pillar environments (i.e. pillarenvs). This allows a single branch/tag to appear in all saltenvs. Example:

ext_pillar:
  - git:
    - __env__ https://mydomain.tld/top.git
      - all_saltenvs: master
    - __env__ https://mydomain.tld/pillar-nginx.git:
      - mountpoint: web/server/
    - __env__ https://mydomain.tld/pillar-appdata.git:
      - mountpoint: web/server/

git_pillar_update_interval¶

New in version 3000.

This option defines the default update interval (in seconds) for git_pillar remotes. The update is handled within the global loop, hence git_pillar_update_interval should be a multiple of loop_interval.

git_pillar_update_interval: 120

fallback¶

New in version 3001.

Setting fallback per-remote or global configuration parameter will map non-existing environments to a default branch. Example:

ext_pillar:
  - git:
    - __env__ https://mydomain.tld/top.git
      - all_saltenvs: master
    - __env__ https://mydomain.tld/pillar-nginx.git:
      - mountpoint: web/server/
      - fallback: master
    - __env__ https://mydomain.tld/pillar-appdata.git:
      - mountpoint: web/server/
      - fallback: master

salt.pillar.git_pillar.ext_pillar(minion_id, pillar, *repos)¶: Checkout the ext_pillar sources and compile the resulting pillar SLS