salt.states.boto_kinesis module¶

Manage Kinesis Streams¶

Create and destroy Kinesis streams. Be aware that this interacts with Amazon's services, and so may incur charges.

This module uses boto3, which can be installed via package, or pip.

This module accepts explicit Kinesis credentials but can also utilize IAM roles assigned to the instance through Instance Profiles. Dynamic credentials are then automatically obtained from AWS API and no further configuration is necessary. More information available here.

If IAM roles are not used you need to specify them either in a pillar file or in the minion's config file:

keyid: GKTADJGHEIQSXMKKRBJ08H
key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
region: us-east-1

It's also possible to specify key, keyid and region via a profile, either passed in as a dict, or as a string to pull from pillars or minion config:

myprofile:
    keyid: GKTADJGHEIQSXMKKRBJ08H
    key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
        region: us-east-1

Ensure Kinesis stream does not exist:
  boto_kinesis.absent:
    - name: new_stream
    - keyid: GKTADJGHEIQSXMKKRBJ08H
    - key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
    - region: us-east-1

Ensure Kinesis stream exists:
  boto_kinesis.present:
    - name: new_stream
    - retention_hours: 168
    - enhanced_monitoring: ['ALL']
    - num_shards: 2
    - keyid: GKTADJGHEIQSXMKKRBJ08H
    - key: askdjghsdfjkghWupUjasdflkdfklgjsdfjajkghs
    - region: us-east-1

salt.states.boto_kinesis.absent(name, region=None, key=None, keyid=None, profile=None)¶

Delete the kinesis stream, if it exists.

name (string)

Stream name

region (string)

Region to connect to.

key (string)

Secret key to be used.

keyid (string)

Access key to be used.

profile (dict)

A dict with region, key and keyid, or a pillar key (string) that contains a dict with region, key and keyid.

salt.states.boto_kinesis.present(name, retention_hours=None, enhanced_monitoring=None, num_shards=None, do_reshard=True, region=None, key=None, keyid=None, profile=None)¶

Ensure the kinesis stream is properly configured and scaled.

name (string)

Stream name

retention_hours (int)

Retain data for this many hours. AWS allows minimum 24 hours, maximum 168 hours.

enhanced_monitoring (list of string)

Turn on enhanced monitoring for the specified shard-level metrics. Pass in ['ALL'] or True for all metrics, [] or False for no metrics. Turn on individual metrics by passing in a list: ['IncomingBytes', 'OutgoingBytes'] Note that if only some metrics are supplied, the remaining metrics will be turned off.

num_shards (int)

Reshard stream (if necessary) to this number of shards !!!!! Resharding is expensive! Each split or merge can take up to 30 seconds, and the reshard method balances the partition space evenly. Resharding from N to N+1 can require 2N operations. Resharding is much faster with powers of 2 (e.g. 2^N to 2^N+1) !!!!!

do_reshard (boolean)

If set to False, this script will NEVER reshard the stream, regardless of other input. Useful for testing.

region (string)

Region to connect to.

key (string)

Secret key to be used.

keyid (string)

Access key to be used.

profile (dict)

A dict with region, key and keyid, or a pillar key (string) that contains a dict with region, key and keyid.