Module: tf.compat.v2.distribute

Library for running a computation across multiple devices.

See the guide for overview and examples: TensorFlow v2.x, TensorFlow v1.x. # pylint: disable=line-too-long

The intent of this library is that you can write an algorithm in a stylized way and it will be usable with a variety of different tf.distribute.Strategy implementations. Each descendant will implement a different strategy for distributing the algorithm across multiple devices/machines. Furthermore, these changes can be hidden inside the specific layers and other library classes that need special treatment to run in a distributed setting, so that most users' model definition code can run unchanged. The tf.distribute.Strategy API works the same way with eager and graph execution.

Glossary

Note that we provide a default version of tf.distribute.Strategy that is used when no other strategy is in scope, that provides the same API with reasonable default behavior.

Modules

cluster_resolver module: Library imports for ClusterResolvers.

experimental module: Experimental Distribution Strategy library.

Classes

class CrossDeviceOps: Base class for cross-device reduction and broadcasting algorithms.

class HierarchicalCopyAllReduce: Reduction using hierarchical copy all-reduce.

class InputContext: A class wrapping information needed by an input function.

class InputReplicationMode: Replication mode for input function.

class MirroredStrategy: Mirrors vars to distribute across multiple devices and machines.

class NcclAllReduce: Reduction using NCCL all-reduce.

class OneDeviceStrategy: A distribution strategy for running on a single device.

class ReduceOp: Indicates how a set of values should be reduced.

class ReductionToOneDevice: Always do reduction to one device first and then do broadcasting.

class ReplicaContext: tf.distribute.Strategy API when in a replica context.

class Server: An in-process TensorFlow server, for use in distributed training.

class Strategy: A state & compute distribution policy on a list of devices.

class StrategyExtended: Additional APIs for algorithms that need to be distribution-aware.

Functions

experimental_set_strategy(...): Set a tf.distribute.Strategy as current without with strategy.scope().

get_replica_context(...): Returns the current tf.distribute.ReplicaContext or None.

get_strategy(...): Returns the current tf.distribute.Strategy object.

has_strategy(...): Return if there is a current non-default tf.distribute.Strategy.

in_cross_replica_context(...): Returns True if in a cross-replica context.