Groups

Groups can be used to perform a remote operation in parallel across a number of hosts, and collect the results.

Group API

class chopsticks.group.Group(hosts)[source]

A group of hosts, for performing operations in parallel.

__init__(hosts)[source]

Construct a group from a list of tunnels or hosts.

hosts may contain hostnames - in which case the connections will be made via SSH using the default settings. Alternatively, it may contain tunnel instances.

call(callable, *args, **kwargs)[source]

Call the given callable on all hosts in the group.

The given callable and parameters must be pickleable.

However, the callable’s return value has a tighter restriction: it must be serialisable as JSON, in order to ensure the orchestration host cannot be compromised through pickle attacks.

The return value is a GroupResult.

fetch(remote_path, local_path=None)[source]

Fetch files from all remote hosts.

If local_path is given, it is a local path template, into which the tunnel’s host name will be substituted using str.format(). Hostnames generated in this way must be unique.

For example:

group.fetch('/etc/passwd', local_path='passwd-{host}')

If local_path is not given, a temporary file will be used for each host.

Return a GroupResult of dicts, each containing:

  • local_path - the local path written to
  • remote_path - the absolute remote path
  • size - the number of bytes received
  • sha1sum - a sha1 checksum of the file data
filter(predicate, exclude=False)[source]

Return a Group of the tunnels for which predicate returns True.

predicate must be a no-argument callable that can be pickled.

If exclude is True, then return a Group that only contains tunnels for which predicate returns False.

Raise RemoteException if any hosts could not be connected or fail to evaluate the predicate.

put(local_path, remote_path=None, mode=420)[source]

Copy a file to all remote hosts.

If remote_path is given, it is the remote path to write to. Otherwise, a temporary filename will be used (which will be different on each host).

mode gives the permission bits of the files to create, or 0o644 if unspecified.

This operation supports arbitarily large files (file data is streamed, not buffered in memory).

Return a GroupResult of dicts, each containing:

  • remote_path - the absolute remote path
  • size - the number of bytes received
  • sha1sum - a sha1 checksum of the file data

Results

class chopsticks.group.GroupResult[source]

The results of a Group.call() operation.

GroupResult behaves as a dictionary of results, keyed by hostname, although failures from individual hosts are represented as ErrorResult objects.

Methods are provided to easily process successes and failures separately.

failures()[source]

Iterate over failed results as (host, err) pairs.

raise_failures()[source]

Raise a RemoteException if there were any failures.

successful()[source]

Iterate over successful results as (host, value) pairs.

class chopsticks.group.ErrorResult(msg, tb=None)[source]

Indicates an error returned by the remote host.

Because tracebacks or error types cannot be represented across hosts this will simply consist of a message.

Error results provide the following attributes:

msg

A human-readable error message.

tb

The traceback from the remote host as a string, or None if unavailable.

Set operations

Groups also behave like sets over tunnels. Tunnels are compared by name for this purpose (in general, tunnels need unique names due to the way results are returned from group methods).

For example:

webservers = Group(['web1', 'web2'])
celery_workers = Group(['worker1', 'worker2', 'worker3'])

(webservers + celery_workers).call(install_virtualenv)

For this purpose, individual tunnels act as a group containing just one tunnel:

>>> dck1 = Docker('docker1')
>>> dck2 = Docker('docker2')
>>> dck1 + dck2
Group([Docker('docker1'), Docker('docker2')])

Examples

For example, this code:

from chopsticks.facts import ip
from chopsticks.group import Group

group = Group([
    'web1.example.com',
    'web2.example.com',
    'web3.example.com',
])
for host, addr in group.call(ip).items():
    print('%s ip:' % host, addr)

might output:

web1.example.com ip: 196.168.10.5
web3.example.com ip: 196.168.10.7
web2.example.com ip: 196.168.10.6

You could also construct a group from existing tunnels - or mix and match:

all_hosts = Group([
    'web1.example.com',
    Docker('example'),
    Local('worker')
])