Queues

While Group lets you run one operation across many hosts, Chopsticks’ Queue class lets you run a number of different operations across many hosts, so that each host is kept as busy as possible.

Conceptually, a Queue is actually a separate queue of operations for each host. All hosts start their first operation as soon as Queue.run() is called.

Queue is also Chopsticks’ primary asynchronous API; callbacks can be registered which are called as soon as a result is available.

Queue API

class chopsticks.queue.Queue

A queue of tasks to be performed.

Queues build on Groups and Tunnels in order to feed tasks as quickly as possible to all connected hosts.

All methods accept a parameter target, which specifies which tunnels the operation should be performed with. This can be specified as a Tunnel or a Group.

Each one returns an AsyncResult which can be used to receive the result of the operation.

call(target, *args, **kwargs)

Queue a call() operation to be run on the target.

connect(target, *args, **kwargs)

Queue a connect() operation to be run on the target.

fetch(target, remote_path, local_path=None)

Queue a fetch() operation to be run on the target.

put(target, *args, **kwargs)

Queue a put() operation to be run on the target.

run()

Run all items in the queue.

This method does not return until the queue is empty.

class chopsticks.queue.AsyncResult

The deferred result of a queued operation.

value

Get the value of the result.

Raise NotCompleted if the task has not yet run.

with_callback(callback)

Attach a callback to be called when a value is set.

Example

Let’s put three separate files a.txt, b.txt and c.txt onto three hosts:

group = Group([
    'host1.example.com',
    'host2.example.com',
    'host3.example.com',
])

queue = Queue()
for f in ['a.txt', 'b.txt', 'c.txt']:
    queue.put(group, f, f)
queue.run()

Let’s compare this to an approach using the Group alone:

group = Group([
    'host1.example.com',
    'host2.example.com',
    'host3.example.com',
])

for file in ['a.txt', 'b.txt', 'c.txt']:
    group.put(f, f)

The Queue approach will typically run faster, because we do not wait for all the tunnels to catch up after every transfer:

With a lengthy list of tasks, and inevitable variability in how long they take, the Queue is likely to finish much sooner.