Group lets you run one operation across many hosts, Chopsticks’
Queue class lets you run a number of different operations across many
hosts, so that each host is kept as busy as possible.
Queue is actually a separate queue of operations for each
host. All hosts start their first operation as soon as
Queue is also Chopsticks’ primary asynchronous API; callbacks can be
registered which are called as soon as a result is available.
A queue of tasks to be performed.
Queues build on Groups and Tunnels in order to feed tasks as quickly as possible to all connected hosts.
All methods accept a parameter target, which specifies which tunnels the operation should be performed with. This can be specified as a
Each one returns an
AsyncResultwhich can be used to receive the result of the operation.
connect(target, *args, **kwargs)¶
connect()operation to be run on the target.
Run all items in the queue.
This method does not return until the queue is empty.
Let’s put three separate files
c.txt onto three
group = Group([ 'host1.example.com', 'host2.example.com', 'host3.example.com', ]) queue = Queue() for f in ['a.txt', 'b.txt', 'c.txt']: queue.put(group, f, f) queue.run()
Let’s compare this to an approach using the
group = Group([ 'host1.example.com', 'host2.example.com', 'host3.example.com', ]) for file in ['a.txt', 'b.txt', 'c.txt']: group.put(f, f)
Queue approach will typically run faster, because we do not wait for
all the tunnels to catch up after every transfer:
With a lengthy list of tasks, and inevitable variability in how long they take, the Queue is likely to finish much sooner.