Tunnels

Tunnels are the lowest-level API, used for invoking commands on an individual host or container. For a higher-level API that allows invoking commands in parallel across a range of hosts, see Groups.

An established tunnel can be used to invoke commands and receive results.

Tunnel reference

All tunnels support the following methods:

class chopsticks.tunnel.BaseTunnel

call(callable, *args, **kwargs)

Call the given callable on the remote host.

The parameters must be pickleable.

The callable must return a value that can be serialised using Chopsticks’ binary encoding - generally just primitive Python types. This is somewhat stricter than pickle, because using pickle for results would enable remote hosts to compromise the control host.

close()

Disconnect the tunnel.

Note that this will terminate the remote process and any state will be lost. This does not destroy the Tunnel object, which can be reconnected with connect().

fetch(remote_path, local_path=None)

Fetch one file from the remote host.

If local_path is given, it is the local path to write to. Otherwise, a temporary filename will be used.

This operation supports arbitarily large files (file data is streamed, not buffered in memory).

The return value is a dict containing:

• local_path - the local path written to
• remote_path - the absolute remote path
• size - the number of bytes received
• sha1sum - a sha1 checksum of the file data

put(local_path, remote_path=None, mode=420)

Copy a file to the remote host.

If remote_path is given, it is the remote path to write to. Otherwise, a temporary filename will be used.

mode gives is the permission bits of the file to create, or 0o644 if unspecified.

This operation supports arbitarily large files (file data is streamed, not buffered in memory).

The return value is a dict containing:

• remote_path - the absolute remote path
• size - the number of bytes received
• sha1sum - a sha1 checksum of the file data

SSH

class chopsticks.tunnel.SSHTunnel(host, user=None, port=None, sudo=False)

A tunnel that connects to a remote host over SSH.

Parameters:

• host – The hostname to connect to, as would be specified on an ssh command line.
• user – The username to connect as.
• port – The tcp port to connect to.
• sudo – If true, use sudo on the remote end in order to run as the root user. Use this when you can sudo to root but not ssh directly as the root user.

chopsticks.tunnel.Tunnel

alias of SSHTunnel

Docker

class chopsticks.tunnel.Docker(name, image='python:2.7', rm=True)

A tunnel connected to a throwaway Docker container.

Parameters:

• name – The name of the Docker instance to create.
• image – The Docker image to launch. By default, download and run an official Docker Python image corresponding to the running Python version. Official images are curated by Docker.
• rm – If true, destroy the container when the tunnel is closed.

Subprocess

class chopsticks.tunnel.Local(name='localhost')

A tunnel to a subprocess on the same host.

Sudo

class chopsticks.tunnel.Sudo(user='root', name=None)

A tunnel to a process on the same host, launched with sudo.

The Sudo tunnel does not deal with password dialogues etc. In order for this to work you must configure sudo not to need a password. You can do this with these lines in /etc/sudoers:

Cmnd_Alias PYTHON_CMDS = /usr/bin/python, /usr/bin/python2, /usr/bin/python3
%somegroup   ALL=NOPASSWD: PYTHON_CMDS

This would allow users in the group somegroup to be able to run the system Python interpreters using sudo, without passwords.

Warning

Naturally, as Chopsticks is a framework for executing arbitrary code, this allows executing arbitrary code as root. Only make this change if you are happy with relaxing security in this way.

Writing new tunnels

It is possible to write a new tunnel driver for any system that allows you to execute a python binary with direct relay of stdin and stdout pipes. To do this, simply subclass chopsticks.group.SubprocessTunnel. Note that all tunnel instances must have a host attibute which is used as the key for the result in the GroupResult dictionary when executing tasks in a Group.

So, strictly, these requirements apply:

• The tunnel setup machinery should not write to stdout - else you will have to identify and consume this output.
• The tunnel setup machinery should not read from stdin - else you will have to feed the required input.
• Both stdin and stdout must be binary-safe pipes.

The tunnel machinery may write to stderr; this output will be presented to the user.

Recursively tunnelling

Chopsticks can be imported and used on the remote side of a tunnel. This situation is called recursive tunnelling, and it has its uses. For example:

• You could create an SSHTunnel to a remote host and then Sudo to execute certain actions as root.
• You could maintain a group of SSHTunnels to physical hosts, that each construct a pool of Docker tunnels - for an instant cluster.

Recursion could be dangerous. For example, consider this function:

def recursive():
    with Local() as tun:
        tun.call(recursive)

This would effectively fork-bomb your host! To avoid this pitfall, Chopsticks has a built-in depth limit of 2. You can override this limit by setting

chopsticks.DEPTH_LIMIT = 3

Caution

Do not write

chopsticks.DEPTH_LIMIT += 1

This will undo the limiting!