Circus Library

The Circus package is composed of a high-level get_arbiter() function and many classes. In most cases, using the high-level function should be enough, as it creates everything that is needed for Circus to run.

You can subclass Circus’ classes if you need more granularity than what is offered by the configuration.

The get_arbiter function

get_arbiter() is just a convenience on top of the various circus classes. It creates a arbiter (class Arbiter) instance with the provided options, which in turn runs a single Watcher with a single Process.

circus.get_arbiter(watchers, controller=None, pubsub_endpoint=None, stats_endpoint=None, env=None, name=None, context=None, background=False, stream_backend='thread', plugins=None, debug=False)

Creates a Arbiter and a single watcher in it.

Options:

  • watchers – a list of watchers. A watcher in that case is a dict containing:

    • name – the name of the watcher (default: None)

    • cmd – the command line used to run the Watcher.

    • args – the args for the command (list or string).

    • executable – When executable is given, the first item in the args sequence obtained from cmd is still treated by most programs as the command name, which can then be different from the actual executable name. It becomes the display name for the executing program in utilities such as ps.

    • numprocesses – the number of processes to spawn (default: 1).

    • warmup_delay – the delay in seconds between two spawns (default: 0)

    • shell – if True, the processes are run in the shell (default: False)

    • working_dir - the working dir for the processes (default: None)

    • uid – the user id used to run the processes (default: None)

    • gid – the group id used to run the processes (default: None)

    • env – the environment passed to the processes (default: None)

    • send_hup: if True, a process reload will be done by sending the SIGHUP signal. (default: False)

    • stdout_stream: a mapping containing the options for configuring the stdout stream. Default to None. When provided, may contain:

      • class: the fully qualified name of the class to use for streaming. Defaults to circus.stream.FileStream
      • refresh_time: the delay between two stream checks. Defaults to 0.3 seconds.
      • any other key will be passed the class constructor.
    • stderr_stream: a mapping containing the options for configuring the stderr stream. Default to None. When provided, may contain:

      • class: the fully qualified name of the class to use for streaming. Defaults to circus.stream.FileStream
      • refresh_time: the delay between two stream checks. Defaults to 0.3 seconds.
      • any other key will be passed the class constructor.
    • max_retry: the number of times we attempt to start a process, before we abandon and stop the whole watcher. (default: 5)

  • controller – the zmq entry point (default: ‘tcp://127.0.0.1:5555‘)

  • pubsub_endpoint – the zmq entry point for the pubsub (default: ‘tcp://127.0.0.1:5556‘)

  • stats_endpoint – the stats endpoint. If not provided, the circusd-stats process will not be launched. (default: None)

  • context – the zmq context (default: None)

  • background – If True, the arbiter is launched in a thread in the background (default: False)

  • stream_backend – the backend that will be used for the streaming process. Can be thread or gevent. When set to gevent you need to have gevent and gevent_zmq installed. (default: thread)

  • plugins – a list of plugins. Each item is a mapping with:

    • use – Fully qualified name that points to the plugin class
    • every other value is passed to the plugin in the config option
  • debug – If True the arbiter is launched in debug mode (default: False)

Example:

from circus import get_arbiter

arbiter = get_arbiter("myprogram", numprocesses=3)
try:
    arbiter.start()
finally:
    arbiter.stop()

The classes collection

Circus provides a series of classes you can use to implement your own process manager:

  • Process: wraps a running process and provides a few helpers on top of it.
  • Watcher: run several instances of Process against the same command. Manage the death and life of processes.
  • Arbiter: manages several Watcher.
class circus.process.Process(wid, cmd, args=None, working_dir=None, shell=False, uid=None, gid=None, env=None, rlimits=None, executable=None, use_fds=False, watcher=None, spawn=True)

Wraps a process.

Options:

  • wid: the process unique identifier. This value will be used to replace the $WID string in the command line if present.
  • cmd: the command to run. May contain any of the variables available that are being passed to this class. They will be replaced using the python format syntax.
  • args: the arguments for the command to run. Can be a list or a string. If args is a string, it’s splitted using shlex.split(). Defaults to None.
  • executable: When executable is given, the first item in the args sequence obtained from cmd is still treated by most programs as the command name, which can then be different from the actual executable name. It becomes the display name for the executing program in utilities such as ps.
  • working_dir: the working directory to run the command in. If not provided, will default to the current working directory.
  • shell: if True, will run the command in the shell environment. False by default. warning: this is a security hazard.
  • uid: if given, is the user id or name the command should run with. The current uid is the default.
  • gid: if given, is the group id or name the command should run with. The current gid is the default.
  • env: a mapping containing the environment variables the command will run with. Optional.
  • rlimits: a mapping containing rlimit names and values that will be set before the command runs.
  • use_fds: if True, will not close the fds in the subprocess. default: False.
pid

Return the pid

stdout

Return the stdout stream

stderr

Return the stdout stream

send_signal(*args, **kw)

Sends a signal sig to the process.

stop(*args, **kw)

Terminate the process.

age()

Return the age of the process in seconds.

info()

Return process info.

The info returned is a mapping with these keys:

  • mem_info1: Resident Set Size Memory in bytes (RSS)
  • mem_info2: Virtual Memory Size in bytes (VMS).
  • cpu: % of cpu usage.
  • mem: % of memory usage.
  • ctime: process CPU (user + system) time in seconds.
  • pid: process id.
  • username: user name that owns the process.
  • nice: process niceness (between -20 and 20)
  • cmdline: the command line the process was run with.
children()

Return a list of children pids.

is_child(pid)

Return True is the given pid is a child of that process.

send_signal_child(*args, **kw)

Send signal signum to child pid.

send_signal_children(*args, **kw)

Send signal signum to all children.

status

Return the process status as a constant

  • RUNNING
  • DEAD_OR_ZOMBIE
  • UNEXISTING
  • OTHER

Example:

>>> from circus.process import Process
>>> process = Process('Top', 'top', shell=True)
>>> process.age()
3.0107998847961426
>>> process.info()
'Top: 6812  N/A tarek Zombie N/A N/A N/A N/A N/A'
>>> process.status
1
>>> process.stop()
>>> process.status
2
>>> process.info()
'No such process (stopped?)'
class circus.watcher.Watcher(name, cmd, args=None, numprocesses=1, warmup_delay=0.0, working_dir=None, shell=False, uid=None, max_retry=5, gid=None, send_hup=False, env=None, stopped=True, graceful_timeout=30.0, prereload_fn=None, rlimits=None, executable=None, stdout_stream=None, stderr_stream=None, stream_backend='thread', priority=0, singleton=False, use_sockets=False, copy_env=False, copy_path=False, max_age=0, max_age_variance=30, **options)

Class managing a list of processes for a given command.

Options:

  • name: name given to the watcher. Used to uniquely identify it.

  • cmd: the command to run. May contain $WID, which will be replaced by wid.

  • args: the arguments for the command to run. Can be a list or a string. If args is a string, it’s splitted using shlex.split(). Defaults to None.

  • numprocesses: Number of processes to run.

  • working_dir: the working directory to run the command in. If not provided, will default to the current working directory.

  • shell: if True, will run the command in the shell environment. False by default. warning: this is a security hazard.

  • uid: if given, is the user id or name the command should run with. The current uid is the default.

  • gid: if given, is the group id or name the command should run with. The current gid is the default.

  • send_hup: if True, a process reload will be done by sending the SIGHUP signal. Defaults to False.

  • env: a mapping containing the environment variables the command will run with. Optional.

  • rlimits: a mapping containing rlimit names and values that will be set before the command runs.

  • stdout_stream: a mapping that defines the stream for the process stdout. Defaults to None.

    Optional. When provided, stdout_stream is a mapping containing up to three keys:

    • class: the stream class. Defaults to circus.stream.FileStream
    • filename: the filename, if using a FileStream
    • refresh_time: the delay between two stream checks. Defaults to 0.3 seconds.

    This mapping will be used to create a stream callable of the specified class. Each entry received by the callable is a mapping containing:

    • pid - the process pid
    • name - the stream name (stderr or stdout)
    • data - the data
  • stderr_stream: a mapping that defines the stream for the process stderr. Defaults to None.

    Optional. When provided, stderr_stream is a mapping containing up to three keys: - class: the stream class. Defaults to circus.stream.FileStream - filename: the filename, if using a FileStream - refresh_time: the delay between two stream checks. Defaults

    to 0.3 seconds.

    This mapping will be used to create a stream callable of the specified class.

    Each entry received by the callable is a mapping containing:

    • pid - the process pid
    • name - the stream name (stderr or stdout)
    • data - the data
  • stream_backend – the backend that will be used for the streaming process. Can be thread or gevent. When set to gevent you need to have gevent and gevent_zmq installed. (default: thread)

  • priority – integer that defines a priority for the watcher. When the Arbiter do some operations on all watchers, it will sort them with this field, from the bigger number to the smallest. (default: 0)

  • singleton – If True, this watcher has a single process. (default:False)

  • use_sockets – If True, the processes will inherit the file descriptors, thus can reuse the sockets opened by circusd. (default: False)

  • copy_env – If True, the environment in which circus is running run will be reproduced for the workers. (default: False)

  • copy_path – If True, circusd sys.path is sent to the process through PYTHONPATH. You must activate copy_env for copy_path to work. (default: False)

  • max_age: If set after around max_age seconds, the process is replaced with a new one. (default: 0, Disabled)

  • max_age_variance: The maximum number of seconds that can be added to max_age. This extra value is to avoid restarting all processes at the same time. A process will live between max_age and max_age + max_age_variance seconds.

  • options – extra options for the worker. All options found in the configuration file for instance, are passed in this mapping – this can be used by plugins for watcher-specific options.

notify_event(topic, msg)

Publish a message on the event publisher channel

reap_processes(*args, **kw)

Reap all the processes for this watcher.

manage_processes(*args, **kw)

manage processes

reap_and_manage_processes(*args, **kw)

Reap & manage processes.

spawn_processes(*args, **kw)

Spawn processes.

spawn_process()

Spawn process.

kill_process(process, sig=15)

Kill process.

kill_processes(*args, **kw)

Kill all the processes of this watcher.

send_signal_child(*args, **kw)

Send signal to a child.

stop(*args, **kw)

Stop.

start(*args, **kw)

Start.

restart(*args, **kw)

Restart.

reload(*args, **kw)
class circus.arbiter.Arbiter(watchers, endpoint, pubsub_endpoint, check_delay=1.0, prereload_fn=None, context=None, loop=None, stats_endpoint=None, plugins=None, sockets=None, warmup_delay=0, httpd=False, httpd_host='localhost', httpd_port=8080, debug=False, stream_backend='thread', ssh_server=None)

Class used to control a list of watchers.

Options:

  • watchers – a list of Watcher objects

  • endpoint – the controller ZMQ endpoint

  • pubsub_endpoint – the pubsub endpoint

  • stats_endpoint – the stats endpoint. If not provided, the circusd-stats process will not be launched.

  • check_delay – the delay between two controller points (default: 1 s)

  • prereload_fn – callable that will be executed on each reload (default: None)

  • context – if provided, the zmq context to reuse. (default: None)

  • loop: if provided, a zmq.eventloop.ioloop.IOLoop instance

    to reuse. (default: None)

  • plugins – a list of plugins. Each item is a mapping with:

    • use – Fully qualified name that points to the plugin class
    • every other value is passed to the plugin in the config option
  • sockets – a mapping of sockets. Each key is the socket name, and each value a CircusSocket class. (default: None)

  • warmup_delay – a delay in seconds between two watchers startup. (default: 0)

  • httpd – If True, a circushttpd process is run (default: False)

  • httpd_host – the circushttpd host (default: localhost)

  • httpd_port – the circushttpd port (default: 8080)

  • debug – if True, adds a lot of debug info in the stdout (default: False)

  • stream_backend – the backend that will be used for the streaming process. Can be thread or gevent. When set to gevent you need to have gevent and gevent_zmq installed. All watchers will use this setup unless stated otherwise in the watcher configuration. (default: thread)

start(*args, **kw)

Starts all the watchers.

The start command is an infinite loop that waits for any command from a client and that watches all the processes and restarts them if needed.

reload(*args, **kw)

Reloads everything.

Run the prereload_fn() callable if any, then gracefuly reload all watchers.

numprocesses()

Return the number of processes running across all watchers.

numwatchers()

Return the number of watchers.

get_watcher(name)

Return the watcher name.

add_watcher(name, cmd, **kw)

Adds a watcher.

Options:

  • name: name of the watcher to add
  • cmd: command to run.
  • all other options defined in the Watcher constructor.

Project Versions

Table Of Contents

Previous topic

Circus Sockets

Next topic

The Plugin System

This Page

Feedback