rycus86/docker-pygen repository overview

⁠docker-pygen

Configuration generator based on Docker containers state and parameters.

⁠Motivation

As we break our applications down to individual microservices more and more the harder it gets to configure the supporting infrastructure around them. If we think about managing HTTP proxying to them with servers like Nginx or configuring any other system that has to know about a set of (or all) of the running services - that can become quite an overhead done manually.

If you're using Docker to run those microservices then this project could provide an easy solution to the problem. By inspecting the currently running containers and their settings it can generate configuration files for basically anything that works with those. It can also notify other services about the configuration change by signalling or restarting them.

⁠Usage

To run it as a Python application (tested on versions 2.7, 3.4 and 3.6) clone the project and install the dependencies:

pip install -r requirements.txt

Then run it as python cli.py <args> where the arguments are:

usage: cli.py [-h] --template TEMPLATE [--target TARGET]
              [--restart <CONTAINER>] [--signal <CONTAINER> <SIGNAL>]
              [--interval <MIN> [<MAX> ...]] [--events <EVENT> [<EVENT> ...]]
              [--swarm-manager] [--workers <TARGET> [<TARGET> ...]]
              [--retries RETRIES] [--no-ssl-check] [--one-shot]
              [--docker-address <ADDRESS>] [--debug]

Template generator based on Docker runtime information

optional arguments:
  -h, --help            show this help message and exit
  --template TEMPLATE   The base Jinja2 template file or inline template as
                        string if it starts with "#"
  --target TARGET       The target to save the generated file (/dev/stdout by
                        default)
  --restart <CONTAINER>
                        Restart the target container, can be: ID, short ID,
                        name, Compose service name, label ["pygen.target"] or
                        environment variable ["PYGEN_TARGET"]
  --signal <CONTAINER> <SIGNAL>
                        Signal the target container, in <container> <signal>
                        format. The <container> argument can be one of the
                        attributes described in --restart
  --interval <MIN> [<MAX> ...]
                        Minimum and maximum intervals for sending
                        notifications. If there is only one argument it will
                        be used for both MIN and MAX. The defaults are: 0.5
                        and 2 seconds.
  --events <EVENT> [<EVENT> ...]
                        Docker events to watch and trigger updates for
                        (default: start, stop, die, health_status)
  --swarm-manager       Enable the Swarm manager HTTP endpoint on port 9411
  --workers <TARGET> [<TARGET> ...]
                        The target hostname of PyGen workers listening on port
                        9412 (use "tasks.service_name" for Swarm workers)
  --retries RETRIES     Number of retries for sending an action to a Swarm
                        worker
  --no-ssl-check        Disable SSL verification when loading templates over
                        HTTPS (not secure)
  --one-shot            Run the update once and exit, also execute actions if
                        the target changes
  --docker-address <ADDRESS>
                        Alternative address (URL) for the Docker daemon
                        connection
  --debug               Enable debug log messages

The application will need access to the Docker daemon too.

You can also run it as a Docker container to make things easier:

docker run -d --name config-generator                         \
              -v /var/run/docker.sock:/var/run/docker.sock:ro \
              -v shared-volume:/etc/share/config              \
              -v $PWD/template.conf:/etc/share/template.conf  \
              --template /etc/share/template.conf             \
              --target   /etc/share/config/auto.conf          \
              --restart  config-loader                        \
              --signal   web-server HUP                       \
              rycus86/docker-pygen

This command will:

• attach the Docker socket from /var/run/docker.sock
• attach a shared folder from the shared-volume to /etc/share/config
• attach the template file template.conf from the current host directory to /etc/share/template.conf
• use the template (at /etc/share/template.conf inside the container)
• write to the auto.conf target file on the shared volume (at /etc/share/config/auto.conf inside the container)
• restart containers matching "config-loader" when the configuration file is updated
• send a SIGHUP signal to containers matching "web-server"

Matching containers can be based on container ID, short ID, name, Compose or Swarm service name. You can also add it as the value of the pygen.target label or as the value of the PYGEN_TARGET environment variable.

The connection to the Docker daeamon can be overridden from the default location to an alternative (for TCP for example) using the --docker-address flag. For testing (or for other reasons) the app can also run in --one-shot mode that generates the configuration using the template once and exits without watching for events (this also executes any actions given if the target file changes).

The Docker image is available in three flavors:

• latest: for x86 hosts
  
• armhf: for 32-bits ARM hosts
  
• aarch64: for 64-bits ARM hosts
  

latest is auto-built on Docker Hub⁠ while the ARM builds are uploaded from Travis⁠.

⁠Templating

To generate the configuration files, the app uses Jinja2 templates⁠. Templates have access to these variables:

• containers list containing a list of running Docker containers wrapped as models.ContainerInfo objects on a resources.ContainerList
• services list containing Swarm services with their running tasks using models.ServiceInfo and models.TaskInfo objects wrapped in resources.ServiceList and resources.TaskList collections.
• all_containers lazy-loaded list of all Docker containers (even if not running)
• all_services lazy-loaded list of Swarm services with all their tasks (even if not in running state)
• nodes lazy-loaded list of Swarm nodes as models.NodeInfo objects wrapped in a resources.ResourceList list

Templates can be loaded from a file, from an HTTP/HTTPS address or can be given inline if the --template parameters starts with a # sign.

A small example from a template could look like this:

{% set server_name = 'test.example.com' %}
upstream {{ server_name }} {
    {% for container in containers
          if  container.networks.first_value.ip_address
          and container.ports.tcp.first_value %}
        # {{ container.name }}
        server {{ container.networks.first_value.ip_address }}:{{ container.ports.tcp.first_value }};
    {% endfor %}
}

This example from the nginx.example⁠ file would output server_name as the value set on the first line then iterate through the containers having an IP address and TCP port exposed to finally output them prefixed with the container's name.

The available properties on a models.ContainerInfo object are:

• raw: The original container object from docker-py⁠
• id: The container's ID
• short_id: The container's short ID
• name: The container's name
• image: The name of the image the container uses
• status: The current status of the container
• labels: The labels of the container (as EnhancedDict - see below)
• env: The environment variables of the container as EnhancedDict
• networks: The list of networks the container is attached to (as NetworkList)
• ports: The list of ports exposed by the container as EnhancedDict having tcp and udp ports as EnhancedList

The utils.EnhancedDict class is a Python dictionary extension to allow referring to keys in it as properties - for example: container.ports.tcp instead of container['ports']['tcp']. Property names are also case-insensitive.
The models.ContainerInfo class extends utils.EnhancedDict to provide these features.

The utils.EnhancedList class is a Python list extension having additional properties for getting the first or last element and the first_value - e.g. the first element that is not None or empty.

The resources.ResourceList extends EnhancedList to provide a matching(target) method that allows getting the first element of the list having a matching ID or name. The resources.ContainerList extends the matching method to also match by Compose or Swarm service name for containers. The resources.ServiceList extends matching by Swarm service name and the resources.TaskListcan also match by container ID, service ID or service name. Tasks can also be filtered using their status and thewith_statusmethod. Theresources.NetworkList` class adds matching by network ID.

An example for matching could be containers on the same network in a Compose project:

{% set reference = containers.matching('web').first %}
targets:
{% for container in containers %}
  - "http://{{ container.networks.matching(reference).first.ip_address }}:{{ container.ports.tcp.first_value }}/{{ container.name }}"
{% endfor %}

This would take the web container as a reference and list targets with the IP address taken from the first matching network using the reference.

Apart from the built-in Jinja template filters⁠ the any and all filters are also available to evaluate conditions using the Python built-in functions with the same name.

⁠Updating the target file

The application listens for Docker start, stop, die and health_status events by default from containers and schedules an update (can be configured by the --events flag). If the generated content didn't change and the target already has the same content then the process stops.

If the template and the runtime information produces changes in the target file's content then a notification is scheduled according to the intervals set at startup. If there is another notification scheduled before the minimum interval is reached then it is being rescheduled unless the time since the first generation has passed the maximum interval already. This ensures batching notifications together in case many events arrive close to each other. See the timer.NotificationTimer class for implementation details.

⁠Signalling others

When the contents of the target file have changed the application can either restart containers or send UNIX signals to them to let them know about the change. Matching containers is done as described on the help text of the --restart argument.

For example if we have a couple of containers running with the service name nginx managed by a Compose project, a --signal nginx HUP command would send a SIGHUP signal to each of them to get them to reload their configuration.

Both of these work with Swarm when target containers might be running on different nodes than the app itself - using a Swarm manager and workers that alters the behavior slightly. For restarts, the manager app will restart matched Swarm services then stop if any of them was found, otherwise the workers will execute the restarts against containers matched locally. Signalling tasks in Swarm is not supported AFAIK, so it is always done using workers that will send the signal one-by-one to containers matched locally.

See how to configure the Swarm manager and workers below.

⁠Swarm support

To be able to execute actions as described above and to be notified of container events happening on remote Swarm nodes the app can be run as a cooperating pair of a Swarm manager and a number of Swarm workers. The manager should be run as a single instance on a manager node (the node.role==manager constraint can be used when scheduling the tasks) while the workers should run in global mode so every node in the Swarm would have one instance running.

Communication between the manager and the workers is done using HTTP requests. The manager uses port 9411 to accept events from the workers and those use port 9412 to accept action commands from the manager. None of these ports have to be exposed externally, the instances will be able to talk to each other as long as they are on the same overlay network. If the app is not running from Docker containers then these ports will have to be accessible though.

To enable the Swarm manager mode on the main app, use the --swarm-manager flag along with the --workers parameter that contains the hostname(s) of the workers to contact when executing actions.

The Swarm worker app is started using an alternative cli module:

usage: swarm_worker.py [-h] --manager <HOSTNAME> [--retries RETRIES]
                       [--events <EVENT> [<EVENT> ...]] [--debug]

PyGen cli to send HTTP updates on Docker events

optional arguments:
  -h, --help            show this help message and exit
  --manager <HOSTNAME>  The target hostname of the PyGen manager instance
                        listening on port 9411
  --retries RETRIES     Number of retries for sending an update to the manager
  --events <EVENT> [<EVENT> ...]
                        Docker events to watch and trigger updates for
                        (default: start, stop, die, health_status)
  --debug               Enable debug log messages

The only required parameter is the --manager containing the hostname of the Swarm manager app listening for remote events.

The worker app is available as a Docker image too using tags prefixed with worker:

• worker for x86 architecture
• worker-armhf for 32-bits ARM
• worker-aarch64 for 64-bits ARM

An example configuration for a Swarm manager and workers in a Composefile could be:

version: '3.4'
services:

  nginx:
    image: nginx
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - /var/pygen/nginx-config:/etc/nginx/conf.d
  
  nginx-pygen:
    image: rycus86/docker-pygen
    command: >
      --template /etc/docker-pygen/templates/nginx.tmpl
      --target /etc/nginx/conf.d/default.conf
      --signal nginx HUP
      --interval 3 10
      --swarm-manager
      --workers tasks.mystack_nginx-pygen-worker
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == manager
    volumes:
      - /var/pygen/nginx-config:/etc/nginx/conf.d
      - /var/pygen/nginx-pygen.tmpl:/etc/docker-pygen/templates/nginx.tmpl:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro

  nginx-pygen-worker:
    image: rycus86/docker-pygen:worker
    command: --manager mystack_nginx-pygen
    read_only: true
    deploy:
      mode: global
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro

When deployed using the mystack stack name the nginx-pygen manager app will handle updates to the target configuration file while the nginx-pygen-worker worker apps will collect Docker events and forward it to the manager. They will also take care of signalling the nginx container on configuration change, in particular the worker app running on the same node will, the others will ignore the action.

⁠Testing

The project uses the built-in Python unittest library for testing. The test files are in the tests folder and they use the test_*.py file name pattern.

The unit tests can be started with:

PYTHONPATH=src python -m unittest discover -s tests -v

The integration tests are also written in Python and use Docker in Docker (dind)⁠ (more information⁠). It will start containers having the Docker daemon and start containers inside those to execute the tests and check the expected outcome.

The integration tests are in the same tests folder with the it_*.py pattern and they can be executed using:

PYTHONPATH=tests python -m unittest -v integrationtest_helper

⁠Acknowledgement

This tool was inspired by the awesome jwilder/docker-gen⁠ project that is written in Go and uses Go templates for configuration generation. Many of the functionality here match or are related to what's available there.