rycus86/webhook-proxy repository overview

⁠Webhook Proxy

A simple Python Flask⁠ REST server to accept JSON webhooks and run actions as a result.

⁠Usage

To start the server, run:

python app.py [server.yml]

If the parameter is omitted, the configuration file is expected to be server.yml in the current directory (see configuration details below).

The application can be run using Python 2 or 3.

⁠Configuration

The configuration for the server and its endpoints is described in a YAML file.

A short example:

server:
  host: '127.0.0.1'
  port: '5000'

endpoints:
  - /endpoint/path
      method: 'POST'

      headers:
        X-Sender: 'regex for X-Sender HTTP header'

      body:
        project:
          name: 'regex for project.name in the JSON payload'
          items:
            name: '^example_[0-9]+'
      
      actions:
        - log:
            message: 'Processing {{ request.path }} ...'

⁠server

The server section defines settings for the HTTP server receiving the webhook requests.

Set the host to 0.0.0.0 to accept connections from any hosts.

⁠endpoints

The endpoints section configures the list of endpoints exposed on the server.

Each endpoint supports the following configuration (all optional):

The message body validation supports lists too, the project.item.name in the example would accept {"project": {"name": "...", "items": [{"name": "exam ple_12"}]}} as an incoming body.

⁠actions

Action definitions support variables for most properties using Jinja2 templates. By default, these receive the following objects in their context:

• request : the incoming Flask request being handled
• timestamp : the Epoch timestamp as time.time()
• datetime : human-readable timestamp as time.ctime()
• error(..) : a function with an optional message argument to raise errors when evaluating templates
• context : a thread-local object for passing information from one action to another

Jinja2 does not let you execute code in the templates directly, so to use the error and context objects you need to do something like this:

{% if 'something' is 'wrong' %}
  
  {# treat it as literal (will display None) #}
  {{ error('Something is not right }}

  {# or use the assignment block with a dummy variable #}
  {% set _ = error() %}

{% else %}

  {% context.set('verdict', 'All good') %}

{% endif %}

## In another action's template:

  Previously we said {{ context.verdict }}

The following actions are supported (given their dependencies are met).

⁠log

The log action prints a message on the standard output.

⁠execute

The execute action executes an external command using subprocess.check_output. The output (string) of the invocation is passed to the Jinja2 template as result.

The shell parameter accepts:

• boolean : whether to use the default shell or run the command directly
• string : a shell command that supports -c
• list : for the complete shell prefix, like ['bash', '-c']

⁠http

The http action sends an HTTP request to a target and requires the requests Python module. The HTTP response object (from the requests module) is available to the Jinja2 template as response.

⁠docker

The docker action interacts with the Docker daemon and requires the docker Python module. It also needs access to the Docker UNIX socket at /var/run/docker.sock.

The action supports exactly one invocation on the Docker client (per action). Invocations (or properties) are keys starting with $ in the configuration, for example listing the containers would use $containers with $list as a sub-item. The result of the invocation (as an object from the Docker client) is available to the Jinja2 templates as result.

Examples:

...
  actions:
    - docker:
        $containers:
          $list:
            filters:
              name: '{{ request.json.repo.name }}'
        output: |
          Containers matching "{{ request.json.name }}":
          {% for container in result %}
           - {{ container.name }} @ {{ container.short_id }}
          {% endfor %}

    - docker:
        $info:
        output: 'Docker version: {{ result.ServerVersion }} on {{ result.OperatingSystem }}'

    - docker:
        $images:
          $pull:
            repository: '{{ request.json.namespace }}/{{ request.json.name }}'
            tag: '{{ request.json.get('tag', 'latest') }}'

    - docker:
        $run:
          image: 'alpine'
          command: 'echo "Hello {{ request.json.message }}!"'
          remove: true

⁠docker-compose

The docker-compose action interacts with Docker Compose and requires the docker-compose Python module.

The action supports exactly one invocation on the Docker Compose project (per action). The invocations are in the same format as with the docker action and the result is available for Jinja2 templates as result that is the return object from the Docker Compose invocation.

Examples:

...
  actions:
    - docker-compose:
        project_name: 'web'
        directory: '/opt/projects/web'
        $get_services:
        output: |
          Compose services:
          {% for service in result %}
           - service: {{ service.name }}
          {% endfor %}

    - docker-compose:
        project_name: 'backend'
        directory: '/opt/projects/compose_project'
        $up:
          detached: true
        output: |
          Containers started:
          {% for container in result %}
           - {{ container.name }}
          {% endfor %}

    - docker-compose:
        project_name: 'backend'
        directory: '/opt/projects/compose_project'
        $down:
          remove_image_type: false
          include_volumes: true
        output: 'Compose project stopped'

⁠Docker

The application can be run in Docker containers using images based on Alpine Linux for 3 processor architectures with the following tags:

• latest: for x86 hosts
  
• armhf: for 32-bits ARM hosts
  
• aarch64: for 64-bits ARM hosts
  

latest is auto-built on Docker Hub⁠ while the ARM builds are uploaded from Travis⁠.

The containers run as a non-root user.

To start the server:

docker run -d --name=webhook-proxy -p 5000:5000      \
    -v $PWD/server.yml:/etc/conf/webhook-server.yml  \
    rycus86/webhook-proxy:latest                     \
        /etc/conf/webhook-server.yml

Or put the configuration file at the default location:

docker run -d --name=webhook-proxy -p 5000:5000  \
    -v $PWD/server.yml:/app/server.yml           \
    rycus86/webhook-proxy:latest

There are 3 more tags available for images that can use the docker and docker-compose actions which are running as root user:

• docker: for x86 hosts
  
• armhf-docker: for 32-bits ARM hosts
  
• aarch64-docker: for 64-bits ARM hosts
  

Each of these are built on Travis⁠ and pushed to Docker Hub⁠.

To run these, the Docker daemon's UNIX socket needs to be mounted into the container too apart from the configuration file:

docker run -d --name=webhook-proxy -p 5000:5000      \
    -v $PWD/server.yml:/app/server.yml               \
    -v /var/run/docker.sock:/var/run/docker.sock:ro  \
    rycus86/webhook-proxy:docker

In Docker Compose on a 64-bit ARM machine the service definition could look like this:

version: '2'
services:

  webhooks:
    image: rycus86/webhook-proxy:aarch64
    ports:
      - 8080:5000
    volumes:
      - ./webhook-server.yml:/app/server.yml:ro

⁠Examples

Have a look at the sample.yml⁠ included in this repo to get a better idea of the configuration.

You can also find some examples with short explanation below.

• An externally available server listening on port 7000 and printing details about a GitHub push webhook

server:
  host: '0.0.0.0'
  port: '7000'

endpoints:
  - /github:
      method: 'POST'

      headers:
        X-GitHub-Delivery: '^[0-9a-f\-]+$'
        X-GitHub-Event: 'push'

      body:
        ref: 'refs/heads/.+'
        before: '^[0-9a-f]{40}'
        after: '^[0-9a-f]{40}'
        repository:
          id: '^[0-9]+$'
          full_name: 'sample/.+'
          owner:
            email: '.+@.+\..+'
        commits:
          id: '^[0-9a-f]{40}'
          message: '.+'
          author:
            name: '.+'
          added: '^(src/.+)?'
          removed: '^(src/.+)?'
        pusher:
          name: '.+'
          email: '.+@.+\..+'

      actions:
        - log:
            message: |
              Received a GitHub push from the {{ request.json.repository.full_name }} repo:
              - Pushed by {{ request.json.pusher.name }} <{{ request.json.pusher.email }}>
              - Commits included:
              {% for commit in request.json.commits %}
              +   {{ commit.id }}
              +   {{ commit.committer.name }} at {{ commit.timestamp }}
              +   {{ commit.message }}
              
              {% endfor %}
              Check this change out at {{ request.json.compare }}

The validators for the /github endpoint require that the X-GitHub-Delivery header is hexadecimal separated by dashes and the X-GitHub-Event header has the push value. The event also has to come from one of the repos under the sample namespace. Some of the commit hashes are checked that they are 40 character long hexadecimal values and the commit author's name has to be non-empty. The commits field is actually a list in the GitHub webhook so the validation is applied to each commit data individually. The added and removed checks for example accept if the commit has not added or removed anything but if it did it has to be in the src folder.

For valid webhooks the repository's name, the pushers name and emails are printed to the standard output followed by the ID, committer name, timestamp and message of each commit in the push. The last line displays the URL for the GitHub compare page for the change.

For more information about using the Jinja2 templates have a look at the official documentation⁠.

• Update a Docker Compose project on image changes

Let's assume we have a Compose project with a few services. When their image is updated in Docker Hub we want to pull it and get Compose to restart the related containers.

server:
  host: '0.0.0.0'
  port: '5000'

endpoints:
  - /webhook/dockerhub:
      method: 'POST'

      body:
        repository:
          repo_name: 'somebody/.+'
          owner: 'somebody'
        push_data:
          tag: 'latest'
      
      actions:
        - docker:
            $containers:
              $list:
            output: |
              {% for container in result if request.json.repository.repo_name in container.image.tags %}
                Found {{ container.name }} with {{ container.image }}
              {% else %}
                {% set _ = error('No containers found using %s'|filter(request.json.repo_name)) %}
              {% endfor %}
        - docker:
            $images:
              $pull:
                repository: '{{ request.json.repo_name }}'
                tag: '{{ request.json.tag }}'
        - docker-compose:
            project_name: 'autoupdate'
            directory: '/var/compose/project'
            $up:
              detached: true
            output: |
              Containers affected:
              {% for container in result %}
              {{ container.name }} <{{ container.short_id }}>

The /webhook/dockerhub endpoint will accept webhooks from somebody/* repos when an image's latest tag is updated. First a docker action checks that we already have containers running that use the image then another docker action pulls the updated image and finally the docker-compose action applies the changes by restarting any related containers.