fnndsc/cube repository overview

⁠ ChRIS_ultron_backEnd

ChRIS is an open-source platform for containerized medical compute.

https://chrisproject.org/⁠

⁠TL;DR

With Docker Compose⁠ and just⁠ installed, run

git clone https://github.com/FNNDSC/ChRIS_ultron_backEnd.git
cd ChRIS_ultron_backEnd
just

⁠Introduction

The ChRIS backend, a.k.a. ChRIS Ultron Backend or CUBE for short, is a component of the ChRIS system. It is responsible for maintaining the database of users, files, plugins, and pipelines.

Here lives the code of CUBE. It is a Django project using PostgreSQL and Celery. The HTTP API primarily supports the collection+json⁠ content-type.

⁠Development

Development is mainly supported on Linux. MacOS and WSL on Windows also work (because Docker Desktop is a Linux VM). You will need at least 8GM RAM, 20GB disk space, and a good internet connection.

Install Docker (version 27 or above) or Podman (version 5.2 or above), Docker Compose, and just⁠.

systemctl --user start podman.service

just prefer podman  # or
just prefer docker

⁠Just Commands

Development is handled by just⁠. Running CUBE in development mode is as-simple-as running the command:

just

The first run of just will take 5-20 minutes because it needs to pull and build container images. Subsequent runs should only take 1-5 minutes.

CUBE is now running at http://localhost:8000/api/v1/⁠. You can click around in the web browser. Alternatively, check out chrs⁠ and/or ChRIS_ui⁠.

Run tests:

just test-all                       # run all tests
just test-unit                      # run unit tests
just test-integration               # run integration tests
just test feeds.tests.test_views    # run chris_backend/feeds/tests/test_views.py

Shut down and clean up:

just nuke

List all just commands:

just --list --unsorted

⁠Just Commands for Object Storage Backend Environments (s3, swift)

Just use the same commands as explained above but explicitly set the storage env at the beginning of the session. For instance running CUBE in development mode with S3 storage is as-simple-as running the command:

just set-storage s3 
just

You can later unset the storage env with:

just unset-storage

⁠Development Tips and Tricks

⁠Recreate containers after changing docker-compose.yml

If you modify docker-compose.yml (or the override compose files when developing for other storage envs), you need to recreate/restart the affected services.

just up

⁠Rebuild the image after changing package dependencies

If you modify Dockerfile, pyproject.toml or uv.lock, you need to rebuild the image and recreate your containers.

just build && just up

Dependencies are managed with uv⁠ and the resolved versions are committed in uv.lock. Change a version constraint in pyproject.toml, then refresh the lockfiles:

just lock                    # re-resolve both CUBE and the benchmark harness
just lock-upgrade django     # bump a single dependency
just lock-check              # fail if a lockfile is stale (also runs in CI)

These run uv in a throwaway container, so no local uv installation is required. The benchmark harness has its own benchmarks/pyproject.toml and benchmarks/uv.lock, because docker-compose.benchmark.yml builds it with context: ./benchmarks.

⁠Trying HTTP requests from the CLI

For CLI tools, I recommend xh⁠ and jnv⁠ or jq⁠. Example:

xh -a chris:chris1234 :8000/api/v1/ | jnv

⁠Interactive shell

It is often easiest to debug things using a shell.

just bash    # run bash in a container
# -- or --
just shell   # run a Python REPL

In the Python REPL, you can import models and interact with them. Here is some common starter code:

from django.conf import settings
from django.contrib.auth.models import User, Group
from plugins.models import *
from plugininstances.models import *
from core.storage import connect_storage

storage = connect_storage(settings)

⁠IDE Setup

Visual Studio Code and PyCharm both support using Docker containers to run the Python interpreter. Run just build, then point your IDE to use the container image localhost/fnndsc/cube:dev.

Not all text editors support using Docker, or configuring the LSP might be inconvenient. In these cases, you need to install Python and the dependencies on-the-metal.

⁠Installing Python Dependencies On-The-Metal

Install uv⁠, then run

uv sync

uv sync creates .venv/, fetches Python 3.12 if you do not have it, and installs the exact versions recorded in uv.lock — including the dev dependency group, which is what an LSP wants.

Some dependencies (such as python-ldap) build C code during installation, requiring clang and the OpenLDAP headers regardless of which installer you use. If that is inconvenient, use micromamba to provide Python 3.12 and a prebuilt python-ldap, and point uv at that environment for everything else. Install micromamba⁠, then run

micromamba create -p ./.mambaenv -c conda-forge -y python=3.12 python-ldap=3.4
UV_PROJECT_ENVIRONMENT=./.mambaenv uv sync --inexact

--inexact leaves the packages micromamba installed alone instead of pruning them. Now the environment is ready. Activate the environment

micromamba activate -p ./.mambaenv

And you are ready to run your LSP + text editor.

⁠OpenAPI Compatibility

drf-spectacular⁠ is used to generate an OpenAPI schema. Generally, its introspection is capable and accurate, but some special care is necessary to make everything work. See:

• https://github.com/FNNDSC/ChRIS_ultron_backEnd/pull/584⁠
• https://github.com/tfranzel/drf-spectacular/issues/1303#issuecomment-2387220962⁠
• https://drf-spectacular.readthedocs.io/en/latest/faq.html#my-get-queryset-depends-on-some-attributes-not-available-at-schema-generation-time⁠

Be sure to lint your OpenAPI annotations for errors and warnings by running:

just openapi > /dev/null
just openapi-split > /dev/null

⁠Production Deployment

See https://chrisproject.org/docs/run/helm⁠

⁠GitHub Actions

This repository can also be used as a GitHub Actions step for running CUBE integration tests, e.g.

name: CI

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

jobs:
  test:
    runs-on: ubuntu-24.04
    steps:
      - name: Run ChRIS backend integration tests
        uses: FNNDSC/ChRIS_ultron_backEnd@master
        # all inputs are optional
        with:
          engine: docker  # or podman
          command: test-integration  # or test-unit, ...
        # optionally change image used for pfcon, or cube
        env:
          CUBE_IMAGE: localhost/fnndsc/cube:dev
          PFCON_IMAGE: localhost/fnndsc/pfcon:dev

⁠Load & Scalability Benchmarks

A repo-native load & scalability benchmark harness lives under benchmarks/⁠. It drives real plugin-instance DAGs end-to-end against the dev stack and auto-escalates one factor at a time (file count, concurrent feeds, chain depth, branching/merging) until CUBE breaks, then writes a ticket-ready report with the breaking points and approach-to-failure curves.

just bench-start              # bring up the fslink stack with the uvicorn envelope + plugins
just bench-run --tier smoke   # ~2 min sanity sweep (also: default, full, aging-grow, aging-probe)
just bench-down               # stop the stack

See benchmarks/README.md⁠ for usage and benchmarks/STRATEGY.md⁠ for the design.

⁠Documentation

⁠REST API reference

Available here⁠.

Install Sphinx and the http extension (useful to document the REST API)

pip install Sphinx
pip install sphinxcontrib-httpdomain

Build the html documentation

cd docs/
make html

⁠ChRIS REST API design.

Available here⁠.

⁠ChRIS backend database design.

Available here⁠.