ictu/semantiql repository overview

2021-12-20: This repository is archived

⁠SemantiQL, a GraphQL server for ICTU infra facts

SemantiQL is a GraphQL service, that we use as a configuration management database. It gathers information from several sources and exposes it in a unified way. This allows every client interested in this information to query a single service using a single protocol. This approach has helped us to greatly reduce the complexity of the platform operation.

⁠Data sources

SemantiQL gathers data from a few sources. Some are relatively static, for example the project deployment data, and some are much more dynamic like the current state of the running instances.

⁠Project facts

These are largely static facts that describe the configuration of each project. We use them for nightly deployments of the platform.

These facts are just a set of simple YAML files. They are currently at http://www.gitlab.infra.ictu/automation/facts/⁠.

These facts include information about the IP addresses of the VMs (nodes) each project id deployed on, IP address of the storage (NFS) node, address of the jenkins build slave, what is the project vlan tag, what essential services this project uses and what are their versions (and some additional configuration data), what kind of monitoring checks does the platform perform for them, etc.

If you need to add project facts just clone the facts repository, add/change the relevant files and push it. SemantiQL will query the git repository at FACTS_BASE_URL. This means, that you can branch/tag and deploy a SemantiQL that will use different set of facts, the ones from your branch.

⁠Rocket.Chat

This is a very simple data source, that provides the private room id of every project. We use this to secure the capabilities we expose to the teams via Ava⁠.

⁠Bigboat instance data

This data source provides information about the instances running on the project dashboard(s). The information includes what instances are running, what application definition and version was used to start them, etc. It also provides system data about the dashboard itself, for example (API) version. We use this information to assemble an overview of the entire platform.

⁠Queries

⁠Schema

SemantiQL supports the following top level queries (GraphQL syntax):

projects(key: String, archived: Boolean, name: String, fullName: String, rocketChatRoomId: String, vlanIsolated: Boolean): [Project]
dashboards(projectKey: String, projectName: String, url: String, dbName: String, dbUser: String, dbPassword: String, instanceName: String, infraAgent: String, host: String, nfs: String, runningAppNames: [String], archived: Boolean, vlanIsolated: Boolean): [Dashboard]
reportings(projectKey: String): [Reporting]
dockerRegistries(projectKey: String): [DockerRegistry]
nexus(projectKey: String): [DefaultApp]
sonarqubes(projectKey: String): [Sonarqube]
wekans(projectKey: String): [DefaultApp]
jenkins(projectKey: String): [Jenkins]
gitLabs(projectKey: String): [GitLab]
appStarter(projectKey: String): [AppStarter]
monitoring(archived: Boolean): Monitoring
httpChecks(archived: Boolean): [HTTPcheck]
sshChecks(archived: Boolean): [SSHcheck]

⁠Querying on the command line

You'd query SemantiQL via HTTP POST requests.

To simplify command line queries we have developed a Docker image, that makes it rather easy. For example, if you want to find out what is the vlan tag of the RWS project you'll do something like

docker run --rm ictu/gql '{projects(key: "rws") {name, vlan}}'

This will return

{"projects":[{"name":"RWS","vlan":"3131"}]}

This image has jq⁠ installed so you can extract values easily.

⁠Pipeline use

The main usage scenario for SemantiQL is to support the automatic (re)deployment of the BigBoat platform. Main pipelines that use it are:

• coreos-deploy⁠
• coreos-deploy-simple⁠
• bigboat-deploy⁠

⁠SemantiQL UI

SemantiQL UI⁠ is a React⁠ web application that makes the configuration database easier to use by humans. Accessible at http://www.semantiql.infra.ictu/⁠.

⁠How to develop

SemantiQL is a NodeJS⁠ application written in TypeScript⁠. We use ts-node⁠ to run it directly, without transpilation.

CircleCI⁠ is used to automatically build the SemantiQL Docker image - ictu/semantiql⁠. Docker images are tagged with the git tags.