svendowideit/cronicle

•Updated almost 8 years ago

asdf

Image

4.8K

svendowideit/cronicle repository overview

⁠Node.js v10.x ALERT WARNING CAUTION

There is a known bug in Node.js v10⁠ that causes all timers to stop firing after approximately 25 days of continuous running. Since Cronicle is a scheduler daemon, it relies completely on the Node.js timer system, so this is a death sentence. The bug affects Node.js versions v10.0.0 to v10.8.0. The bug is fixed in Node v10.9.0. Your options are as follows:

If you are running Node v10.x, make sure you are on v10.9.0 or later!
Run Cronicle under Node LTS (i.e. Node version 8). The bug was introduced in Node v10.0.0, so any version of Node prior to that will work fine.
Setup a weekly restart job. You can do this as an actual Cronicle event using the Shell Plugin⁠ and Detached Mode⁠, or just add a crontab to your server(s) like this:

echo "0 3 * * 0 /opt/cronicle/bin/control.sh restart" > /etc/cron.d/cronicle-weekly-restart.cron

Cronicle will emit a warning to the console if it is started up on any affected version of Node. For more information on this bug, see Issue #108⁠.

⁠Overview

Cronicle is a multi-server task scheduler and runner, with a web based front-end UI. It handles both scheduled, repeating and on-demand jobs, targeting any number of slave servers, with real-time stats and live log viewer. It's basically a fancy Cron⁠ replacement written in Node.js⁠. You can give it simple shell commands, or write Plugins in virtually any language.

Main Screenshot

⁠Features at a Glance

Single or multi-server setup.
Automated failover to backup servers.
Auto-discovery of nearby servers.
Real-time job status with live log viewer.
Plugins can be written in any language.
Schedule events in multiple timezones.
Optionally queue up long-running events.
Track CPU and memory usage for each job.
Historical stats with performance graphs.
Simple JSON messaging system for Plugins.
Web hooks for external notification systems.
Simple REST API for scheduling and running events.
API Keys for authenticating remote apps.

⁠Table of Contents

Glossary⁠

Installation⁠
Setup⁠
Configuration⁠
Web UI⁠
Plugins⁠
Command Line⁠
Inner Workings⁠
API Reference⁠
Development⁠
Colophon⁠
License⁠

⁠Glossary

A quick introduction to some common terms used in Cronicle:

Term	Description
Master Server	The primary server which keeps time and runs the scheduler, assigning jobs to other servers, and/or itself.
Backup Server	A slave server which will automatically become master and take over duties if the current master dies.
Slave Server	A server which sits idle until it is assigned jobs by the master server.
Server Group	A named group of servers which can be targeted by events, and tagged as "master eligible", or "slave only".
API Key	A special key that can be used by external apps to send API requests into Cronicle. Remotely trigger jobs, etc.
User	A human user account, which has a username and a password. Passwords are salted and hashed with bcrypt⁠.
Plugin	Any executable script in any language, which runs a job and reads/writes JSON to communicate with Cronicle.
Schedule	The master list of events, which are scheduled to run at particular times, on particular servers.
Category	Events can be assigned to categories which define defaults and optionally a color highlight in the UI.
Event	An entry in the schedule, which may run once or many times at any interval. Each event points to a Plugin, and a server or group to run it.
Job	A running instance of an event. If an event is set to run hourly, then a new job will be created every hour.

⁠Installation

Please note that Cronicle currently only works on POSIX-compliant operating systems, which basically means Unix/Linux and OS X. If there is enough interest, I'll look into making it work on Windows.

You'll need to have Node.js⁠ pre-installed on your server. Then become root and type this:

curl -s https://raw.githubusercontent.com/jhuckaby/Cronicle/master/bin/install.js | node

This will install the latest stable release of Cronicle and all of its dependencies⁠ under: /opt/cronicle/

If you'd rather install it manually (or something went wrong with the auto-installer), here are the raw commands:

mkdir -p /opt/cronicle
cd /opt/cronicle
curl -L https://github.com/jhuckaby/Cronicle/archive/v1.0.0.tar.gz | tar zxvf - --strip-components 1
npm install
node bin/build.js dist

Replace v1.0.0 with the desired Cronicle version from the release list⁠, or master for the head revision (unstable).

⁠Setup

If this is your first time installing, please read the Configuration⁠ section first. You'll likely want to customize a few configuration parameters in the /opt/cronicle/conf/config.json file before proceeding. At the very least, you should set these properties:

Key	Description
`base_app_url`	A fully-qualified URL to Cronicle on your server, including the `http_port` if non-standard. This is used in e-mails to create self-referencing URLs.
`email_from`	The e-mail address to use as the "From" address when sending out notifications.
`smtp_hostname`	The hostname of your SMTP server, for sending mail. This can be `127.0.0.1` or `localhost` if you have sendmail⁠ running locally.
`secret_key`	For multi-server setups (see below) all your servers must share the same secret key. Any randomly generated string is fine.
`job_memory_max`	The default maximum memory limit for each job (can also be customized per event and per category).
`http_port`	The web server port number for the user interface. Defaults to 3012.

Now then, the only other decision you have to make is what to use as a storage back-end. Cronicle can use local disk (easiest setup), Couchbase⁠ or Amazon S3⁠. For single server installations, or even single master with multiple slaves, local disk is probably just fine, and this is the default setting. But if you want to run a true multi-server cluster with automatic master failover, please see Multi-Server Cluster⁠ for details.

With that out of the way, run the following script to initialize the storage system. You only need to do this once, and only on the master server. Do not run this on any slave servers:

/opt/cronicle/bin/control.sh setup

Among other things, this creates an administrator user account you can use to login right away. The username is admin and the password is admin. It is recommended you change the password as soon as possible, for security purposes (or just create your own administrator account and delete admin).

At this point you should be able to start the service and access the web UI. Enter this command:

/opt/cronicle/bin/control.sh start

Give it a minute to decide to become master, then send your browser to the server on the correct port:

http://YOUR_SERVER_HOSTNAME:3012/

You only need to include the port number in the URL if you are using a non-standard HTTP port (see Web Server Configuration⁠).

See the Web UI⁠ section below for instructions on using the Cronicle web interface.

⁠Single Server

For a single server installation, there is nothing more you need to do. After installing the package, running the bin/control.sh setup script and starting the service, Cronicle should be 100% ready to go. You can always add more servers later (see below).

⁠Single Master with Slaves

The easiest multi-server Cronicle setup is a single "master" server with one or more slaves. This means that one server is the scheduler, so it keeps track of time, and assigns jobs for execution. Jobs may be assigned to any number of slave servers, and even the master itself. Slave servers simply sit idle and wait for jobs to be assigned by the master server. Slaves never take over master scheduling duties, even if the master server goes down.

This is the simplest multi-server setup because the master server can use local disk for all its storage. Slaves do not need access to the file storage. This is the default configuration, so you don't have to change anything at all. What it means is, all the scheduling data, event categories, user accounts, sessions, plugins, job logs and other data is stored as plain JSON files on local disk. Cronicle can also be configured to use a NoSQL database such as Couchbase⁠ or Amazon S3⁠, but this is not required.

So by default, when you run the setup script above, the current server is placed into a "Master Group", meaning it is the only server that is eligible to become master. If you then install Cronicle on additional servers, they will become slaves only. You can change all this from the UI, but please read the next section before running multiple master backup servers.

When installing Cronicle onto slave servers, please do not run the bin/control.sh setup script. Instead, simply copy over your conf/config.json file, and then start the service.

⁠Multi-Server Cluster

Cronicle also has the ability to run with one or more "backup" servers, which can become master if need be. Failover is automatic, and the cluster negotiates who should be master at any given time. But in order for this to work, all the master eligible servers need access to the same storage data. This can be achieved in one of three ways:

Use a shared filesystem such as NFS⁠.
Use a Couchbase⁠ server.
Use an Amazon S3⁠ bucket.

See the Storage Configuration⁠ section below for details on these.

The other thing you'll need to do is make sure all your master backup servers are in the appropriate server group. By default, a single "Master Group" is created which only contains your primary master server. Using the UI, you can simply change the hostname regular expression so it encompasses all your eligible servers, or you can just add additional groups that match each backup server. More details can be found in the Servers Tab⁠ section below.

⁠Load Balancers

You can run Cronicle behind a load balancer, as long as you ensure that only the master server and eligible backup servers are in the load balancer pool. Do not include any slave-only servers, as they typically do not have access to the back-end storage system, and cannot serve up the UI.

You can then set the base_app_url⁠ configuration parameter to point to the load balancer, instead of an individual server, and also use that hostname when loading the UI in your browser.

Note that Web UI needs to make API calls and open WebSocket⁠ connections to the master server directly, so it needs to also be accessible directly via its hostname.

You must set the web_direct_connect⁠ configuration property to true. This ensures that the Web UI will make API and WebSocket connections directly to the master server, rather than via the load balancer hostname.

⁠Ops Notes

For teams setting up multi-server clusters, here are some operational concerns to keep in mind:

All servers should have the same exact configuration file (/opt/cronicle/conf/config.json).
All servers need to have correct clocks (timezones do not matter, clock sync does).
Server auto-discovery happens via UDP broadcast on port 3014 (by default). This is not required.
The master server will also open TCP WebSocket⁠ connections to each slave on the web server port.
Each server in the cluster needs to have a fully-qualified hostname that resolves in DNS.
The server hostnames determine priority of which server becomes master (alphabetical sort).
All servers need to have unique hostnames (very bad things will happen otherwise).
All servers need to have at least one active IPv4 interface.
For the "live log" feature in the UI to work, the user needs a network route to the server running the job, via its hostname.
If you have to change any server IP addresses, they'll have to be removed and re-added to the cluster.
See the Cron Noncompliance⁠ section for differences in how Cronicle schedules events, versus the Unix Cron standard.

⁠Configuration

The main Cronicle configuration file is in JSON format, and can be found here:

/opt/cronicle/conf/config.json

Please edit this file directly. It will not be touched by any upgrades. A pristine copy of the default configuration can always be found here: /opt/cronicle/sample_conf/config.json.

⁠Basics

Here are descriptions of the top-level configuration parameters:

⁠base_app_url

This should be set to a fully-qualified URL, pointing to your Cronicle server, including the HTTP port number if non-standard. Do not include a trailing slash. This is used in e-mails to create self-referencing URLs. Example:

http://local.cronicle.com:3012

If you are running Cronicle behind a load balancer, this should be set to the load balanced virtual hostname.

⁠email_from

The e-mail address to use as the "From" address when sending out notifications. Most SMTP servers require this to be a valid address to accept mail.

⁠smtp_hostname

The hostname of your SMTP server, for sending mail. This can be set to 127.0.0.1 or localhost if you have sendmail⁠ running locally.

⁠smtp_port

The port number to use when communicating with the SMTP server. The default is 25.

⁠mail_options

Set specific mailer options, such as SMTP SSL and authentication, passed directly to pixl-mail⁠ (and then to nodemailer⁠). Example:

"mail_options": {
	"secure", true,
	"auth", { user: "fsmith", pass: "12345" },
	"connectionTimeout": 10000,
	"greetingTimeout": 10000,
	"socketTimeout": 10000
}

The connectionTimeout, greetingTimeout and socketTimeout properties are all expressed in milliseconds.

You can also use mail_options to use local sendmail⁠, if you have that configured on your server. To do this, set the following properties, and tune as needed:

"mail_options": {
	"sendmail": true,
	"newline": "unix",
	"path": "/usr/sbin/sendmail"
}

You can omit smtp_hostname and smtp_port if you are using sendmail.

⁠secret_key

For multi-server setups, all your Cronicle servers need to share the same secret key. Any randomly generated string is fine.

The install script will automatically set to this to a random string for you, but you are free to change it to anything you want. Just make sure all your servers have the same shared secret key. This is used to authenticate internal server-to-server API requests.

⁠log_dir

The directory where logs will be written, before they are archived. This can be a partial path, relative to the Cronicle base directory (/opt/cronicle) or a full path to a custom location. It defaults to logs (i.e. /opt/cronicle/logs).

⁠log_filename

The filename to use when writing logs. You have three options here: a single combined log file for all logs, multiple log files for each component, or multiple log files for each category (debug, transaction, error). See the Logs⁠ section below for details.

⁠log_columns

This is an array of column IDs to log. You are free to reorder or remove some of these, but do not change the names. They are specific IDs that match up to log function calls in the code. See the Logs⁠ section below for details.

⁠log_archive_path

Every night at midnight (local server time), the logs can be archived (gzipped) to a separate location. This parameter specifies the path, and the directory naming / filenaming convention of the archive files. It can utilize date placeholders including [yyyy], [mm] and [dd].

This can be a partial path, relative to the Cronicle base directory (/opt/cronicle) or a full path to a custom location. It defaults to logs/archives/[yyyy]/[mm]/[dd]/[filename]-[yyyy]-[mm]-[dd].log.gz.

⁠copy_job_logs_to

Your job logs (i.e. the output from Plugins) are stored separately from the main Cronicle log files. They are actually written to the Cronicle storage system, and made available in the UI. This parameter allows you to copy them to a separate directory on disk, presumably for some kind of log processing system (such as Splunk⁠) to then pick them up, index them, etc. It is optional, and defaults to disabled.

This can be a partial path, relative to the Cronicle base directory (/opt/cronicle) or a full path to a custom location.

⁠queue_dir

The queue directory is used internally for misc. background tasks, such as handling detached jobs sending messages back to the daemon. You shouldn't ever have to deal with this directly, and the directory is auto-created on install.

This can be a partial path, relative to the Cronicle base directory (/opt/cronicle) or a full path to a custom location.

⁠pid_file

The PID file is simply a text file containing the Process ID of the main Cronicle daemon. It is used by the control.sh script to stop the daemon, and detect if it is running. You should never have to deal with this file directly, and it defaults to living in the logs directory which is auto-created.

This can be a partial path, relative to the Cronicle base directory (/opt/cronicle) or a full path to a custom location. However, it should probably not be changed, as the control.sh script expects it to live in logs/cronicled.pid.

⁠debug_level

The level of verbosity in the debug logs. It ranges from 1 (very quiet) to 10 (extremely loud). The default value is 9.

⁠maintenance

Cronicle needs to run storage maintenance once per day, which generally involves deleting expired records and trimming lists which have grown too large. This only r

Tag summary

Recent tags

Content type

Image

Digest

Size

276.9 MB

Last updated

almost 8 years ago

docker pull svendowideit/cronicle