averemee/a2-cassandra-connect repository overview

You can dynamically specify configuration values with environment variables. You can use the Docker -e/--env or --env-file flags to specify various settings. You need to convert connect-distributed.properties file variables as described below and pass them as environment variables:

1. Convert to upper case
2. Prefix with A2_CASSANDRA_
3. Replace a period [.] with a single underscore (_]
4. Replace a dash [-] with double underscores [__]
5. Replace an underscore [_] with triple underscores [___]

⁠Ports and volumes

Standard Kafka Connect port 8083 is exposed, for storing secrets mountpoint /var/lib/connector is present

⁠Mandatory settings

A2_CASSANDRA_BOOTSTRAP_SERVERS A list of Kafka brokers to connect to

A2_CASSANDRA_GROUP_ID A unique string that identifies the Connect cluster group this Worker belongs to

A2_CASSANDRA_OFFSET_STORAGE_TOPIC The name of the existing topic where connector and task configuration offsets are stored. This must be the same for all Workers with the same A2_ORACDC_GROUP_ID (group.id). Always create it as a compacted, highly replicated (3x or more) topic with a large number of partitions (e.g., 25 or 50, just like Kafka’s built-in __consumer_offsets topic) to support large Kafka Connect clusters.

A2_CASSANDRA_CONFIG_STORAGE_TOPIC The name of the existing topic where connector and task configuration data are stored. This must be the same for all Workers with the same A2_ORACDC_GROUP_ID (group.id). Always create this topic as a compacted topic with a single partition and a high replication factor (3x or more).

A2_CASSANDRA_STATUS_STORAGE_TOPIC The name of the existing topic where connector and task configuration status updates are stored. This must be the same for all Workers with the same A2_ORACDC_GROUP_ID (group.id). Always create this topic as a compacted, highly replicated (3x or more) topic with multiple partitions

⁠a2-cassandra-connect configuration

Full list of connector configuration parameters is available here⁠

⁠Example configuration for connector itself

Create JSON config file (do not forget to replace connection details with your actual data) e.g.

{
  "name" : "cassandra-sink-topic-driven",
  "config" : {
    "connector.class" : "solutions.a2.kafka.cassandra.CassandraSinkConnector",
    "tasks.max" : "1",
    "topics" : "EMP",
    "errors.tolerance" : "none",
    "errors.log.enable" : "true",
    "errors.log.include.messages" : "true",
    "a2.contact.points" : "cassandra.eu-north-1.amazonaws.com",
    "a2.local.datacenter" : "eu-north-1",
    "a2.security" : "AWS_SIGV4",
    "a2.keyspace" : "AWS_TEST",
    "a2.keyspace.create.enabled" : "true",
    "a2.autocreate" : "true"
  }
}

Create connector by invoking REST API at exposed port

curl -s -X POST -H 'Content-Type: application/json' --data @connector.json http://<CONTAINER_IP>:8083/connectors