Export an Elasticsearch index to S3
5.9K
Easily export an Elasticsearch index to Amazon S3. The image will export the index to a gzip file on Amazon S3.
docker run --rm -e "ES_URL=http://my.elasticsearch.server.com/" -e "ES_INDEX=my-index" -e "S3_BUCKET=my-bucket" -e "S3_KEY=es-test.gz" ianneub/elasticsearch-to-s3
All configuration is done using environment variables. See the table below for all configuration settings.
| Variable | Default | Description |
|---|---|---|
ES_URL | none | This is the Elasticsearch server URL. |
ES_INDEX | none | This is the Elasticsearch index you want to export. |
S3_BUCKET | none | This is the S3 bucket that the exported data will be saved to. |
S3_KEY | none | This is the name of the key that will be saved in the S3 bucket. |
AWS_ACCESS_KEY_ID | none | Your AWS credentials. Do not specify if you want to use an IAM profile while opperating in EC2. |
AWS_SECRET_ACCESS_KEY | none | Your AWS credentials. Do not specify if you want to use an IAM profile while opperating in EC2. |
AWS_DEFAULT_REGION | us-east-1 | The region your S3 bucket is in. |
The file that will be output to S3 is in gzip format. The output file consists of multiple lines of JSON. The first line is a header JSON string that contains information about the index. Each following line is a JSON string representing a document from the Elasticsearch index.
The first line in the output contains JSON with information about the index. It has the following keys:
| Key | Description |
|---|---|
| settings | This is the settings for the index returned from the Elasticsearch index settings API. |
| mapping | This is the mapping for the index returned from the Elasticsearch mapping API. |
Development of this docker image uses Docker Compose. Simply install docker and then run the following commands to get going:
docker-compose up -d es, then wait about 10 seconds for Elasticsearch to start up.docker-compose up fill to load some dummy data into Elasticsearch.docker-compose up export to run the export script.docker-compose up import to run the import script.Content type
Image
Digest
Size
278.1 MB
Last updated
over 8 years ago
docker pull ianneub/elasticsearch-to-s3