It is a very simple method if you want to migrate your current elasticsearch cluster to a new version or To backup and restoration due to any failure, without loss of any data (Indices).

This article will show you how to create Elasticsearch snapshots from one elasticsearch cluster and restore those snapshots with another Elasticsearch cluster or with same easily.

First we need to know what is Elasticsearch Snapshots

ES Snapshots and Restore

A elasticsearch snapshotsis a backup of running elasticsearch cluster. The snapshot module allows you to create snapshots of your more than one or individual indices, or a snapshot of the whole cluster. The snapshot of your specific indices or whole cluster and store in a repository on a shared file system.

There are different types of repositories that are supported. If you have a shared file system, for example

• A NFS filesystem that is accessible by all elasticsearch nodes at the same mounting point, then you can use that to store your specific individual indices or whole cluster snapshot too.

Elasticsearch was designed to be run in different different environments,The snapshot and restore module also supports various cloud repositories such as:

• AWS (You can store the backups on S3)
• Google Cloud Storage
• Azure Cloud etc

Version compatibility

A snapshot contains a copy of the on-disk data structures that make up an index. This means that snapshots can only be restored to versions of Elasticsearch that can read the indices:

• A snapshot of an index created in 5.x can be restored to 6.x.
• A snapshot of an index created in 2.x can be restored to 5.x.
• A snapshot of an index created in 1.x can be restored to 2.x.

Conversely, snapshots of indices created in 1.x cannot be restored to 5.x or 6.x, and snapshots of indices created in 2.x cannot be restored to 6.x. Ref

Environment

1. Linux ( Ubuntu)
2. Elasticsearch 6.1.4
   10.0.0.1 ES Snapshot’s Cluster — — Cluster 1
   10.0.0.2 Restore’s ES Cluster — — Cluster 2
3. Files System (Local) you can use NFS or S3 too
4. Postman to hit Elasticsearch Snapshot and Restore module API or You can run from terminal too, here below both examples are explained.
5. Assume that login as root’s user

Snapshots

Create a folder es-backup it is accessible by all elasticsearch nodes at the same mounting point.

Let’s create a snapshot directory # mkdir  -p /elasticseacrhData/es-backup Now, we need to give elasticsearch permissions to write 
to this directory.
# chown  -R elasticsearch:elasticsearch  /elasticseacrhData/es-backup Or# chmod  -R o+rwx  /elasticseacrhData/es-backup  (it is Restricted  but you can do if this directory doesn’t publicly)

This will be the path where the snapshots repositories will be create.

10.0.0.1 — ES Snapshot’s — Cluster 1

Now we have to add the path.repo in the elasticsearch configuration file elasticsearch.ymlby editing the /etc/elasticsearch/elasticsearch.yml config following at the end of the file.

# cat >> /etc/elasticsearch/elasticsearch.yml  << EOFpath.repo: ["/elasticseacrhData/es-backup"]EOFOR# vim  /etc/elasticsearch/elasticsearch.yml
# action.destructive_requires_name: true
# ***********************************************
# Shared repo
#path.repo: ["/mount/backups", "/mount/longterm_backups"]path.repo: ["/elasticseacrhData/es-backup"]:wq Save and exit.

Now, restart the elasticsearch service and create a repository that we are going to use to store our snapshots.

# service elasticsearch restart

We can check if we have any repositories already setup with,

#curl -XGET 'http://localhost:9200/_snapshot/_all?pretty'{ }

The blank response that we got indicates that we don’t have any repositories setup yet.

Now we can create a repository on our cluster to backup to and then copy our backup directory to another server and restore.

We can create repository though Elasticsearch’s REST API. When creating the repository there are several options we can add, but for now I am only using to enable compression.

You can put other option as per your requirement like the chunk_sizeconfiguration option might be handy if you are compressing very large indices, max_restore_bytes_per_sec– Throttles per node restore rate- Defaults to 40mb per second, readonly-Makes repository read-only- Defaults to false

Create repository

From terminal

#curl -XPUT -H "Content-Type: application/json;charset=UTF-8" 'http://localhost:9200/_snapshot/esbackup' -d '{
  "type": "fs",
  "settings": {
     "location": "/elasticseacrhData/es-backup",
     "compress": true
  }
}'

We can check the repository was successfully created by listing all the repositories with same as above

#curl -XGET 'http://localhost:9200/_snapshot/_all?pretty
…………Response
{
    "esbackup": {
       "type": "fs",
        "settings": {
            "compress": "true",
            "location": "/elasticseacrhData/es-backup"
        }
    }
}

Once the repository created we can create a backup of the entire cluster or specific indices.

We are going to take full backup (entire cluster ) using a single curl command:

“A repository can contain multiple snapshots of the same cluster. Snapshots are identified by unique names within the cluster. A snapshot with the name linuxpoint_snapshot in the repository esbackupcan be created by executing the following command”

# curl -XPUT "http://localhost:9200/_snapshot/esbackup/linuxpoint_snapshot?wait_for_completion=true"……Response------{"snapshot":{"snapshot":"linuxpoint_snapshot","uuid":"OExYfcXNS5KAWFgUgZmCzQ","version_id":6010499,"version":"6.1.4","indices":[<==>],"state":"SUCCESS","start_time":"2018-09-20T07:45:20.662Z","start_time_in_millis":1537429520662,"end_time":"2018-09-20T07:45:36.793Z","end_time_in_millis":1537429536793,"duration_in_millis":16131,"failures":[],"shards":{"total":30,"failed":0,"successful":30}}}

This is how you create a snapshot by command prompt or by Postman.

“The wait_for_completionparameter specifies whether or not the request should return immediately after snapshot initialization (default) or wait for snapshot completion.”

We can list our snapshots by the below curl command

#curl -XGET "http://localhost:9200/_snapshot/esbackup/_all?pretty"

Restoring Process

You can use the restore process to even move data from one cluster to another cluster and then restore it.

Let’s follow the process, step by step.

We are going to move our data from cluster1 to cluster2.

# cd   elasticseacrhData/# ls 
--------
  es-backup
--------# tar czf   es-backup.tar.gz es-backupNow scp this tar.gz file to cluster 2 and then decompress it on cluster2.#scp /elasticseacrhData/es-backup.tar.gz root@10.0.0.2:/root/

10.0.0.2 — ES Restore’s — Cluster 2

Logged in cluster2 and decompress in elasticsearch snapshot directory#mv ~/es-backup.tar.gz   /elasticsearchData2/ 
#cd  elasticsearchData2  #tar xzf  es-backup.tar.gz  #chown -R elasticsearch:elasticsearch /elasticseacrhData2/es-backup

Now we have to add the path.repo in the elasticsearch configuration file elasticsearch.ymlby editing the /etc/elasticsearch/elasticsearch.ymlconfig following at the end of the file.

# cat >> /etc/elasticsearch/elasticsearch.yml << EOFpath.repo: ["/elasticseacrhData2/es-backup"]EOFOR# vim  /etc/elasticsearch/elasticsearch.yml
# action.destructive_requires_name: true
# ***********************************************
# Shared repo
#path.repo: ["/mount/backups", "/mount/longterm_backups"]
path.repo: ["/elasticseacrhData2/es-backup"]:wq Save and exit.

Now, restart the elasticsearch service and create a repository that we are going to use to store our snapshots.

# service elasticsearch restart

Note: Just do same as earlier done for above cluster 1

The next step is to register it as a repository. Now need to register a repository by Postmanor command prompt i.e curl.

# curl -XPUT -H "Content-Type: application/json;charset=UTF-8" 'http://10.0.0.2:9200/_snapshot/esrestore -d ‘{
  "type": "fs",
  "settings": {
     "location": "/elasticseacrhData2/es-backup",
     "compress": true
  }
}'# curl -XGET 'http://10.0.0.2:9200/_snapshot/_all?pretty'
...... response.....
{
    "esrestore": {
       "type": "fs",
        "settings": {
            "compress": "true",
            "location": "/elasticseacrhData2/es-backup"
        }
    }
}

Now restore our snapshot from the repository.

# curl  -XPOST "http://10.0.0.2:9200/_snapshot/esrestore/linuxpoint_snapshot/_restore?wait_for_completion=true"

ES Snapshot of specifies indices or individuals indices in the cluster is created

Logged in Cluster 1

# curl -X PUT -H "Content-Type: application/json" "http://localhost:9200/_snapshot/esbackup/linuxpoint_snapshot_2?wait_for_completion=true" -d '
{
  "indices": "index_1,index_2, index_3",
  "ignore_unavailable": true,
  "include_global_state": false
}'

“The snapshot request also supports theignore_unavailableoption. Setting it to truewill cause indices that do not exist to be ignored during snapshot creation. By default, when ignore_unavailableoption is not set and an index is missing the snapshot request will fail. By setting include_global_stateto false it’s possible to prevent the cluster global state to be stored as part of the snapshot.”

Restoring Process same as above.