elasticsearch backup and restore

elasticsearch 备份说明

Elasticsearch的备份分两步:
1.创建一个仓库
  备份数据之前,要创建一个仓库来保存数据,仓库的类型支持共享文件系统,Amazon S3,HDFS和Azure Cloud
2.备份指定索引

elasticsearch 备份

elasticsearch 创建一个仓库

1
2
3

//查看当前所有仓库
[root@10 ~]# curl -XGET http://10.0.3.41:9200/_snapshot/_all?pretty
{ }

//创建一个仓库
[wisdom@10 ~]$ curl -XPUT '10.0.3.41:9200/_snapshot/esbackup' -d '
> {
>  "type": "fs",
>  "settings": {
>    "location": "/mnt/ops/elasticsearch",
>    "compress" : "true",
>    "max_snapshot_bytes_per_sec" : "80mb",
>    "max_restore_bytes_per_sec" : "80mb"
>  }
> }'
{"acknowledged":true}

// 使用POST来修改配置
[wisdom@10 ~]$ curl -XPUT '10.0.3.41:9200/_snapshot/esbackup' -d '
> {
>  "type": "fs",
>  "settings": {
>    "max_snapshot_bytes_per_sec" : "100mb",
>    "max_restore_bytes_per_sec" : "100mb"
>  }
> }'
{"acknowledged":true}

//查看当前仓库信息
[wisdom@10 ~]$ curl -XGET http://10.0.3.41:9200/_snapshot/esbackup?pretty     
{
  "esbackup" : {
    "type" : "fs",
    "settings" : {
      "location" : "/mnt/ops/elasticsearch",
      "max_restore_bytes_per_sec" : "80mb",
      "compress" : "true",
      "max_snapshot_bytes_per_sec" : "80mb"
    }
  }
}

//查看当前所有仓库信息
[wisdom@10 ~]$ curl -XGET http://10.0.3.41:9200/_snapshot/_all?pretty
{
  "esbackup" : {
    "type" : "fs",
    "settings" : {
      "location" : "/mnt/ops/elasticsearch",
      "max_restore_bytes_per_sec" : "80mb",
      "compress" : "true",
      "max_snapshot_bytes_per_sec" : "80mb"
    }
  }
}

//查看仓库是否创建成功
[wisdom@10 ~]$ curl -XPOST http://10.0.3.41:9200/_snapshot/esbackup/_verify?pretty
{
  "nodes" : {
    "j0bxQZGZSbW1lNoD9d2WHQ" : {
      "name" : "10.0.3.41"
    },
    "lhlUZILyRAm2Mtmp8czEOw" : {
      "name" : "10.0.3.42"
    },
    "DKfbgvGYTNO48BdMI3eeQA" : {
      "name" : "10.0.3.40"
    }
  }
}

//删除仓库
[root@10 ~]# curl -XDELETE http://10.0.3.40:9200/_snapshot/esbackup?pretty  
{
  "acknowledged" : true
}

补充: 一个权限错误
[wisdom@10 ~]$ curl -XPUT '10.0.3.41:9200/_snapshot/es' -d '      
{
 "type": "fs",
 "settings": {
   "location": "/mnt/ops/elasticsearch",
   "max_snapshot_bytes_per_sec" : "80mb",
   "max_restore_bytes_per_sec" : "80mb"
 }
}'
{"error":{"root_cause":[{"type":"access_denied_exception","reason":"access_denied_exception: /mnt/ops/elasticsearch/tests-ggM3LzbjTn2d0pWjwj7WpQ/master.dat-temp"}],"type":"repository_verification_exception","reason":"[es] path  is not accessible on master node","caused_by":{"type":"access_denied_exception","reason":"access_denied_exception: /mnt/ops/elasticsearch/tests-ggM3LzbjTn2d0pWjwj7WpQ/master.dat-temp"}},"status":500}

问题解决:
1. 查看test目录
[root@10 ~]# ls -l /mnt/ops/elasticsearch/                                 
total 0
drwxr-xr-x 2 1001 1001 0 Mar 12 03:10 tests-ggM3LzbjTn2d0pWjwj7WpQ

2. 我使用的是普通用户wisdom启动的elasticsearch,三台es的uid和gid都是1000
[wisdom@10 ~]$ tail /etc/passwd|grep wisdom
wisdom:x:1000:1000::/home/wisdom:/bin/bash

此时,我们已经发现问题,创建的test目录的权限是1001,和我们使用的用户对不上号

3. 1001是什么用户？经排查是samba的权限账号
//Samba server:
[root@10 ~]# tail /etc/passwd|grep 1001
smbuser:x:1001:1001::/home/smbuser:/sbin/nologin

//查看ES集群挂载点
[root@10 ~]# cat /etc/fstab |grep elasticsearch
//10.0.2.115/elasticsearch /mnt/ops/elasticsearch cifs defaults,username=smbuser,password=xxxxxx 0 0

至此，问题已经找到. 解决就很快了.

4. 在Samba服务器上创建UID和GID与三台ES主机中ES启动用户相同的用户
//Samba server:
[root@10 ~]# useradd -u 1000 -g 1000 wisdom
[root@10 ~]# smbpasswd -a wisdom

//ES集群
[root@10 ~]# umount /mnt/ops/elasticsearch
[root@10 ~]# vim /etc/fstab |grep elasticsearch
//10.0.2.115/elasticsearch /mnt/ops/elasticsearch cifs defaults,username=wisdom,password=xxxxxx 0 0
[root@10 ~]# mount -a
[root@10 ~]# smbclient -U wisdom //10.0.2.115/elasticsearch

5. 重新创建仓库,成功

elasticsearch 备份索引

1
2
3

一个仓库可以包含多个快照(snapshots),快照可以存所有的索引,部分索引或者一个单独的索引

注意: 快照只会备份正在运行open状态的索引！！！

//备份所有索引到一个快照中
[wisdom@10 ~]$ curl -XPUT http://10.0.3.40:9200/_snapshot/esbackup/snapshot_all
直接返回{"accepted":true},然后备份在后台执行

or:
[wisdom@10 ~]$ curl -XPUT http://10.0.3.40:9200/_snapshot/esbackup/snapshot_all?wait_for_completion=true
只有备份完成后,才会显示{"accepted":true}

1
2
3

//备份部分索引
[wisdom@10 ~]$ curl -XPUT http://10.0.3.40:9200/_snapshot/esbackup/snapshot_index -d '{ "indices": "index_1,index_2" }'
会将索引index_1和index_2备份到快照snapshot_index中

1
2
3

//备份单个索引
[wisdom@10 ~]$ curl -XPUT http://10.0.3.40:9200/_snapshot/esbackup/snapshot_index_1 -d '{ "indices": "index_1" }'
会将索引index_1备份到快照snapshot_index_1中

//查看快照信息
[wisdom@10 ~]$ curl -XGET http://10.0.3.41:9200/_snapshot/esbackup/snapshot_index_1?pretty
{
  "snapshots" : [ {
    "snapshot" : "snapshot_index_1",
    "version_id" : 2030399,
    "version" : "2.3.3",
    "indices" : [ "snapshot_index_1" ],
    "state" : "SUCCESS",
    "start_time" : "2017-03-11T20:53:11.478Z",
    "start_time_in_millis" : 1489265591478,
    "end_time" : "2017-03-11T20:53:46.128Z",
    "end_time_in_millis" : 1489265626128,
    "duration_in_millis" : 34650,
    "failures" : [ ],
    "shards" : {
      "total" : 5,
      "failed" : 0,
      "successful" : 5
    }
  } ]
}

1 2	//查看所有快照信息 [wisdom@10 ~]$ curl -XGET http://10.0.3.41:9200/_snapshot/esbackup/_all?pretty

//查看快照状态信息
[wisdom@10 ~]$ curl -XGET http://10.0.3.41:9200/_snapshot/esbackup/snapshot_index_1/_status?pretty
{
  "snapshots" : [ {
    "snapshot" : "snapshot_index_1",
    "repository" : "esbackup",
    "state" : "SUCCESS",
    "shards_stats" : {
      "initializing" : 0,
      "started" : 0,
      "finalizing" : 0,
      "done" : 5,
      "failed" : 0,
      "total" : 5
    },
    "stats" : {
      "number_of_files" : 497,
      "processed_files" : 497,
      "total_size_in_bytes" : 2141849055,
      "processed_size_in_bytes" : 2141849055,
      "start_time_in_millis" : 1489265591607,
      "time_in_millis" : 34498
    },
    "indices" : {
      "snapshot_index_1" : {
        "shards_stats" : {
          "initializing" : 0,
          "started" : 0,
          "finalizing" : 0,
          "done" : 5,
          "failed" : 0,
          "total" : 5
        },
        "stats" : {
          "number_of_files" : 497,
          "processed_files" : 497,
          "total_size_in_bytes" : 2141849055,
          "processed_size_in_bytes" : 2141849055,
          "start_time_in_millis" : 1489265591607,
          "time_in_millis" : 34498
        },
        "shards" : {
          "0" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 94,
              "processed_files" : 94,
              "total_size_in_bytes" : 429108117,
              "processed_size_in_bytes" : 429108117,
              "start_time_in_millis" : 1489265591663,
              "time_in_millis" : 12916
            }
          },
          "1" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 82,
              "processed_files" : 82,
              "total_size_in_bytes" : 429223456,
              "processed_size_in_bytes" : 429223456,
              "start_time_in_millis" : 1489265591607,
              "time_in_millis" : 32220
            }
          },
          "2" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 115,
              "processed_files" : 115,
              "total_size_in_bytes" : 428510263,
              "processed_size_in_bytes" : 428510263,
              "start_time_in_millis" : 1489265591615,
              "time_in_millis" : 34152
            }
          },
          "3" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 97,
              "processed_files" : 97,
              "total_size_in_bytes" : 427697132,
              "processed_size_in_bytes" : 427697132,
              "start_time_in_millis" : 1489265591619,
              "time_in_millis" : 34261
            }
          },
          "4" : {
            "stage" : "DONE",
            "stats" : {
              "number_of_files" : 109,
              "processed_files" : 109,
              "total_size_in_bytes" : 427310087,
              "processed_size_in_bytes" : 427310087,
              "start_time_in_millis" : 1489265591611,
              "time_in_millis" : 34494
            }
          }
        }
      }
    }
  } ]
}

1 2	//删除快照 [wisdom@10 ~]$ curl -XDELETE http://10.0.3.41:9200/_snapshot/esbackup/snapshot_index_1

elasticsearch 恢复

1 2	//恢复快照 [wisdom@10 ~]$ curl -XPOST http://10.0.3.41:9200/_snapshot/esbackup/snapshot_index_1/_restore

//恢复快照中指定索引
[wisdom@10 ~]$ curl -XPOST http://10.0.3.41:9200/_snapshot/esbackup/snapshot_index_1/_restore -d '
{
  "indices": "index_1",
  "rename_pattern": "index_(.+)",
  "rename_replacement": "restored_index_$1"
}'
说明:
* indices表示只恢复索引index_1
* 参数rename_pattern和rename_replacement 用来正则匹配要恢复的索引,并且重命名
* 和备份一样,api会立刻返回值,然后在后台执行恢复,使用wait_for_completion 标记强制同步执行

1
2
3

//查看恢复状态
[wisdom@10 ~]$ curl -XGET http://10.0.3.41:9200/_recovery/snapshot_index_1
[wisdom@10 ~]$ curl -XGET http://10.0.3.41:9200/_recovery/

1 2	//取消恢复过程(不管是已经恢复完,还是正在恢复)直接删除索引即可 [wisdom@10 ~]$ curl -XDELETE http://10.0.3.41:9200/_recovery/snapshot_index_1

smallasa

I known what i do !

elasticsearch backup and restore

elasticsearch 备份说明

elasticsearch 备份

elasticsearch 恢复