How to quickly change the number of replicas setting in an Elasticsearch Index

Written by Claudio Kuenzler - 0 comments

Published on July 19th 2021 - Listed in Elasticsearch ELK Monitoring

---

Depending on the environment where an Elasticsearch cluster is running, the (default) index settings might need to be changed. Especially if the cluster only exists with a one-node setup (which is nothing abnormal) indexes with a replica setting of 1 or more will report unassigned shards.

This can be verified with the check_es_system Elasticsearch monitoring plugin:

root@elasticsearch:~# /usr/lib/nagios/plugins/check_es_system.sh -H localhost -t status
ES SYSTEM WARNING - Elasticsearch Cluster "mycluster" is yellow (1 nodes, 1 data nodes, 3 shards, 0 relocating shards, 0 initializing shards, 3 unassigned shards, 10934 docs)

Unassigned shards and replica settings

The reason can be spotted when listing all the available shards:

root@elasticsearch:~# curl -q -s "http://localhost:9200/_cat/shards"
magento2_product_1_v204 0 p STARTED    3639 6.2mb 127.0.0.1 elasticsearch
magento2_product_1_v204 0 r UNASSIGNED                      
magento2_product_2_v204 0 p STARTED    3639 7.1mb 127.0.0.1 elasticsearch
magento2_product_2_v204 0 r UNASSIGNED                      
magento2_product_3_v204 0 p STARTED    3656 6.2mb 127.0.0.1 elasticsearch
magento2_product_3_v204 0 r UNASSIGNED  

All the unassigned shards have "r" as second flag - which stands for replicated shard. As this is a single node cluster, Elasticsearch does not find any other nodes where to replicate this shard to -> UNASSIGNED.

By looking at the settings of these three indexes, especially the number_of_replicas setting, all indexes have a (default) setting of 1 replica defined:

root@elasticsearch:~# curl -q -s "http://localhost:9200/magento2_product_1_v204/_settings" | jq -r '.magento2_product_1_v204.settings.index.number_of_replicas'
1

root@elasticsearch:~# curl -q -s "http://localhost:9200/magento2_product_2_v204/_settings" | jq -r '.magento2_product_2_v204.settings.index.number_of_replicas'
1

root@elasticsearch:~# curl -q -s "http://localhost:9200/magento2_product_3_v204/_settings" | jq -r '.magento2_product_2_v204.settings.index.number_of_replicas'
1

Changing the number_of_replicas setting

Fortunately this setting can be changed without downtime. With a PUT request the index settings can be updated:

root@elasticsearch:~# curl -X PUT -H "Content-Type: application/json" "http://localhost:9200/magento2_product_1_v204/_settings" -d '{ "number_of_replicas": 0 }'
{"acknowledged":true}

Verification of the current setting value:

root@elasticsearch:~# curl -q -s "http://localhost:9200/magento2_product_1_v204/_settings" | jq -r '.magento2_product_1_v204.settings.index.number_of_replicas'
0

Yes, it worked!

Now looking at the shards again, we can see there are only two unassigned shards left:

root@elasticsearch:~# curl -q -s "http://localhost:9200/_cat/shards"
magento2_product_1_v204 0 p STARTED    3639 6.2mb 127.0.0.1 elasticsearch
magento2_product_2_v204 0 p STARTED    3639 7.1mb 127.0.0.1 elasticsearch
magento2_product_2_v204 0 r UNASSIGNED                      
magento2_product_3_v204 0 p STARTED    3656 6.2mb 127.0.0.1 elasticsearch
magento2_product_3_v204 0 r UNASSIGNED  

The number_of_replicas setting is now also changed on the two remaining indexes:

root@elasticsearch:~# curl -X PUT -H "Content-Type: application/json" "http://localhost:9200/magento2_product_2_v204/_settings" -d '{ "number_of_replicas": 0 }'
{"acknowledged":true}

root@elasticsearch:~# curl -X PUT -H "Content-Type: application/json" "http://localhost:9200/magento2_product_3_v204/_settings" -d '{ "number_of_replicas": 0 }'
{"acknowledged":true}

And voilà, no more unassigned shards are shown:

root@elasticsearch:~# curl -q -s "http://localhost:9200/_cat/shards"
magento2_product_1_v204 0 p STARTED 3639 6.2mb 127.0.0.1 elasticsearch
magento2_product_2_v204 0 p STARTED 3639 7.1mb 127.0.0.1 elasticsearch
magento2_product_3_v204 0 p STARTED 3656 6.2mb 127.0.0.1 elasticsearch

Elasticsearch Status change

Now with all unassigned shards gone, the Elasticsearch status immediately changed to green:

root@elasticsearch:~# /usr/lib/nagios/plugins/check_es_system.sh -H localhost -t status
ES SYSTEM OK - Elasticsearch Cluster "mycluster" is green (1 nodes, 1 data nodes, 3 shards, 10934 docs)|total_nodes=1;;;; data_nodes=1;;;; total_shards=3;;;; relocating_shards=0;;;; initializing_shards=0;;;; unassigned_shards=0;;;; docs=10934;;;;

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.