Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

If you getting this error - Reallocating unassigned shards might solve the following situations:

  1. An "Error Accessing Store" dialog is displayed after signing in to the system: 
    Image Modified

Or in case you have seen in the Store screen a message that alert about a Red cluster status with unassigned shards,

Image Removed

 you should read and do the following steps:

  1. In the Store page, the status of the cluster is RED with unassigned shards:Image Added



The store is divided into nodes (some used to keep data, others can handle communication and there is one Master that coordinates the work).To know each node's role you can learn from the Node Type marked in redcommunications or manage the cluster). The type of each node can be found in the Nodes table under "Type" column, marked in red in the following screenshot:

Shards need to be allocated/assigned to data nodes so they can in order to be able to be populated with data. If shards are unassigned to a data node then no data can flow to the system and this the problem we want to solve.

This situation is usually happens when there is an caused by unexpected recovery of the cluster and at the time of recovery , where automatic assignment of shards was not applicable at the time of recovery for unknown reason.

What can you do ?



There is a script at To fix the issue, you may execute the following script: /app/scripts/realloc_shards.sh <Data Node 1> <Data Node 2> ...

You only need to rake the number at the end of the nodes name for Example es-raw-trans-Node-3 means data node number is 3

The scripts gets a space separated list of data nodes (Those with Type = D) and scan all shards and reallocate unassigned shards to available data node from the list<Node Name>

  • The <Node Name> parameter should be the name of the data node (a node of type "D" in the nodes table)

The script scans all shards and reallocates unassigned shards to the data node provided.

In the example above we , the script should be run commandas follows:

Code Block
/app/scripts/realloc_shards.sh 2

 

It might take a while but when you refresh the screen you will see the number of unassigned is decrease and the number  of assigned is increased until all are assigned.

...

MonTier-es-raw-trans-Node-2

This operation may take a while, depending on your installation and the amount of data. Refreshing the screen in "head" application (available from Store page) will show up-to-date state of the assigned and unassigned shards.

Warning

In cell environments, unassigned shards can only reallocate to a Store node from the same cell member or cell manager.
i.e. if there are unassigned shards from index of node N002, they can be reallocated to Store node named MonTier-es-raw-trans-N002-Node-X.


Below is an example of a successful execution:

Code Block
Reallocation node: MonTier-es-raw-trans-Node-2
Testing index 0;wdp-wsm_i2
About to reallocate. index=wdp-wsm_i2, shard=0, node=MonTier-es-raw-trans-Node-2
Reallocation http response status is: 200, output={"acknowledged":true,"state":{"version":
Testing index 1;wdp-syslog-sys-auth_i1
About to reallocate. index=wdp-syslog-sys-auth_i1, shard=1, node=MonTier-es-raw-trans-Node-2
Reallocation http response status is: 200, output={"acknowledged":true,"state":{"version":
Testing index 2;system-health_i1
About to reallocate. index=system-health_i1, shard=2, node=MonTier-es-raw-trans-Node-2
Reallocation http response status is: 200, output={"acknowledged":true,"state":{"version":
Testing index 0;system-health_i1
About to reallocate. index=system-health_i1, shard=0, node=MonTier-es-raw-trans-Node-2
Reallocation http response status is: 200, output={"acknowledged":true,"state":{"version":
...


Below is an example of a unsuccessful execution:

Code Block
Reallocation node: MonTier-es-raw-trans-Node-5
Testing index 0;wdp-syslog-sys-auth_i6
About to reallocate. index=wdp-syslog-sys-auth_i6, shard=0, node=MonTier-es-raw-trans-Node-5
Reallocation http response status is: 400, output={"error":{"root_cause":[{"type":"remote_
Testing index 0;wdp-syslog-sys-auth_i3
About to reallocate. index=wdp-syslog-sys-auth_i3, shard=0, node=MonTier-es-raw-trans-Node-5
Reallocation http response status is: 400, output={"error":{"root_cause":[{"type":"remote_
Testing index 1;logical-tran-full_i1
About to reallocate. index=logical-tran-full_i1, shard=1, node=MonTier-es-raw-trans-Node-5
Reallocation http response status is: 400, output={"error":{"root_cause":[{"type":"remote_
Testing index 1;wdp-config-service_i1
About to reallocate. index=wdp-config-service_i1, shard=1, node=MonTier-es-raw-trans-Node-5
Reallocation http response status is: 400, output={"error":{"root_cause":[{"type":"remote_
...