...
Symptoms
- An "Error Accessing Store" dialog is displayed after signing in to the system:
...
...
- In the Store page, the status of the cluster is RED with unassigned shards:
...
Cause
This situation is usually caused by an unexpected recovery of the Store, where automatic assignment of shards was not applicable at the time of recovery for some reason. For example, this might be caused by an I/O error of the storage device or emergency reboot of the machine.
The Store is divided into nodes (, some used to keep data, others can handle communications or manage the cluster). The type of each node can be found in the Nodes table under "Type" column, marked in red in the following screenshot:
Shards need to Store. Shards must be allocated/assigned to Store data nodes in order to be able to be populated with data. If some shards are unassigned to a data node, then no data can flow to be persisted in the system.
This situation is usually caused by unexpected recovery of the cluster, where automatic assignment of shards was not applicable at the time of recovery for unknown reason.
To fix the issue, you may execute the following script: /app/scripts/realloc_shards.sh <Node Name>
- The <Node Name> parameter should be the name of the data node (a node of type "D" in the nodes table)
...
Resolution
Use Reallocating Unassigned Shards utility in Admin Console. The utility scans all shards and reallocates unassigned shards to the data node provided.
In the example above, the script should be run as follows:
...
This operation may take a while, depending on your installation and the amount of data. Refreshing the screen in "head" application (available from Store page) will show up-to-date state of the assigned and unassigned shards.
Below is an example of a successful execution:
| Code Block |
|---|
Reallocation node: MonTier-es-raw-trans-Node-2
Testing index 0;wdp-wsm_i2
About to reallocate. index=wdp-wsm_i2, shard=0, node=MonTier-es-raw-trans-Node-2
Reallocation http response status is: 200, output={"acknowledged":true,"state":{"version":
Testing index 1;wdp-syslog-sys-auth_i1
About to reallocate. index=wdp-syslog-sys-auth_i1, shard=1, node=MonTier-es-raw-trans-Node-2
Reallocation http response status is: 200, output={"acknowledged":true,"state":{"version":
Testing index 2;system-health_i1
About to reallocate. index=system-health_i1, shard=2, node=MonTier-es-raw-trans-Node-2
Reallocation http response status is: 200, output={"acknowledged":true,"state":{"version":
Testing index 0;system-health_i1
About to reallocate. index=system-health_i1, shard=0, node=MonTier-es-raw-trans-Node-2
Reallocation http response status is: 200, output={"acknowledged":true,"state":{"version":
... |
Below is an example of a unsuccessful execution:
...
.
| Warning |
|---|
In cell environments, unassigned shards can only be reallocated to a Store node from the same cell member or cell manager. |