Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Symptoms

  • An "Error Accessing Store" dialog is displayed after signing in to the system:

...

...

  • Image Added

  • In the Store page, the status of the cluster is RED with unassigned shards:

...

  • Image Added

Cause

This situation is usually caused by an unexpected recovery of the Store, where automatic assignment of shards was not applicable at the time of recovery for some reason. For example, this might be caused by an I/O error of the storage device or emergency reboot of the machine.

The Store is divided into nodes (, some used to keep data, others can handle communications or manage the cluster). The type of each node can be found in the Nodes table under "Type" column, marked in red in the following screenshot:

Image Removed

Shards need to Store. Shards must be allocated/assigned to Store data nodes in order to be able to be populated with data. If some shards are unassigned to a data node, then no data can flow to be persisted in the system.

This situation is usually caused by unexpected recovery of the cluster, where automatic assignment of shards was not applicable at the time of recovery for unknown reason.

To fix the issue, you may execute the following script: /app/scripts/realloc_shards.sh <Node Name>

  • The <Node Name> parameter should be the name of the data node (a node of type "D" in the nodes table)

...

Resolution

Use Reallocating Unassigned Shards utility in Admin Console. The utility scans all shards and reallocates unassigned shards to the data node provided.

In the example above, the script should be run as follows:

Code Block
/app/scripts/realloc_shards.sh MonTie-es-raw-trans-Node-2

This operation may take a while, depending on your installation and the amount of data. Refreshing the screen in "head" application (available from Store page) will show up-to-date state of the assigned and unassigned shards.

Below is an example of a successful execution:

...

Warning

In cell environments, unassigned shards can only be reallocated to a Store node from the same cell member or cell manager.
i.e. if there are unassigned shards from index of node N002, they can be reallocated to a Store node named MonTier-es-raw-trans-

Node-2 Reallocation http response status is: 200, output={"acknowledged":true,"state":{"version": Testing index 0;system-health_i1 About to reallocate. index=system-health_i1, shard=0, node=MonTier-es-raw-trans-Node-2 Reallocation http response status is: 200, output={"acknowledged":true,"state":{"version": ..

N002-Node-X.