Versions Compared

Old Version 10

changes.mady.by.user Avi Falah

Saved on

compared with

New Version Current

changes.mady.by.user Amit Munwes

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Symptoms

  • An "Error Accessing Store"

...

Image Removed

Or in case you have seen in the Store screen a message that alert about a Red cluster status with unassigned shards,

Image Removed

 you should read and do the following steps:

...

  • dialog is displayed after signing in to the system:

    Image Added

  • In the Store page, the status of the cluster is RED with unassigned shards:

    Image Added

Cause

This situation is usually caused by an unexpected recovery of the Store, where automatic assignment of shards was not applicable at the time of recovery for some reason. For example, this might be caused by an I/O error of the storage device or emergency reboot of the machine.

The Store is divided into nodes, some used to keep data, others can handle communication and there is one Master that coordinates the work).

To know each node's role you can learn from the Node Type marked in red

Image Removed

Shards need to communications or manage the Store. Shards must be allocated/assigned to Store data nodes so they can in order to be able to be populated with data. If some shards are unassigned to a data node, then no data can flow to be persisted in the system and this the problem we want to solve.

This situation is usually happens when there is an unexpected recovery of the cluster and at the time of recovery automatic assignment of shards was not applicable for unknown reason.

What can you do ?

There is a script at /app/scripts/realloc_shards.sh <Node Name>

The scripts gets data node name (of with Type = D) and scan all shards and reallocate .

Resolution

Use Reallocating Unassigned Shards utility in Admin Console. The utility scans all shards and reallocates unassigned shards to the available data node provided.

In the example above we should run command

...

...

/app/scripts/realloc_shards.sh MonTie-es-raw-trans-Node-2

This operation may take a while depends , depending on your installation and the amount of data. Refreshing the screen in "head" application available from Store dashboard will show updated state of the assigned and unassigned shards.

Below is example of successfull execution:

...

Warning

In cell environments, unassigned shards can only be reallocated to a Store node from the same cell member or cell manager.
i.e. if there are unassigned shards from index of node N002, they can be reallocated to a Store node named MonTier-es-raw-trans-

Node-2 Reallocation return code is : 0, output={"acknowledged":true,"state":{"version":886,"state Testing index 1;internal-operations_i6 About to reallocate. index=internal-operations_i6, shard=1, node=MonTier-es-raw-trans-Node-2 Reallocation return code is : 0, output={"acknowledged":true,"state":{"version":888,"state Testing index 1;logical-tran-compact_i16 About to reallocate. index=logical-tran-compact_i16, shard=1, node=MonTier-es-raw-trans-Node-2 ..

N002-Node-X.