IBM Storwize V7000 Unified Problem Determination Manual page 409

Attention:
v Run service actions only when directed by the fix procedures. If used
inappropriately, service actions can cause loss of access to data or even data loss.
Before you attempt to recover a storage system, investigate the cause of the
failure and attempt to resolve those issues by using other fix procedures. Read
and understand all of the instructions before you complete any action.
v The recovery procedure can take several hours if the system uses large-capacity
devices as quorum devices.
Do not attempt the recover system procedure unless the following conditions are
met:
v All of the conditions have been met in "When to run the recover system
procedure" on page 382.
v All hardware errors are fixed. See "Fix hardware errors" on page 382
v All node canisters have candidate status. Otherwise, see step 1.
v All node canisters must be at the same level of code that the storage system had
before the system failure. If any node canisters were modified or replaced, use
the service assistant to verify the levels of code, and where necessary, to reinstall
the level of code so that it matches the level that is running on the other node
canisters in the system.
v If the system is encrypted, insert the USB flash drive that contains the
encryption key file into the node canister that will be running system recovery.
v System recovery will fail if the system has encryption enabled and a USB flash
device containing the key is not found. Insert the USB flash drive that contains
the encryption key file into the node canister that is running the system
recovery, and then retry system recovery.
The system recovery procedure is one of several tasks that must be completed. The
following list is an overview of the tasks and the order in which they must be
completed:
1. Preparing for system recovery
a. Review the information regarding when to run the recover system
procedure.
b. Fix your hardware errors and make sure that all nodes in the system are
shown in service assistant or in the output from sainfo lsservicenodes.
c. Remove the system information for node canisters with error code 550 or
error code 578 by using the service assistant, but only if the recommended
user response for these node errors has already been followed. See
"Removing system information for node canisters with error code 550 or
error code 578 using the service assistant" on page 384.
d. For Virtual Volumes (VVols), shut down the services for any instances of
Spectrum Control Base that are connecting to the system. Use the Spectrum
Control Base command service ibm_spectrum_control stop.
2. Running the system recovery. After you prepared the system for recovery and
met all the pre-conditions, run the system recovery.
Note: Run the procedure on one system in a fabric at a time. Do not run the
procedure on different node canisters in the same system. This restriction also
applies to remote systems.
3. Completing actions to get your environment operational.
v Recovering from offline volumes by using the CLI.
Chapter 5. Control enclosure
381