ORA-19914: the backup fails, and the wallet is open

ab_kot

Sep 14

5 min read

An RMAN backup on Oracle Database Cloud Service failed with ORA-28365: wallet is not open. The wallet was open. Here is what that error actually means, why a restore is what puts you there, and what to check before you hand a restored instance back.

What it looked like

Mid-migration: a Customer-to-Meter application moving to OCI, WebLogic on compute, the database on Oracle Database Cloud Service. The application administrator wanted a preventive backup before starting a data upgrade — the ordinary, careful thing to do before an irreversible step.

The backup, kicked off from the console, came back with this:

Error: DCS-10429: Request failed with RMAN Errors:
RMAN-03002: failure of backup command at 09/15/2023 15:54:29
Error Stack: [RMAN-03009: failure of backup command on ORA_SBT_TAPE_2 channel
at 09/15/2023 13:58:22, ORA-19914: unable to encrypt backup, ORA-28365

The console truncates the stack at the point where it stops being a console's problem. DCS-10429 is the DBCS control-plane wrapper; everything useful is underneath it, and ORA-28365 arrives with no message text at all.

Where the real error lives

On a DBCS instance, the RMAN log is on the database node, not in the console:

/opt/oracle/dcs/log/<hostname>/rman/bkup/<ORACLE_UNIQUE_NAME>/

That is the first thing worth knowing, because the console error will not tell you which of the two ORA- codes matters. The log did (RMAN's banner rows trimmed):

channel ORA_SBT_TAPE_1: backup set complete, elapsed time: 02:02:05
RMAN-00569: ERROR MESSAGE STACK FOLLOWS
RMAN-03002: failure of backup command at 09/15/2023 15:54:29

RMAN-03009: failure of backup command on ORA_SBT_TAPE_2 channel
           at 09/15/2023 13:58:23
ORA-19914: unable to encrypt backup
ORA-28365: wallet is not open
RMAN-03009: failure of backup command on ORA_SBT_TAPE_2 channel
           at 09/15/2023 13:58:22
ORA-19914: unable to encrypt backup
ORA-28365: wallet is not open

Two details in that log are worth pausing on. Channel 1 completed — two hours of backup set, written fine. Only channel 2 failed. And the failure timestamps (13:58) are nearly two hours before the command failure timestamp (15:54): the channel failed early and RMAN carried on until the rest of the work finished before reporting.

“Wallet is not open” when the wallet is open

This is the part that costs people an afternoon. ORA-28365 reads as a binary state — the wallet is open, or it is not. It is not that simple, because RMAN needs a particular kind of open.

Oracle's Backup and Recovery documentation puts it plainly:

When you use the auto-login software keystore, encrypted backup operations can be performed at any time, because the auto-login keystore is always open. When you use the password-based software keystore, the keystore must be opened before you can perform backup encryption.

A password-protected keystore (ewallet.p12) opens when someone supplies the password, and that open state belongs to the instance until it closes. An auto-login keystore (cwallet.sso) opens by itself, for every process that needs it, without anyone present.

Backup automation has nobody present. DBCS backups are driven by the control plane on a schedule, with no human to type a keystore password — so on those instances, transparent encryption of the backup effectively depends on an auto-login keystore being in place. A keystore that is open in PASSWORD mode satisfies your session and not the backup.

A five-stage timeline of the TDE keystore state through a migration. Stage 1, restore datafiles, keystore is AUTOLOGIN. Stage 2, encrypting the restored files opens and closes the keystore with the password, leaving it PASSWORD. Stages 3 and 4, the instance is handed over and the scheduled backup starts, keystore still PASSWORD. Stage 5, the backup channel fails with ORA-28365. Below, two lanes contrast what you check, STATUS equals OPEN, which is true throughout and proves nothing, against what actually decides it, WALLET_TYPE equals PASSWORD, where unattended backup encryption needs AUTOLOGIN.

STATUS reads OPEN at every stage, including the one where the backup fails. WALLET_TYPE is the column that decides whether an unattended backup can encrypt.

Confirming it

One query settles it:

SELECT wrl_type, wrl_parameter, status, wallet_type
FROM   v$encryption_wallet;

STATUS was OPEN. WALLET_TYPE was PASSWORD — which is the actual finding. The wallet was open; it was open the wrong way. Oracle Support Note 2333646.1 covers the same symptom (a My Oracle Support login is needed to read it).

Two columns, and only one of them was telling the truth about whether a scheduled backup would work.

The fix

Four steps, in order:

1. Find the wallet. WRL_PARAMETER from the query above gives the keystore location.

2. Back up the existing cwallet.sso in that directory before you touch anything. You are about to overwrite it.

3. Create the auto-login keystore from the password-protected one, as SYSDBA:

ADMINISTER KEY MANAGEMENT CREATE AUTO_LOGIN KEYSTORE
  FROM KEYSTORE '<keystore_location>'
  IDENTIFIED BY "<wallet_password>";

Use CREATE LOCAL AUTO_LOGIN KEYSTORE only if you want the keystore bound to the host it was created on. Oracle's guidance is to omit LOCAL on Oracle RAC, because each node has a different hostname and a host-bound keystore in a shared location locks out every other node.

4. Restart the database. Oracle does not document a restart as a requirement for creating the auto-login keystore — but the password-protected keystore was already the open one, and a restart is the unambiguous way to hand over to the auto-login keystore on a DBCS instance. After the restart, WALLET_TYPE read AUTOLOGIN, and the backup ran.

One caution from Oracle's own documentation, worth repeating because the fix makes it newly relevant:

If you use an auto-login keystore, do not back it up along with your encrypted backup data, because users can read the encrypted backups if they obtain both the backups and the autologin keystore.

The cwallet.sso you just created is a key that opens itself. Where it lives, and who can read that directory, is now part of your backup's security posture.

Why it happened

Nobody misconfigured anything. The migration involved an RMAN restore, and restored datafiles have to be encrypted afterwards. That work opens and closes the TDE wallet repeatedly, sometimes with the password, and the last open wins. The instance was left with a keystore open in PASSWORD mode — perfectly functional for everything a human does interactively, and invisible until the first unattended operation needed it.

Which is the general shape of this class of failure: the migration succeeded, the database worked, every interactive check passed, and the thing that broke was the first operation with nobody watching. The preventive backup was, in fact, the first honest test of the instance.

What to check before handing back a restored instance

Restore work leaves state behind. On any instance where TDE and automated backups both apply:

SELECT status, wallet_type FROM v$encryption_wallet; — OPEN is not the answer, AUTOLOGIN is.
Confirm cwallet.sso exists in the keystore location and is readable by the database owner.
Run one backup manually, end to end, before you declare the instance handed over. Not a test of the backup job — a test of everything the restore touched.
Check the RMAN log rather than the console result, at least once, so you know where it is before you need it at 2am.