Troubleshooting Oracle Database@AWS: The Journey Through TDE Wallet Failures, Media Manager Decryption Errors, and Incomplete Console Terminations

 A real-world troubleshooting log detailing root-cause investigations, Oracle Support escalations, and clean-up workarounds on Oracle Database@AWS.

Migrating and managing Oracle workloads on Oracle Database@AWS (OCI Exadata Database Service on AWS) brings native cloud elasticity to enterprise-grade database hardware. However, when working with cross-cloud automation, Transparent Data Encryption (TDE), and managed backup workflows, hitting automation edge cases is part of the journey.

Recently, our team worked through a complex database restore scenario via the "Create Database from Backup" feature. Along the way, we navigated multiple failure points across TDE wallet handling, OCI/AWS object storage media managers, and metadata synchronization between the web console and the OS layer.

Here is the step-by-step breakdown of the errors encountered, the mitigations applied, and key takeaways for cloud DBAs.

Failure 1: The Null TDE Wallet Password Bug

Our initial attempt was standard: initiate a fresh database provisioning flow directly from a backup via the management console.

The job immediately failed at the orchestration layer:

Error: The password of TDE wallet for backup cannot be null

Analysis & Resolution

Even though all encryption credentials and wallet parameters were specified during configuration, the API payload dropped or failed to parse the TDE wallet secret down to the deployment automation scripts.

  • Action Taken: Opened an urgent Service Request (SR) with Oracle Support.

  • Outcome: The Oracle development team identified a backend validation issue and patched the console workflow, instructing us to retry the restore.

Failure 2: RMAN / Media Manager Decryption Failure (KBHS-01106)

With the console-level TDE bug patched, the orchestration moved past the initial validation stage. Database Home creation succeeded, datafiles were mapped to ASM disk groups (+DATAC2), and RMAN channels initiated data piece allocation.

However, mid-restore, RMAN crashed with an SBT/Media Manager error:

KBHS-01106: unable to decrypt backup



Analysis

The KBHS-01106 error points directly to the Oracle Cloud / AWS SBT library (libopc.so / libobk.so media management layer). While the RMAN layer had access to the backup pieces, the underlying key supplied to the media management interface could not decrypt the object storage backup pieces.

While Oracle development investigates the key mismatch on the backend, we pivoted to an alternative strategy: creating a fresh, known-state Level 0 backup via dbaascli on the source database to eliminate any potential legacy wallet desyncs.

Failure 3: The DTRS Encryption Key Crash (DBAAS-10612)

We attempted the restore again using the new L0 backup. The job failed during the post-provisioning encryption configuration:

[FATAL] [DBAAS-70425] Unable to restore database from backup. 

CAUSE: Response from DTRS: DBAAS-10612 : Failed to set encryption primary key for the database with unique name 'QRESTORE'.. 

ACTION: Refer to log /var/opt/oracle/log/QRESTORE/database/createFromBackup/pilot_2026-08-18_05-50-26-AM_136691 for more information

[FATAL] [DBAAS-70425] Unable to restore database from backup. 

CAUSE: Response from DTRS: DBAAS-10612 : Failed to set encryption primary key 

for the database with unique name 'QRESTORE'.. 

ACTION: Refer to log /var/opt/oracle/log/QRESTORE/database/createFromBackup/pilot_2026-08-18_05-50-26-AM_136691 for more information


At this point, we had a broken target environment with a half-configured database home and residual cluster metadata.

The Pivot: The "Ghost Database" Console Desync

To retry cleanly, we terminated the failed target instance (QRESTORE) directly from the AWS/OCI console. The console reported:

Database terminated successfully.

To ensure no corrupted environment variables or wallet references remained, we attempted to delete the associated ORACLE_HOME from the console. That request immediately failed:




[FATAL] [DBAAS-70356] Operation not supported in a home with databases configured. 

Cause: Oracle home '/u02/app/oracle/product/19.0.0.0/dbhome_2' contains these databases: 'QRESTORE,' 

Action: Verify the requested home and try again. 

(WorkRequestId: 7943f968-cc44cc-4140-XXXX-1987e031094f)

The Root Cause: Incomplete Console Termination

The cloud control plane issued a successful database termination status, but the underlying nodes in the RAC cluster never completed the teardown at the OS/CRS layer.

Upon logging into both database nodes via SSH:

  1. The QRESTORE RAC instance was still running on both nodes.

  2. Oracle Clusterware (Grid Infrastructure) still held the resource registrations for QRESTORE inside dbhome_2.

The Fix: Manual Server-Side Cleanup

Because the management agent will prevent DB Home deletion if Grid Infrastructure reports registered databases, we performed a manual cleanup directly on the cluster nodes:

Step 1: Stop Instances on All RAC Nodes

# Verify instance status across nodes
srvctl status database -d QRESTORE 

  # Force stop the database cluster-wide 
srvctl stop database -d QRESTORE -o immediate

Step 2: Unregister the Database from Clusterware

# Force unregister the orphaned database resource from CRS
srvctl remove database -d QRESTORE -f

Step 3: Delete the Database Home from the Console

Once the CRS configuration was wiped clean, the console-level dependency check passed. We triggered the Delete DB Home operation from the console, and it completed successfully.

Key Lessons for Cloud DBAs

  1. Console Status != Cluster Reality on Hard Failures: When complex orchestration fails mid-step (especially during encryption provisioning), trust the control plane UI, but verify with srvctl and crsctl on the nodes before retrying.

  2. KBHS-01106 is Almost Always Key/Context Mismatch: Whenever the Media Management Layer throws KBHS-01106: unable to decrypt backup, verify whether the backup encryption password, target keystore password, or AWS KMS/OCI Vault integration payload has experienced a key rollover or metadata mismatch.

  3. Clean-Slate Retries Save Time: When troubleshooting corrupted deployment hooks, deleting down to the DB Home and rebuilding ensures that stale wallet auto-login states (cwallet.sso) or partial CRS registrations do not interfere with subsequent restore attempts.

We are currently running our next restore validation pass - stay tuned for part two on the root-cause resolution for the KBHS-01106 decryption patch.


Comments

Post a Comment

Popular posts from this blog

ORA-65139: Mismatch between XML metadata file and data file

Error: CDB$ROOT@EBTUPRDC> CREATE PLUGGABLE DATABASE EBTUSPRDPDB1 using '/home/oracle/EBTUSPRDC.xml' nocopy tempfile reuse; CREATE PLUGGABLE DATABASE EBTUSPRDPDB1 using '/home/oracle/EBTUSPRDC.xml' nocopy tempfile reuse * ERROR at line 1: ORA-65139: Mismatch between XML metadata file and data file /u05/odb/ORADATA_1/EBTUSPRDC/datafile/system.1531.807800993 for value of fcpsb (2230502002 in the plug XML file, 2230505492 in the data file) Fix: Don't open Non-CDB database until we complete pdb creation other wise we will get above error. Shutdown immediate; startup mount; alter database open read only; create xml file. shutdown immediate; Connect to CDB and Create Pluggable database .
Read more

Useful OEM Queries to get Target details from OEM Repository

List Targets with TNS Listener ports configured : SELECT mgmt$target.host_name , mgmt$target.target_name , mgmt$target.target_type , mgmt$target_properties.property_name , mgmt$target_properties.property_value FROM mgmt$target , mgmt$target_properties WHERE ( mgmt$target.target_name = mgmt$target_properties.target_name ) AND ( mgmt$target.target_type = mgmt$target_properties.target_type ) and ( mgmt$target.target_type = 'oracle_listener' ) and ( mgmt$target_properties.property_name = 'Port' ); Devora02       LISTENER_ora02                       oracle_listener Port 1529 Devora01       LISTENER_ora01                       oracle_listener Port 1529 Devora04       LISTENE...
Read more

TFA-00002 : Oracle Trace File Analyzer (TFA) is not running

TFA Failed to start listening for commands on one of the Rac Node : As a root user /etc/init.d/init.tfa  start Starting TFA.. start: Job is already running: oracle-tfa Waiting up to 100 seconds for TFA to be started.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Successfully started TFA Process.. . . . . . TFA-00002 : Oracle Trace File Analyzer (TFA) is not running TFA Failed to start listening for commands Solution : run synctfanodes.sh  if you have root password . if no root password copy below files to TFA Failed node and try to restart TFA .. Please check whether these 4 files exist on all nodes are similar. TFA_HOME/server.jks TFA_HOME/client.jks TFA_HOME/internal/ssl.properties   TFA_HOME/internal/ portmapping.txt If not similar just copy these files to TFA Failing node and try TFA Restart ..It Worked for me .   [root@host01]# ./tfactl  -v "debug" start Starting TFA...
Read more