If you manage Oracle Exadata environments, you rely on ExaWatcher to quietly collect performance diagnostic data in the background. Because ExaWatcher is standardized across nodes, Database Administrators (DBAs) and Exadata Direct Engineers (DMAs) generally expect archive retention to look uniform across identically configured environments.
However, a common point of confusion arises during capacity audits: Why does one Exadata environment purge files older than 30 days and stay around 1 GB, while another retains months of historical archives up to 2.4 GB - despite having the exact same ExaWatcher configuration?
In this post, we break down a real-world troubleshooting case across Development (DEV), Integration (TESTINT), and Quality Assurance (QA) Exadata database nodes to uncover how ExaWatcher actually governs archive retention, the hidden filesystem triggers inside ExaWatcherCleanup.sh, and how to safely manage historical archives.
The Mystery: Identical Configurations, Different Retention
During a routine review of ExaWatcher archive directories (/opt/oracle.ExaWatcher/archive) across our Exadata Database nodes, we noticed a sharp contrast in retention behavior:
| Environment | Total Archive Size | Files Older Than 30 Days | Size Older Than 30 Days | Oldest Archive Retained |
| DEV | ~989 MB – 1.0 GB | 0 | 0 MB | 27-Sep-2026 |
| TESTINT | ~2.4 GB | 29,677 | ~1.2 GB | 12-Jul-2026 |
| QA | ~2.3 GB | 39,147 | ~1.2 GB | 12-Mar-2026 |
At first glance, this looked like a broken purge job on TESTINT and QA. While DEV was cleanly rotating out anything older than 30 days, QA was holding onto nearly 40,000 files dating back seven months.
Verifying the Baseline Configuration
To rule out configuration drift, we audited the ExaWatcher configuration files and daily scheduler settings across all three environments. Every single node had the exact same parameters:
Furthermore, checking /opt/oracle.ExaWatcher/log/OldFilesDeleted.log confirmed that the daily cleanup script was actively running and deleting files in all environments.
When we checked individual collector subdirectories using du -sh /opt/oracle.ExaWatcher/archive/*, we found that the primary space consumers were standard system collectors:
Ps.ExaWatcherDiskinfo.ExaWatcherTop.ExaWatcher&TopPid.ExaWatcherNetstat.ExaWatcherIostat.ExaWatcherLsof.ExaWatcher
Interestingly, QA also held roughly 2,000 legacy files each under Bsw.ExaWatcher, Edv.ExaWatcher, Egs.ExaWatcher, and Esnp.ExaWatcher, whereas DEV had zero files in those directories.
How ExaWatcher Retention Actually Works
To understand why DEV was purging aggressively while TESTINT and QA held onto months of data, we have to dispel a common myth: ExaWatcher does not have a time-based (age-based) retention policy.
There is no "30-day rule" coded into ExaWatcher. Instead, cleanup decisions are driven purely by storage capacity and quota thresholds managed inside /opt/oracle.ExaWatcher/ExaWatcherCleanup.sh.
Specifically, ExaWatcher cleanup is governed by three distinct mechanisms:
1. The Configured <SpaceLimit>
By default, ExaWatcher enforces a <SpaceLimit> of 3 GB (3052 MB) on Exadata Database servers and 600 MB on Exadata Storage Cells. As long as the total size of /opt/oracle.ExaWatcher/archive remains below 3052 MB (and filesystem space is healthy), ExaWatcher has no reason to purge older files—which explains why TESTINT (2.4 GB) and QA (2.3 GB) happily kept files from July and March.
2. The 80% Filesystem Utilization Trigger (Bug 38130263)
Even if your archive directory is well below the 3052 MB <SpaceLimit>, ExaWatcher monitors the underlying filesystem (/opt/oracle.ExaWatcher).
Under logic introduced via Bug 38130263, ExaWatcherCleanup.sh checks overall filesystem utilization (df -h /opt/oracle.ExaWatcher). When filesystem usage reaches 80% (meaning free space drops to 20% or less), ExaWatcher overrides the standard <SpaceLimit> and dynamically reduces its footprint to ensure the filesystem maintains at least 20% free space.
3. Per-Directory Quota Enforcement
Within the archive structure, ExaWatcher enforces sub-quotas across individual collector directories. If high-volume collectors (like Ps or Lsof) churn rapidly, they rotate their own oldest files once their slice of the quota fills up, while low-volume or inactive collectors (like Bsw or Edv in QA) sit untouched until an overall space threshold forces a global reduction.
Cracking the Case: The Filesystem Factor
Once we understood the 80% filesystem rule inside ExaWatcherCleanup.sh, we ran df -h /opt/oracle.ExaWatcher across our three environments and placed the numbers side-by-side:
| Environment | Filesystem Usage | Current Archive Size | Configured <SpaceLimit> |
| DEV | 79% | 1023 MB (~1.0 GB) | 3052 MB |
| TESTINT | 75% | 2.4 GB | 3052 MB |
| QA | 76% | 2.3 GB | 3052 MB |
Here was the smoking gun:
In
DEV, other non-ExaWatcher files on the mount (or a smaller overall filesystem partition) had pushed filesystem utilization right up against the 80% threshold (sitting at 79%). WheneverDEVcrept hit 80%,ExaWatcherCleanup.shstepped in and aggressively pruned the archive down to ~1 GB to preserve the mandatory 20% free space buffer on the mount. Because of that tight 1 GB ceiling,DEVcould only hold about 30 days' worth of logs.In
TESTINTandQA, the filesystem utilization sat comfortably at 75% and 76%. Because they hadn't breached the 80% filesystem ceiling, ExaWatcher allowed the archives to grow toward the configured 3052 MB<SpaceLimit>, retaining historical files from March and July in the process.
Both behaviors are 100% expected and working by design
Frequently Asked Questions (FAQ) & Best Practices
If you are auditing ExaWatcher on your own Exadata racks, here is a quick reference guide based on Oracle Support guidance (see ExaWatcher Utility On Exadata Compute and Storage Nodes — KB478226):
Q1: Is it normal to see 2.3–2.4 GB of archives and files older than 30 days?
Yes. Retention is strictly space-driven, not time-driven. As long as the archive is under the <SpaceLimit> (default 3052 MB on DB nodes) and the filesystem is under 80% utilization, ExaWatcher will retain historical files indefinitely until the space quota is needed by newer collections.
Q2: Can historical files under Bsw, Edv, Egs, and Esnp—or any files older than 30 days—be safely removed?
Yes. Historical archive files older than 30 days (including legacy collector directories) are purely diagnostic logs and can be safely removed if you want to reclaim space, though leaving them for ExaWatcher to manage automatically is the standard best practice.
Q3: How can I perform a supported, one-time manual purge of ExaWatcher archives?
Rather than running raw rm commands inside the archive subfolders, you can use ExaWatcher.sh with the --spacelimit flag to trigger a supported cleanup down to a specific size quota (for example, shrinking the archive down to 2 GB / 2048 MB):
Stop ExaWatcher:
cd /opt/oracle.ExaWatcher
./StopExaWatcher.sh
Run a one-time cleanup with a temporary space limit:
This invokes ExaWatcherCleanup.sh in the foreground and purges the oldest logs across directories until the total archive size drops below 2048 MB. Wait until you see console output similar to:
Once that message appears, press CTRL+C to exit the script.
Restart ExaWatcher normally:
Comments
Post a Comment