Fix db_stress rollback retry exhaustion under concurrent Resume Busy (#14877)

Summary:
Pull Request resolved: https://github.com/facebook/rocksdb/pull/14877

The rollback-after-failed-commit retry loop in `db_stress_test_base.cc` (added in b0b52e1d14f3) counted every loop iteration — including iterations where `Resume()` returned Busy — against a single 100-count limit. With 32 concurrent threads, when write fault injection causes all prepared transaction commits and rollbacks to fail simultaneously, `Resume()` returns Busy (recovery in progress from another thread) for the entire 100-iteration budget (100 × 10ms = 1 second). Recovery itself keeps failing due to ongoing fault injection causing repeated background flush errors, so no thread ever gets a chance to perform an actual rollback.

The fix separates two budgets:
- `kMaxRollbackRetries` (100): counts only actual rollback attempts after `Resume()` returns OK
- `kMaxResumeBusyWaits` (3000): allows up to 30 seconds of busy-waiting for recovery to complete on another thread

Key triggering options: `write_fault_one_in=1000` + `txn_write_policy=1` (write-prepared) + `two_write_queues=1` + 32 threads + `error_recovery_with_no_fault_injection=1`.

Reviewed By: joshkang97

Differential Revision: D108561314

fbshipit-source-id: 03f4578ea33289685ad982208451f0584bf0535f

db_stress_tool/db_stress_test_base.cc

@@ -1219,13 +1219,18 @@ Status StressTest::CommitTxn(Transaction& txn, ThreadState* thread) {
       std::string last_resume_status;
       if (!rollback_s.ok() && IsErrorInjectedAndRetryable(rollback_s) &&
           db_ != nullptr) {
-        constexpr int kMaxRollbackAfterRecoveryRetries = 100;
-        constexpr int kRollbackAfterRecoveryRetryIntervalMicros = 10 * 1000;
-        for (; rollback_recovery_retries < kMaxRollbackAfterRecoveryRetries &&
-               !rollback_s.ok();
-             ++rollback_recovery_retries) {
+        // Retry budget: count only actual rollback attempts (after Resume
+        // succeeds), not time spent waiting for recovery to finish on another
+        // thread. With many concurrent threads hitting write errors, recovery
+        // can take much longer than individual threads' retry budgets.
+        constexpr int kMaxRollbackRetries = 100;
+        constexpr int kMaxResumeBusyWaits = 3000;
+        constexpr int kRetryIntervalMicros = 10 * 1000;
+        for (; rollback_recovery_retries < kMaxRollbackRetries &&
+               resume_busy_count < kMaxResumeBusyWaits && !rollback_s.ok();) {
           const Status resume_s = db_->Resume();
           if (resume_s.ok()) {
+            ++rollback_recovery_retries;
             last_resume_status = resume_s.ToString();
             rollback_s = txn.Rollback();
             if (rollback_s.ok() || !IsErrorInjectedAndRetryable(rollback_s)) {
@@ -1238,8 +1243,7 @@ Status StressTest::CommitTxn(Transaction& txn, ThreadState* thread) {
             ++resume_busy_count;
             last_resume_status = resume_s.ToString();
           }
-          clock_->SleepForMicroseconds(
-              kRollbackAfterRecoveryRetryIntervalMicros);
+          clock_->SleepForMicroseconds(kRetryIntervalMicros);
         }
       }
       if (!rollback_s.ok()) {