Introduction to PolarDB Multi-Master Write Node Switching

In a PolarDB for MySQL Multi-Master (Limitless) cluster, multiple read-write (RW) nodes share a common underlying storage layer. To prevent write conflicts and ensure data consistency, PolarDB employs database-level (or object-level) write ownership. Only one RW node owns write privileges for a specific database at any given time.

When you perform load balancing or maintenance using the following SQL statement:

ALTER DATABASE db1 POLARDB_WRITE_NODE 2;

PolarDB coordinates a handoff of this write ownership from RW1 to RW2. This article details the exact transaction boundary, linearization guarantees, and behavior of active, pending, and new transactions during this switchover process.

The Switchover Workflow: Step-by-Step

When the ALTER DATABASE command is executed, PolarDB initiates a multi-phase distributed transition coordinated by the Global Metadata Service (GMS) and PolarProxy. The process follows these distinct phases:

  1. Routing Freeze: PolarProxy stops routing new write transactions for db1 to RW1.
  2. Draining Phase: RW1 enters a draining state to allow active transactions to finish.
  3. Fencing & Metadata Update: Once drained, RW1's write lease is revoked, and the GMS updates the database ownership metadata to RW2.
  4. Proxy Redirection: PolarProxy is notified of the new owner (RW2) and begins routing queued and new requests there.

Detailed Answers to Your Consistency Questions

1. Does the switch wait for active transactions on RW1 to finish?

Yes, with a timeout. When the switch starts, RW1 enters a draining state. It allows existing, active transactions (like Session A) to either commit or roll back naturally. However, because heavy DML pressure can indefinitely delay a switch, PolarDB enforces a maximum wait timeout. If active transactions do not finish within this threshold, they are aborted (rolled back) to allow the switch to complete.

2. Are new transactions routed to RW1 during the switch?

No. As soon as the ALTER DATABASE statement is received and validated, PolarProxy updates its routing state for db1. No new write transactions or statements will be forwarded to RW1. RW1 is strictly in a draining mode for pre-existing transactions.

3. What happens to new requests (like Session C) during the switch?

While the switch is in progress (after RW1 stops accepting new writes but before RW2 is fully active), PolarProxy temporarily buffers (queues) incoming write requests.

  • If the switch completes quickly (typically within milliseconds to a few seconds), Session C will experience a slight, transparent latency spike and then execute successfully on RW2.
  • If the switch takes longer than the proxy's buffering timeout threshold, Session C will be rejected with a connection timeout or a temporary transaction-abort error, protecting the application from silent failures.

4. Can a transaction started earlier on RW1 commit after RW2 acquires ownership?

No. This is prevented by a strict commit fencing mechanism. PolarDB utilizes epoch/version tokens linked to database write ownership. Once RW2 successfully acquires write ownership, RW1's ownership lease is invalidated. If Session A attempts to execute a COMMIT on RW1 after the handoff, the storage engine detects the ownership epoch mismatch and rejects the commit, forcing a rollback of Session A.

5. What is guaranteed when the ALTER DATABASE statement returns successfully?

The successful return of the ALTER DATABASE statement serves as a strict linearization point. The following conditions are guaranteed:

  • Absolute Fencing: RW1 can no longer start or commit any transactions for db1.
  • Consistent Routing: PolarProxy has fully updated its routing tables, and all subsequent requests for db1 are sent to RW2.
  • State Cleanliness: Any remaining uncommitted transactions on RW1 that failed to drain in time have been aborted and rolled back.

6. What happens if the switch fails midway?

If the switch fails before RW2 successfully completes acquisition (for example, due to a network partition or a timeout during metadata updates), the GMS rolls back the state transition. Ownership safely reverts to RW1, PolarProxy is notified to resume routing to RW1, and queued transactions are dispatched back to RW1. The database does not enter an split-brain state where both or neither node can write.

Production Recommendations

To ensure a smooth, zero-downtime database switchover in your PolarDB Multi-Master environment, follow these best practices:

  • Minimize Long-Running Transactions: Avoid running large batch updates or long-running analytical queries on the target database right before executing the switch.
  • Configure Proxy Buffering: Ensure your PolarProxy configuration has adequate buffer timeout settings to hold client queries during the brief transition window.
  • Handle Transient Errors in Application Code: Implement robust retry logic in your application layer to handle transient connection drops or transaction rollbacks that might occur if a transaction is aborted during the draining timeout.