Skip to content

Add support for the CLIENT PAUSE/UNPAUSE commands to provide a lightweight temporary solution for failover(takeover) #3377

@Paragrf

Description

@Paragrf

Search before asking

  • I had searched in the issues and found no similar issues.

Motivation

There has been ongoing discussion in the community regarding Graceful Failover (see apache/kvrocks#2848). While a specific solution was previously proposed in #3295, the process has remained stalled due to several unresolved questions.

To move forward, I have implemented a simplified failover scheme previously discussed in the community using the CLIENT PAUSE and CLIENT UNPAUSE commands. This approach serves two purposes: it implements missing Redis protocols and prevents sequence number (seq) mismatches during the Failover (Takeover) process. Ultimately, this enhances Kvrocks' manual failover capabilities for production environments.

Solution

By introducing CLIENT PAUSE/UNPAUSE, the controller can now manage external writes to Kvrocks, allowing for a secure Failover once the master and slave sequences ($seq$) are aligned.

The behavior of CLIENT PAUSE/UNPAUSE is consistent with Redis:

  • Blocking Mechanism: After executing CLIENT PAUSE, Kvrocks will block connections attempting to run restricted commands.
  • Release/Error Handling: These commands are released once the timeout expires or CLIENT UNPAUSE is called.
  • Role Transition: If a role change occurs (e.g., Master $\rightarrow$ Slave) during the pause, any previously blocked write requests will return a READONLY error upon resumption.

While simply returning an immediate error for write requests during a failover might be a "cleaner" or more concise implementation, I chose the blocking approach to maintain strict parity with Redis behavior and remain faithful to the standard CLIENT PAUSE semantics.

Are you willing to submit a PR?

  • I'm willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions