Search before asking

• I searched in the issues and found nothing similar.

Motivation

Currently, Fluss uses HighWatermark to ensure data synchronization between the leader and followers. However, this approach can lead to data loss or data inconsistency during cluster upgrades or when tabletServers crash. Referencing Kafka KIP-101, we plan to replace the HighWatermark mechanism in Fluss with a leader epoch update mechanism.

To achieve this goal, we need to build a consistent LeaderEpochCache across different tabletServers. The update of this LeaderEpochCache relies on the fetchLogRequest pulling the leader's RecordBatch and reading the LeaderEpoch from the batch to update the cache.

Currently, the LogRecordBatch does not include the LeaderEpoch, so it needs to be introduced, and the magic version of LogRecordBatch should be upgraded accordingly.

Solution

No response

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!