Skip to content

[hotfix][docs] Recommend 8MB for KV leader replica memory protection #3730

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
swuferhong merged 1 commit into from
Jul 22, 2026
Merged

[hotfix][docs] Recommend 8MB for KV leader replica memory protection #3730

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion fluss-common/src/main/java/org/apache/fluss/config/ConfigOptions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -347,7 +347,9 @@ public class ConfigOptions {
+ "KV leader replica capacity. This value does not reserve or "
+ "enforce memory on the tablet server. The default value of 0 "
+ "disables memory-based KV leader replica capacity control. "
+ "Set a positive value to enable it.");
+ "Set a positive value to enable it. When enabled, the recommended "
+ "value is 8 MB, based on the measured memory consumption of an "
+ "empty KV leader replica.");

/**
* The network address and port the server binds to for accepting connections.
Expand Down
1 change: 1 addition & 0 deletions website/docs/maintenance/configuration.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,7 @@ during the Fluss cluster working.
| allow.create.kv.tables | Boolean | true | Whether to allow creation of kv tables (primary key tables). When set to false, attempts to create kv tables (tables with primary key) will be rejected. The default value is true. |
| max.partition.num | Integer | 1000 | Limits the maximum number of partitions that can be created for a partitioned table to avoid creating too many partitions. |
| max.bucket.num | Integer | 4096 | The maximum number of buckets that can be created for a non-partitioned table or for each partition of a partitioned table. The default value is 4096. This default is capped to reduce the risk that an assignment znode exceeds ZooKeeper's packet size limit. |
| kv.leader-replica.memory-reserved | MemorySize | 0 bytes | The estimated memory consumption of each KV leader replica, used by the CoordinatorServer to calculate the cluster-level KV leader replica capacity. This value does not reserve or enforce memory on the TabletServer. The default value of `0` disables memory-based KV leader replica capacity control. Set a positive value to enable it. When enabled, the recommended value is `8 MB`, based on the measured memory consumption of an empty KV leader replica. |
| acl.notification.expiration-time | Duration | 15min | The duration for which ACL notifications are valid before they expire. This configuration determines the time window during which an ACL notification is considered active. After this duration, the notification will no longer be valid and will be discarded. The default value is 15 minutes. This setting is important to ensure that ACL changes are propagated in a timely manner and do not remain active longer than necessary. |
| authorizer.enabled | Boolean | false | Specifies whether to enable the authorization feature. If enabled, access control is enforced based on the authorization rules defined in the configuration. If disabled, all operations and resources are accessible to all users. |
| authorizer.type | String | default | Specifies the type of authorizer to be used for access control. This value corresponds to the identifier of the authorization plugin. The default value is `default`, which indicates the built-in authorizer implementation. Custom authorizers can be implemented by providing a matching plugin identifier. |
Expand Down
30 changes: 15 additions & 15 deletions website/docs/maintenance/operations/resource-protection.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,8 +83,9 @@ KV leader replica capacity protection is a coordinator-side admission guard that
leader replica capacity from TabletServer memory and rejects new KV table or partition creation once that capacity
would be exceeded.

The feature is **disabled by default** and is considered an advanced/opt-in feature. Enable it once you have a
sense of how much memory each KV leader replica costs in your environment.
The feature is **disabled by default** and is considered an advanced/opt-in feature. When enabling it, the
recommended value is **8 MB per KV leader replica**, based on the measured memory consumption of an empty KV
leader replica. Adjust the value if measurements from your workload show a different per-replica footprint.

### When to Enable It

Expand All @@ -108,13 +109,15 @@ The feature is controlled entirely by a single option, `kv.leader-replica.memory
# Disabled (default)
kv.leader-replica.memory-reserved: 0

# Enabled: reserve an estimated 512MB per KV leader replica
kv.leader-replica.memory-reserved: 512mb
# Enabled with the recommended estimate of 8MB per KV leader replica
kv.leader-replica.memory-reserved: 8mb
```

- `0` (the default) disables memory-based KV leader replica capacity control entirely.
- A positive value enables it, and is used as the estimated memory cost of a single KV leader replica when
calculating the cluster-level capacity (see [How Capacity Is Calculated](#how-capacity-is-calculated) below).
- `8mb` is the recommended value when enabling the feature, based on the measured memory consumption of an empty
KV leader replica. Tune it if your workload measurements indicate a different per-replica memory footprint.
- Negative values are invalid and are rejected at both static startup and dynamic reconfiguration time.

:::tip
Expand Down Expand Up @@ -158,12 +161,12 @@ follows:
unknown servers.
5. `kvLeaderReplicaCapacity = totalEstimatedMemory / kv.leader-replica.memory-reserved`.

For example, with `kv.leader-replica.memory-reserved: 1gb` and 4 live TabletServers where 3 report 32GB each and
For example, with `kv.leader-replica.memory-reserved: 8mb` and 4 live TabletServers where 3 report 32GB each and
one reports no memory:
- 3 out of 4 servers report memory, so the limit is enabled (more than half).
- The unknown server is assumed to have the average of the known servers: 32GB.
- Total estimated memory = 32GB × 4 = 128GB.
- Capacity = 128GB / 1GB = 128 KV leader replicas.
- Capacity = 128GB / 8MB = 16,384 KV leader replicas.

The capacity is recalculated on demand from the current live TabletServer set and is not persisted; it is
recomputed on CoordinatorServer leadership startup and whenever a table or partition creation request is checked.
Expand All @@ -181,7 +184,7 @@ Not enough KV leader replica capacity. observedKvLeaderReplicaCount=<count>, req

If you hit this error, you can:
- Increase capacity by adding TabletServers or more memory per TabletServer.
- Raise `kv.leader-replica.memory-reserved` only if your existing estimate was too conservative — raising it
- Raise `kv.leader-replica.memory-reserved` only if your existing estimate was too low — raising it
without cause lowers the effective capacity.
- Temporarily disable the limit (set `kv.leader-replica.memory-reserved` to `0`) if you need to proceed
immediately, then re-tune before re-enabling.
Expand All @@ -198,20 +201,17 @@ The CoordinatorServer exposes two gauges for monitoring:
Monitor `kvLeaderReplicaCount` against `kvLeaderReplicaCapacity` to anticipate when a cluster is approaching its
limit before requests start failing.

### Example: Enabling, Tuning, and Disabling
### Example: Enabling and Disabling

```yaml title="conf/server.yaml"
# Step 1: Enable with an initial conservative estimate per KV leader replica
kv.leader-replica.memory-reserved: 1gb
# Step 1: Enable with the recommended estimate per KV leader replica
kv.leader-replica.memory-reserved: 8mb

# Step 2 (optional): Override auto-detected resource reporting on a TabletServer
# with constrained cgroup visibility
tablet-server.advertised-resource.memory-size: 64gb
tablet-server.advertised-resource.cpu-cores: 16

# Step 3: Tune down once real per-replica memory usage is measured
kv.leader-replica.memory-reserved: 256mb

# Step 4: Disable if no longer needed
# Step 3: Disable if no longer needed
kv.leader-replica.memory-reserved: 0
```
```
Loading