Redpanda Consumer Offsets: Manage and Reset (2026)

Redpanda consumer offsets are how Kafka consumers keep track of what messages they’ve already processed.

Let’s see Redpanda consumer offsets in action. Imagine a user_signups topic. A signup_processor consumer group is reading from it.

# Show current offsets for the signup_processor consumer group
rpk group list signup_processor --offsets

# Example Output:
# GROUP           TOPIC         PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG
# signup_processor user_signups  0          150             200             50
# signup_processor user_signups  1          120             180             60

Here, CURRENT-OFFSET is where the consumer is at, and LOG-END-OFFSET is the latest message in the topic. LAG is the difference – how many messages are waiting to be processed. If LAG stays high, the consumer is falling behind.

The core problem Redpanda consumer offsets solve is exactly-once processing. Without them, if a consumer crashes after processing a message but before committing the offset, it might re-process that message on restart. Offsets prevent this by acting as a reliable checkpoint. Redpanda stores these offsets in a special topic, usually named __consumer_offsets.

Internally, when a consumer fetches messages, it periodically sends a CommitOffset request to Redpanda. This request includes the topic, partition, and the offset of the next message to be fetched. Redpanda then updates the stored offset for that consumer group and partition. When the consumer restarts, it fetches messages starting from the committed offset.

You manage consumer offsets primarily through the rpk command-line tool or by using Kafka client libraries.

# List all consumer groups
rpk group list

# Describe a specific consumer group's state
rpk group describe signup_processor

# List offsets for all topics a consumer group is subscribed to
rpk group offsets signup_processor

The most common operation you’ll perform is resetting offsets. This is crucial when you need to reprocess messages, perhaps after fixing a bug in your consumer logic or if you want to re-analyze historical data.

Here are the common scenarios and how to reset offsets:

1. Resetting to the Earliest Offset (Beginning of the Topic): This is useful if you want your consumer to start processing from the very first message ever written to the topic.

Diagnosis: Check the current offset and log end offset.
```
rpk group offsets signup_processor
```

Fix: Use rpk group reset with the --to-beginning flag.

rpk group reset signup_processor --to-beginning --topic user_signups

Why it works: This command tells Redpanda to update the stored CURRENT-OFFSET for the signup_processor group on the user_signups topic to -1 (which Kafka/Redpanda interprets as the offset before the first message, so the consumer will fetch the first message).

2. Resetting to the Latest Offset (End of the Topic): This is useful if you want your consumer to only process new messages arriving from now on, effectively skipping any backlog.

Diagnosis: Again, check current offsets.
```
rpk group offsets signup_processor
```

Fix: Use rpk group reset with the --to-end flag.

rpk group reset signup_processor --to-end --topic user_signups

Why it works: This updates the CURRENT-OFFSET to the LOG-END-OFFSET, meaning the consumer will start fetching from the next message that appears.

3. Resetting to a Specific Offset: This allows for precise control, perhaps to reprocess messages from a particular point in time or after a known bad event.

Diagnosis: You’ll need to know the specific offset value you want to reset to. This might come from logs, monitoring, or manual inspection of topic data.

Fix: Use rpk group reset with the --to-offset flag.

rpk group reset signup_processor --to-offset 100 --topic user_signups

Why it works: The CURRENT-OFFSET for the specified group and topic will be set to 100. The consumer will then fetch messages starting from offset 100.

4. Resetting for a Specific Partition: Sometimes, you only need to reset offsets for a single partition, not the entire topic for the group.

Diagnosis: Identify the problematic partition.

Fix: Add the --partition flag to any of the above reset commands.

rpk group reset signup_processor --to-beginning --topic user_signups --partition 0

Why it works: This isolates the offset change to only partition 0 of the user_signups topic for the signup_processor group, leaving other partitions untouched.

5. Resetting All Offsets for a Consumer Group: If you want to completely clear the state for a consumer group and have it re-process everything from the beginning across all its topics.

Diagnosis: Confirm the group name and its scope.

Fix: Omit the --topic and --partition flags.

rpk group reset signup_processor --to-beginning

Why it works: This applies the --to-beginning directive to all topics that the signup_processor group is currently tracking offsets for.

When you reset offsets, Redpanda atomically updates the offset in the __consumer_offsets topic. This ensures that even if a consumer or Redpanda itself restarts during the reset operation, the offset state remains consistent. The rpk tool abstracts this, but under the hood, it’s performing targeted writes to __consumer_offsets.

A common pitfall is forgetting to stop your consumer application before resetting offsets. If the consumer is still running and actively committing offsets, its commits can overwrite your reset, making it seem like the reset didn’t work. Always stop consumers, perform the reset, and then restart them.

After successfully resetting offsets and restarting your consumers, the next thing you’ll likely encounter is a period of high consumer lag as the application catches up.