Pulsar Non-Persistent Topics: Low-Latency Use Cases (2026)

Pulsar’s non-persistent topics are designed for use cases where data durability isn’t the primary concern, but rather extreme low latency and high throughput are paramount.

Let’s see it in action. Imagine a real-time fraud detection system. Transactions stream into Pulsar via a producer. A consumer, acting as the fraud detection engine, reads these transactions, analyzes them, and if fraud is detected, publishes an alert to another topic. The original transaction data, if not immediately actionable for fraud detection, doesn’t need to be stored long-term.

Here’s a simplified producer and consumer setup:

Producer (Java)

import org.apache.pulsar.client.api.*;

public class TransactionProducer {
    public static void main(String[] args) throws Exception {
        PulsarClient client = PulsarClient.builder()
                .serviceUrl("pulsar://localhost:6650")
                .build();

        // Create a non-persistent topic
        String topic = "persistent://public/default/transactions_non_persistent"; // Note: even non-persistent topics are created under 'persistent' namespace.
        Producer<String> producer = client.newProducer(Schema.STRING)
                .topic(topic)
                .create();

        for (int i = 0; i < 100; i++) {
            String message = "TransactionData-" + i;
            producer.send(message);
            System.out.println("Sent: " + message);
            Thread.sleep(10); // Simulate some delay between transactions
        }

        producer.close();
        client.close();
    }
}

Consumer (Java)

import org.apache.pulsar.client.api.*;

public class FraudDetectionConsumer {
    public static void main(String[] args) throws Exception {
        PulsarClient client = PulsarClient.builder()
                .serviceUrl("pulsar://localhost:6650")
                .build();

        String topic = "persistent://public/default/transactions_non_persistent";
        String subscriptionName = "fraud-detector-sub";

        Consumer<String> consumer = client.newConsumer(Schema.STRING)
                .topic(topic)
                .subscriptionName(subscriptionName)
                .subscribe();

        System.out.println("Fraud detection consumer started. Listening for transactions...");

        while (true) {
            Message<String> message = consumer.receive();
            String receivedMessage = message.getValue();
            System.out.println("Received: " + receivedMessage);

            // Simulate fraud detection logic
            if (receivedMessage.contains("high_value")) {
                System.out.println("ALERT: Potential fraud detected in " + receivedMessage);
                // Publish alert to another topic, etc.
            }

            consumer.acknowledge(message);
        }
        // Note: This loop is infinite for demonstration; in a real app, you'd have shutdown logic.
    }
}

When you run these, you’ll see messages flowing with minimal delay. The key here is that Pulsar, by default, doesn’t write non-persistent messages to disk. They are held in memory and delivered to active consumers.

The problem Pulsar’s non-persistent topics solve is the overhead associated with durable message storage when that durability is not strictly required. Traditional message queues often incur disk I/O for every message, even if it’s only in memory for a few milliseconds before being consumed. This disk latency can become a bottleneck for applications demanding sub-millisecond processing times. Pulsar’s non-persistent topics bypass this disk I/O entirely for the message data itself.

Internally, when you create a topic with the non-persistent:// prefix, Pulsar’s broker is instructed to store message payloads in memory (RAM) rather than in the bookkeeper bookies. The broker’s internal data structures, like Entry objects, are managed in RAM. Once a message is acknowledged by a consumer, the broker discards it from its memory. If a broker restarts, all non-persistent messages it held will be lost. This is why they are called "non-persistent."

The levers you control are primarily the topic name and the client configuration. The topic name must start with non-persistent://. For example, non-persistent://public/default/my-low-latency-topic. You can also configure the maximum number of messages or bytes a broker will hold in memory for non-persistent topics using broker configuration properties like maxMemoryForNonPersistentTopics and maxNumEntriesForNonPersistentTopics. However, these are cluster-wide settings and should be adjusted with extreme care, as exceeding them can lead to out-of-memory errors.

What most people don’t realize is that while the message data isn’t written to bookkeeper, the metadata for non-persistent topics, such as the topic’s existence, schema, and subscription states, is still managed and persisted by ZooKeeper (for metadata) and potentially bookkeeper (for ledger management, even if empty). This ensures that Pulsar can correctly manage topic discovery and consumer subscriptions, even for topics that don’t store message payloads durably. The topic itself will exist and be discoverable by clients.

The next step you’ll likely explore is how to combine non-persistent topics with persistent ones for hybrid scenarios, or how to implement dead-letter queues for messages that fail processing on non-persistent topics.