Pulsar topic auto-creation is enabled by default, but it doesn’t mean topics are created for every possible topic name.
Let’s see it in action. Imagine you have a Pulsar cluster and you haven’t explicitly created a topic named persistent://my-tenant/my-namespace/my-topic. If a producer tries to send a message to this topic, or a consumer tries to subscribe to it, Pulsar will automatically create the topic for you. This happens seamlessly, without any explicit API calls from your side to create the topic beforehand.
This behavior is controlled by the autoTopicCreationEnable setting in the Pulsar broker configuration. When this is true, the broker intercepts requests for non-existent topics and creates them on the fly. This is incredibly convenient because it abstracts away the need for manual topic provisioning, especially in dynamic environments where topics might be created and deleted frequently based on application needs.
The underlying mechanism is straightforward. When a broker receives a request (publish or subscribe) for a topic that doesn’t exist, it checks its configuration for autoTopicCreationEnable. If enabled, it proceeds to create the topic with default settings. These default settings include things like the number of partitions, message TTL, and retention policies.
However, there’s more granularity. You can configure default policies for these auto-created topics. This is where autoTopicCreationPolicy comes into play. Instead of just creating a topic with the absolute bare minimum, you can specify default values for properties like the number of partitions or the namespace for managed ledgers.
Here’s a snippet of a broker.conf that illustrates this:
# Enable topic auto-creation
autoTopicCreationEnable=true
# Set default number of partitions for auto-created topics
autoTopicCreationPolicy={
"defaultNumPartitions": 16,
"defaultManagedLedgerList": ["my-partitioned-topic-ledger"]
}
In this example, any topic auto-created by this broker will initially have 16 partitions. The defaultManagedLedgerList is a more advanced setting, often used in specific deployment scenarios. For most users, defaultNumPartitions is the most relevant setting to control.
This allows you to enforce a baseline configuration for your topics without requiring explicit creation. For instance, if your applications generally perform better with a higher number of partitions, setting defaultNumPartitions to a value like 16 or 32 ensures that all auto-created topics start with that configuration, avoiding potential performance bottlenecks from topics created with the absolute default of 1 partition.
The autoTopicCreationPolicy itself is a JSON string. The broker parses this string to apply the specified defaults. If a topic is created explicitly with certain properties (e.g., a specific number of partitions), those explicit settings will override the defaults specified in autoTopicCreationPolicy. The auto-creation mechanism only applies defaults when no explicit configuration exists for the topic.
The most surprising true thing about Pulsar topic auto-creation is that even when autoTopicCreationEnable is true, the broker won’t just create any topic. It needs a valid tenant and namespace to exist first. For example, if you try to auto-create persistent://my-tenant/my-namespace/my-topic and my-tenant or my-namespace doesn’t exist, the auto-creation will fail. This is a crucial distinction: auto-creation works within the existing tenant/namespace hierarchy, it doesn’t create tenants or namespaces themselves.
The next logical step after understanding default policies is to explore how to manage topic configurations at a more granular level, such as per-namespace settings, which allow for even finer-grained control over topic behavior.