Redpanda Cloud BYOC (Bring Your Own Cloud) isn’t just about deploying Redpanda in your VPC; it’s about reclaiming control over your cloud infrastructure’s networking and security posture for your Kafka-compatible streaming data.
Let’s see it in action. Imagine you’ve got a Kubernetes cluster running in AWS, and you want to deploy a Redpanda Cloud BYOC cluster. You’re not just spinning up pods; you’re defining network interfaces, security groups, and routing tables that Redpanda will inhabit.
Here’s a simplified look at what the underlying infrastructure might look like in AWS after you provision a BYOC cluster. This isn’t a kubectl apply output; it’s the AWS resources that Redpanda orchestrates for you:
{
"EC2": {
"Instances": [
{
"InstanceId": "i-0123456789abcdef0",
"InstanceType": "m5.xlarge",
"ImageId": "ami-0abcdef1234567890",
"State": {"Name": "running"},
"PrivateIpAddress": "10.0.1.10",
"Tags": [
{"Key": "Name", "Value": "redpanda-cloud-byoc-node-1"},
{"Key": "redpanda.com/cluster-id", "Value": "your-cluster-id"}
]
}
// ... more instances
],
"NetworkInterfaces": [
{
"NetworkInterfaceId": "eni-0123456789abcdef0",
"PrivateIpAddress": "10.0.1.10",
"SubnetId": "subnet-0123456789abcdef0",
"Groups": ["sg-0123456789abcdef0"]
}
// ... more ENIs
],
"Vpcs": [
{
"VpcId": "vpc-0123456789abcdef0",
"CidrBlock": "10.0.0.0/16"
}
],
"Subnets": [
{
"SubnetId": "subnet-0123456789abcdef0",
"CidrBlock": "10.0.1.0/24",
"VpcId": "vpc-0123456789abcdef0"
}
],
"SecurityGroups": [
{
"GroupId": "sg-0123456789abcdef0",
"GroupName": "redpanda-cloud-byoc-sg",
"IpPermissions": [
{
"IpProtocol": "tcp",
"FromPort": 9092,
"ToPort": 9092,
"IpRanges": [{"CidrIp": "10.0.0.0/16"}] // Allowing traffic from within the VPC
},
{
"IpProtocol": "tcp",
"FromPort": 2181, // ZooKeeper port
"ToPort": 2181,
"IpRanges": [{"CidrIp": "10.0.0.0/16"}]
}
// ... other necessary ports like 3306 for SQL, 8081 for REST, etc.
]
}
]
}
}
This JSON snippet represents the pieces of your cloud infrastructure that Redpanda Cloud BYOC provisions and manages within your designated VPC. It’s not just about Redpanda itself, but the secure, isolated network environment it needs to operate. You’re essentially telling Redpanda: "Here’s a secure, private space in my cloud, go set up shop." This allows you to integrate Redpanda deeply with your existing VPC resources, leverage your existing network security policies, and ensure data stays within your control plane boundaries.
The core problem Redpanda Cloud BYOC solves is the tension between the ease of a managed service and the strict requirements for data locality, network isolation, and security that many enterprises have. Traditional managed Kafka services often operate in their own dedicated networks, requiring complex VPC peering or private link configurations to connect to your resources. BYOC sidesteps this by deploying Redpanda directly into your VPC, making it a first-class citizen alongside your other cloud resources.
Internally, Redpanda Cloud BYOC leverages cloud provider APIs to provision and manage the necessary compute, networking, and storage resources. When you create a BYOC cluster, Redpanda Cloud’s control plane interacts with your cloud account (via IAM roles or similar mechanisms) to:
- Provision Compute: Launch EC2 instances (or equivalent in other clouds) configured with the Redpanda binary.
- Configure Networking: Create or select subnets, assign IP addresses, and set up Elastic Network Interfaces (ENIs).
- Enforce Security: Create and associate security groups or network security rules to control inbound and outbound traffic, allowing only necessary ports (like 9092 for Kafka, 2181 for ZooKeeper, 8081 for HTTP API) from your VPC CIDR.
- Manage Storage: Provision and attach block storage (EBS volumes, etc.) for Redpanda’s data directories.
- Orchestrate Deployment: Deploy the Redpanda binaries onto the provisioned instances and configure them to form a cluster, often using Kubernetes or a similar orchestration layer managed by Redpanda Cloud.
The exact levers you control are primarily at the provisioning stage. You define the cloud provider, the region, the VPC, the subnets, and the security groups where Redpanda will be deployed. You also specify instance types and storage configurations. Once deployed, Redpanda Cloud manages the cluster’s lifecycle, upgrades, and operational health, but the underlying network and compute are yours.
The truly elegant part of BYOC is how it leverages your existing VPC’s routing and peering. If your applications are already within the same VPC or peered networks, they can reach the Redpanda brokers using their private IP addresses without any public internet exposure or complex intermediary services. This significantly reduces latency and simplifies network architecture while enhancing security by keeping traffic internal to your cloud environment.
The next step in managing a BYOC deployment involves understanding how to integrate it with your CI/CD pipelines, specifically how to automate the provisioning and updates of these cloud resources as part of your application deployment lifecycle.