OpenShift Hosted Control Planes with KubeVirt Workers and User-Defined Networks

Red Hat's Hosted Control Planes (HCP) represent a significant shift in how we think about Kubernetes cluster architecture. Instead of the traditional model where control plane components run on dedicated master nodes, HCP allows the control plane to run as pods on a management cluster while worker nodes can be virtual or bare metal machines. In this post I will describe how to setup HCP with virtual machine workers using Openshift Virtualisation.

What Are Hosted Control Plans?

In a traditional OpenShift deployment, you have master nodes running etcd, the API server, controller manager, and scheduler. These master nodes are dedicated infrastructure that you need to provision, patch, and maintain. With Hosted Control Planes, all of those control plane components become containers running on a management (hub) cluster.

The workers? They can be anything that can join a cluster as a node. In this demo, we're using KubeVirt virtual machines as worker nodes. So you get a full OpenShift cluster where the control plane is just a set of pods, and the workers are VMs running inside the management cluster.

Why Would You Want This?

Density. Pure and simple. You can run multiple OpenShift clusters on a single management cluster without needing separate physical infrastructure for each cluster's control plane. Each hosted cluster gets its control plane as pods, and workers as VMs. It's multi-tenancy taken to the next level.

And with User-Defined Networks (UDN), you get Layer 2 network isolation for each hosted cluster's data plane. So your hosted clusters are properly isolated from each other at the network level.

The Architecture

Hub cluster (management)
├── openshift-cnv           CNV operator + HyperConverged
├── local-cluster            hypershift-addon ManagedClusterAddOn
│
├── clusters                 HostedCluster + NodePool CRs
│
└── clusters-hcp-demo        (auto-adopted, pre-created with UDN label)
    ├── UDN: hcp-worker-udn  Layer 2 primary network (10.200.0.0/16)
    ├── Control plane pods    etcd, kube-apiserver, openshift-apiserver, ...
    ├── VM: hcp-demo-0        KubeVirt worker node (4 vCPU, 8Gi, 64Gi disk)
    └── VM: hcp-demo-1        KubeVirt worker node (4 vCPU, 8Gi, 64Gi disk)

Guest cluster (hcp-demo)
├── Node: hcp-demo-0          Ready
└── Node: hcp-demo-1          Ready

All VMs and control plane pods in clusters-hcp-demo use the UDN as their primary network interface. The default OVN cluster network remains available as a secondary interface for hub infrastructure connectivity (DNS, API, etc.).

The Deployment Process - GitOps Style

This demo uses Argo CD to manage the entire deployment through GitOps. The manifests are organized with sync waves so everything deploys in the right order:

| Wave | Resource | Purpose | |------|----------|---------| | -5 | CNV operator | Virtualization infrastructure | | -3 | MultiClusterHub patch | Enable CNV/MTV integrations | | -3 | hypershift-addon | HyperShift management addon | | -1 | HyperConverged CR | Activate KubeVirt | | 0 | Namespaces, Secrets, UDN | Foundation resources | | 1 | HostedCluster | Define the guest cluster | | 2 | NodePool | Create the worker VMs |

Prerequisites - The Reality Check

Before you get excited, there are some requirements:

• ACM / MCE installed on the hub cluster
• Bare-metal or nested-virt capable nodes - AWS metal instances, or c8i/m8i/r8i instances with nested virtualization
• gp3-csi StorageClass available
• OpenShift pull secret from console.redhat.com
• Hub cluster sizing - at least 3 worker nodes with 16+ cores and 64+ GiB RAM each

The hosted cluster control plane plus 2 worker VMs consume significant resources. This isn't a lightweight laptop demo.

Step-by-Step Deployment

1. Enable Nested Virtualization (AWS Only)

If you're on AWS with non-metal instances, you need nested virtualization:

./virt-enable.sh

This script finds all worker EC2 instances, stops them, enables nested virtualization via boto3, and starts them back up. It then verifies /dev/kvm is present on each worker node.

2. Enable Wildcard Ingress

The hosted cluster API and apps are exposed through the hub's default ingress via baseDomainPassthrough. This requires wildcard routes:

oc patch ingresscontroller -n openshift-ingress-operator default \
  --type=json -p '[{"op":"add","path":"/spec/routeAdmission","value":{"wildcardPolicy":"WildcardsAllowed"}}]'

3. Generate Secrets

The setup script handles the sensitive bits:

./setup-hcp-secrets.sh ~/pull-secret.json

This generates:

• pull-secret.yaml - Your OpenShift pull secret
• etcd-encryption-secret.yaml - Random etcd encryption key
• Updates hostedcluster.yaml with your hub's base domain

4. Apply Secrets

Secrets are managed out-of-band from GitOps. Apply them before deploying:

oc create namespace clusters --dry-run=client -o yaml | oc apply -f -
oc apply -f manifests/hcp/pull-secret.yaml
oc apply -f manifests/hcp/etcd-encryption-secret.yaml

5. Deploy the Argo CD Application

oc apply -f application.yaml

Argo CD syncs everything automatically. The full deployment takes 15-30 minutes. The CNV operator installs first, then HyperShift enables, and finally the HostedCluster and NodePool create the control plane pods and worker VMs.

6. Fix the DNS Wildcard Record

Here's where it gets interesting. The HyperShift DNS operator creates a *.apps.hcp-demo.<baseDomain> wildcard DNS record, but points it at the private-router LB (which only handles API/OAuth/ignition/konnectivity traffic). App routes must go through the management cluster's default ingress router instead.

./fix-hcp-dns.sh

This script finds every Route53 zone (public and private) that contains the wrong wildcard record and corrects it. Without this fix, the hosted cluster stays in Partial progress because the console and ingress canary checks fail.

Validating the Deployment

CNV Operator

oc get csv -n openshift-cnv | grep kubevirt
# STATUS should be "Succeeded"

oc get hyperconverged kubevirt-hyperconverged -n openshift-cnv -o jsonpath='{.status.conditions[?(@.type=="Available")].status}'
# Should print "True"

HyperShift Addon

oc get managedclusteraddon hypershift-addon -n local-cluster
# Should show Available = True

HostedCluster

oc get hostedcluster hcp-demo -n clusters
# PROGRESS should eventually show "Completed"

NodePool and VMs

oc get nodepool hcp-demo -n clusters
# REPLICAS / READY should both be 2

oc get vm -n clusters-hcp-demo
# Should show 2 VMs in "Running" status

Guest Cluster Access

# Extract the guest cluster kubeconfig
oc get secret -n clusters hcp-demo-admin-kubeconfig -o jsonpath='{.data.kubeconfig}' | base64 -d > /tmp/hcp-demo-kubeconfig

# Verify guest cluster nodes
oc --kubeconfig=/tmp/hcp-demo-kubeconfig get nodes
# Should show 2 nodes in Ready status

The User-Defined Network Piece

The UDN is what provides network isolation for the hosted cluster:

apiVersion: k8s.ovn.org/v1
kind: UserDefinedNetwork
metadata:
  name: hcp-worker-udn
  namespace: clusters-hcp-demo
spec:
  topology: Layer2
  layer2:
    role: Primary
    subnets:
      - "10.200.0.0/16"
    ipam:
      lifecycle: Persistent

This creates a Layer 2 primary network for the clusters-hcp-demo namespace. All pods and VMs in that namespace use this network as their primary interface. The default OVN cluster network remains available as a secondary interface.

If the UDN interferes with infrastructure connectivity (control plane pods can't reach hub services), you can remove it as a fallback:

oc delete userdefinednetwork hcp-worker-udn -n clusters-hcp-demo
oc label namespace clusters-hcp-demo k8s.ovn.org/primary-user-defined-network-

Troubleshooting the Partial State

If your HostedCluster gets stuck in Partial, check the conditions:

oc get hostedcluster hcp-demo -n clusters -o jsonpath='{.status.conditions}' | python3 -m json.tool

Common causes:

• DNS wildcard pointing to wrong LB - run ./fix-hcp-dns.sh
• Pull secret is invalid or missing
• etcd encryption key is missing
• CNV operator not fully ready
• Insufficient hub cluster resources

Tear Down

oc delete -f application.yaml

Argo CD prunes all managed resources. The KubeVirt VMs, HostedCluster, UDN, and operators are removed. If the HostedCluster gets stuck during deletion, you may need to remove its finalizers manually.

Why This Matters

This architecture is the future of Kubernetes multi-tenancy. You get:

• Resource efficiency: Multiple clusters on shared infrastructure
• Rapid provisioning: Spin up new clusters in minutes, not hours
• Strong isolation: UDN provides Layer 2 network separation
• Simplified management: Control planes are just pods - patch and upgrade the hub, all hosted clusters benefit
• VM-based workers: KubeVirt gives you the flexibility of VMs with the management of containers

Complicated? A bit. Powerful? Absolutely. This is the kind of architecture that makes you rethink how you deploy and manage Kubernetes at scale.