Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Set cluster domain inside a tenant cluster #580

rossbachp started this conversation in Feature Requests
Set cluster domain inside a tenant cluster #580
@rossbachp rossbachp
Sep 16, 2024 · 2 comments · 4 replies
Return to top
Discussion options

rossbachp
Sep 16, 2024

We want to generate a tenant cluster with clusterAPI and Kubevirt that can be used there own cluster domain.

We set the serviceDomain at the cluster definition:

apiVersion: cluster.x-k8s.io/v1beta1
kind: Cluster
metadata:
  name: "${CLUSTER_NAME}"
  namespace: "${NAMESPACE}"
  labels:
    cluster.x-k8s.io/cluster-name: "${CLUSTER_NAME}"
spec:
  clusterNetwork:
    pods:
      cidrBlocks:
      - 10.243.0.0/16
    serviceDomain: ${CLUSTER_NAME}.local  
    services:
      cidrBlocks:
      - 10.96.0.0/16
  controlPlaneRef:
    apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
    kind: KamajiControlPlane
    name: ${CLUSTER_NAME}
    namespace: ${NAMESPACE}
  infrastructureRef:
    apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
    kind: KubevirtCluster
    name: ${CLUSTER_NAME}
    namespace: ${NAMESPACE}

Setup the ControlPlane with this config:

apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
  name: ${CLUSTER_NAME}
  namespace: ${NAMESPACE}
spec:
  dataStoreName: ${KAMAJI_DATASTORE}
  addons:
    coreDNS:
      dnsServiceIPs:
      - 10.96.0.10
  kubelet:
    cgroupfs: systemd
    preferredAddressTypes:
    - InternalIP
    - ExternalIP
  network:
    serviceType: LoadBalancer
    serviceAnnotations:
      lbipam.cilium.io/ips: "192.168.108.8"
  deployment: {}
  replicas: ${CONTROL_PLANE_MACHINE_COUNT}
  version: "${KUBERNETES_VERSION}"

Kamaji Version that we used:

helm search repo kamaji
NAME                    CHART VERSION   APP VERSION     DESCRIPTION                                       
clastix/kamaji          1.0.0           v1.0.0          Kamaji is the Hosted Control Plane Manager for ...
clastix/kamaji-console  0.0.5           v0.0.2          Kamaji deploys and operates Kubernetes at scale...
clastix/kamaji-etcd     0.7.0           3.5.6           Helm chart for deploying a multi-tenant `etcd` ...
clastix/console         0.0.1           0.0.1           Kamaji is a tool aimed to build and operate Kub...

Problem is that the resulting cluster use the default "cluster.local" domain

kubectl proxy &
k get nodes
curl -X GET http://127.0.0.1:8001/api/v1/nodes/tenant1-md-0-j8r46-qp5kc/proxy/configz | jq .

    "clusterDomain": "cluster.local",
    "clusterDNS": [
      "10.96.0.10"
    ],

Also included at the coreDNS core file:

kubectl get cm coredns -n kube-system -o jsonpath="{.data.Corefile}" \
  | grep ".local " \
  | awk -F ' ' '{print $2}'
cluster.local
k get cm -n kube-system coredns -o yaml
## output
apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2024-09-13T08:31:44Z"
  name: coredns
  namespace: kube-system
  ownerReferences:
  - apiVersion: rbac.authorization.k8s.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: ClusterRoleBinding
    name: system:coredns
    uid: 3341239f-17aa-4a6c-8c53-3daf4fcc4e94
  resourceVersion: "3615354"
  uid: a519dba3-fd61-4b99-8ad8-aebc5f35eb8a

Is my configuration wrong?
You must be logged in to vote

Replies: 2 comments · 4 replies

Comment options

prometherion
Sep 16, 2024
Maintainer

This is a feature request since we do not currently support this customisation.
You must be logged in to vote
3 replies
@rossbachp
Comment options

rossbachp Sep 16, 2024
Author

OK, but if you want to setup a cilium multi cluster setup, this feature is needed to separate the services to multiple clusters. Hope you can add this feature at some of the next releases:)
@prometherion
Comment options

prometherion Sep 30, 2024
Maintainer

We'll work on this, even though we have different feature requests in pipelines so we can't guarantee when this lands.
@rossbachp
Comment options

rossbachp Oct 1, 2024
Author

I hope you can prioritize supporting this feature, as it is essential for two key scenarios. First, it allows the migration of workloads to newer major Kubernetes cluster releases, which is particularly important when some applications encounter compatibility issues with the latest versions. Second, it facilitates the connection of multi-rack or multi-region clusters using service meshes, enhancing the scalability and resilience of your infrastructure.
Comment options

prometherion
Oct 6, 2024
Maintainer

Implemented with #594, still need to back ported to Cluster API.
You must be logged in to vote
1 reply
@prometherion
Comment options

prometherion Oct 24, 2024
Maintainer

Implemented with the v0.12.0 Kamaji Cluster API Control Plane provider: https://github.com/clastix/cluster-api-control-plane-provider-kamaji/releases/tag/v0.12.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
completed
2 participants
@rossbachp @prometherion
Converted from issue

This discussion was converted from issue #579 on September 16, 2024 05:06.

Morty Proxy This is a proxified and sanitized view of the page, visit original site.