记一次基于small-step对局域网内证书的自签名
我们已经解决了K8s内集群的证书 现在需要解决我们直接跑在虚拟机上的一些服务的证书签名了
之前是自己签名 现在发现可以用根证书+ACME统一签名
因为是根证书 实际上可以在Ingress里设置任何一个域名(即使他已经存在)
甚至可以把baidu换成google(
方案选择
Caddy
我们可以使用Caddy的tls internal去简单的给局域网的服务设置反代
我的打算是Caddy在K8s上运行 我们给他单独分配一个LoadBalanceIP
直接贴上yml
---apiVersion: apps/v1kind: Deploymentmetadata: annotations: {} labels: k8s.kuboard.cn/name: caddy-deployment name: caddy-deployment namespace: caddy-namespacespec: progressDeadlineSeconds: 600 replicas: 1 revisionHistoryLimit: 3 selector: matchLabels: k8s.kuboard.cn/name: caddy-deployment template: metadata: labels: k8s.kuboard.cn/name: caddy-deployment spec: containers: - image: caddy imagePullPolicy: IfNotPresent name: caddy ports: - containerPort: 80 name: http - containerPort: 443 name: https volumeMounts: - mountPath: /etc/caddy/Caddyfile name: volume-piaii subPath: Caddyfile dnsPolicy: ClusterFirst restartPolicy: Always schedulerName: default-scheduler securityContext: {} terminationGracePeriodSeconds: 30 volumes: - name: caddy-volume persistentVolumeClaim: claimName: caddy-pvc - configMap: items: - key: Caddyfile path: Caddyfile name: caddy-config name: volume-piaii
---apiVersion: v1kind: Servicemetadata: annotations: metallb.io/ip-allocated-from-pool: default-address-pool labels: k8s.kuboard.cn/name: caddy-deployment name: caddy-deployment namespace: caddy-namespacespec: allocateLoadBalancerNodePorts: true externalTrafficPolicy: Cluster internalTrafficPolicy: Cluster ipFamilies: - IPv4 ipFamilyPolicy: SingleStack ports: - name: http nodePort: 31922 port: 80 protocol: TCP targetPort: 80 - name: https nodePort: 30408 port: 443 protocol: TCP targetPort: 443 selector: k8s.kuboard.cn/name: caddy-deployment sessionAffinity: None type: LoadBalancerstatus: loadBalancer: ingress: - ip: 192.168.0.241 ipMode: VIP---apiVersion: v1kind: ConfigMapmetadata: name: caddy-configdata: Caddyfile: | gitlab.lan { reverse_proxy 192.168.0.37:80 tls internal }以上yml创建了一个Deployment 并且设置了ConfigMap以及PVC存储Caddy的数据
我这里用的是Longhorn
然后Caddy的Service被分配了一个LoadbalanceIP
只需要把.lan域名全部指向Caddy的LoadbalanceIP就可以用了

没问题 启动后 Caddy 会在 /data/pki/authorities/local/ 下创建内部 CA 及根证书 信任Caddy的证书直接能用
这个很简单 那问题来了
我觉得K8s内和局域网内我需要信任两个证书(*.k8s.lan appname.lan) 太麻烦了
Caddy的TLS internal主要是给测试用的
有没有更高级 可自定义性更强的方案呢
自建ACME
我在思考 既然Let’s Encrypt无法为lan域名签证书
那我们在局域网自己搭建一个证书签发机构不就可以了
然后让Traefik之类的服务去连接我们自己的证书机构负责签名
Step-Ca搭建
因为这玩意折腾我一天了 我这里不想写折腾过程了 懒
首先的话我们需要在本地有Stepca的CLI
winget install Smallstep.step-ca当然你在Linux上也是可以的
apt-get update && apt-get install -y --no-install-recommends curl vim gpg ca-certificatescurl -fsSL https://packages.smallstep.com/keys/apt/repo-signing-key.gpg -o /etc/apt/trusted.gpg.d/smallstep.asc && \ echo 'deb [signed-by=/etc/apt/trusted.gpg.d/smallstep.asc] https://packages.smallstep.com/stable/debian debs main' \ | tee /etc/apt/sources.list.d/smallstep.listapt-get update && apt-get -y install step-cli step-ca我们有主要是为了生成初始证书
也可以用init pod去生成 但那个太麻烦 我懒得写yml 望周知((
然后的话我们自己创建一下证书
# 根证书创建step certificate create \ "Cainong Root CA" \ certs/root_ca.crt \ secrets/root_ca_key \ --profile root-ca \ --no-password \ --insecure \ --not-after 87600h
# 创建中间证书并用根证书签名step certificate create \ "Cainong Intermediate CA" \ certs/intermediate_ca.crt \ secrets/intermediate_ca_key \ --profile intermediate-ca \ --ca certs/root_ca.crt \ --ca-key secrets/root_ca_key \ --no-password \ --insecure \ --not-after 43800hstep-ca需要根证书 中间证书 中间证书的私钥
也就是说,step-ca实际上是使用的中间证书给域名颁发证书
所以我们需要把/secret/intermediate_ca_key intermediate_ca.crt root_ca.crt这两个文件传到Master节点上
然后执行
kubectl create secret generic step-ca-certs -n step-ca --from-file=root-ca.crt=certs/root_ca.crt --from-file=intermediate_ca.crt=certs/intermediate_ca.crt这里只放Deployment的YAML了 Service我相信你会的
apiVersion: apps/v1kind: Deploymentmetadata: annotations: {} labels: k8s.kuboard.cn/name: step-ca-server name: step-ca-server namespace: step-caspec: selector: matchLabels: k8s.kuboard.cn/name: step-ca-server template: metadata: annotations: labels: k8s.kuboard.cn/name: step-ca-server spec: containers: - env: - name: DOCKER_STEPCA_INIT_NAME value: Smallstep - name: DOCKER_STEPCA_INIT_REMOTE_MANAGEMENT value: 'true' - name: DOCKER_STEPCA_INIT_DNS_NAMES value: stepca.k8s.lan - name: CA_PASSWORD valueFrom: secretKeyRef: key: password name: step-ca-password image: 'smallstep/step-ca:latest' imagePullPolicy: IfNotPresent name: step-ca-server ports: - containerPort: 9000 name: stepca protocol: TCP volumeMounts: - mountPath: /home/step/config/ca.json name: config subPath: ca.json - mountPath: /home/step/certs/intermediate_ca.crt name: intermediate-ca subPath: intermediate_ca.crt - mountPath: /home/step/certs/root_ca.crt name: root-ca subPath: root_ca.crt - mountPath: /home/step/secrets/intermediate_ca_key name: intermediate-ca-key subPath: intermediate_ca_key - mountPath: /home/step/secrets/password name: password subPath: password - mountPath: /home/step/db name: data securityContext: runAsGroup: 0 runAsUser: 0 volumes: - configMap: defaultMode: 420 name: step-ca-configmap name: config - name: intermediate-ca secret: defaultMode: 420 items: - key: intermediate_ca.crt path: intermediate_ca.crt secretName: step-ca-certs - name: root-ca secret: defaultMode: 420 items: - key: root-ca.crt path: root_ca.crt secretName: step-ca-certs - name: intermediate-ca-key secret: defaultMode: 420 items: - key: intermediate_ca_key path: intermediate_ca_key secretName: step-ca-password - name: password secret: defaultMode: 420 items: - key: password path: password secretName: step-ca-password - name: data persistentVolumeClaim: claimName: step-ca-pvc不难看到我是用kuboard配置的 手写YAML真要命的
然后是configmap
apiVersion: v1data: ca.json: |2- { "root": "/home/step/certs/root_ca.crt", "crt": "/home/step/certs/intermediate_ca.crt", "key": "/home/step/secrets/intermediate_ca_key", "address": ":9000", "dnsNames": ["step-ca-server.step-ca.svc.cluster.local", "ca.k8s.lan"], "logger": { "format": "text" }, "db": { "type": "badger", "dataSource": "/home/step/db" }, "authority": { "provisioners": [ { "type": "ACME", "name": "acme", "forceCN": true } ] } }kind: ConfigMapmetadata: name: step-ca-configmap namespace: step-ca不出意外step-ca 就能跑起来了
但现在又有个小问题
Ingress与ACME的Service之间通信是用HTTP 这是不允许的
并且ACME要求全程都使用https
也就是说
客户端->Ingress->Service->ACME Pod全部都要用https
也就是说 我们需要先给Ingress申请一个证书 我给ca服务的域名是ca.k8s.lan
回到我们一开始申请证书的地方,我们用根证书先给ca.k8s.lan申请一个先
step certificate create "ca.k8s.lan" ca.k8s.lan.crt ca.k8s.lan.key \ --ca-key secrets/root_ca_key \ --ca certs/root_ca.crt \ --san ca.k8s.lan \ --not-after 87600h \ --insecure --no-password然后把这个证书传到集群内作为TLS Secret 给Ingress装上
还没完 我们需要解决Ingress解密Https请求 我们让Ingress不解密 直接传给Pod
我们需要给Ingres加上这两个设置
annotations: nginx.ingress.kubernetes.io/ssl-passthrough: "true" nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"然后不出意外的话就可以在局域网内给证书签名了 值得一提的是申请证书的设备也要信任我们的根证书才可以

Certmanager
在K8s集群内最好的方式应该还是用Certmanager去负责Ingress的自签名,创建好Issuer之后只需要在Ingress配置里加个标签 Certmanager就会自动帮我们完成http挑战+证书申请+部署
还是很方便的
在Master节点上执行
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager --version v1.19.0 --namespace cert-manager --create-namespace --set crds.enabled=true然后创建一个Cluster Issuer
apiVersion: cert-manager.io/v1kind: ClusterIssuermetadata: name: step-caspec: acme: server: https://ca.k8s.lan/acme/acme/directory #ACME服务器地址 email: admin@k8s.lanspec: acme: privateKeySecretRef: name: my-acme-account-key server: https://ca.k8s.lan/acme/acme/directory solvers: - http01: ingress: class: nginx-ingress # Ingress类名 负责http挑战还没完 因为我们是自己创的证书 所以我们需要让Certmanager也信任根证书才行
简单来说就是把root cert挂载到Certmanager的/etc/ssl/certs/里
还没完 挂了之后重启Pod还是不认 我们还需要添加环境变量SSL_CERT_FILE=/etc/ssl/certs/root_ca.crt
以上都完成之后就可以创建一个Ingress试试了
apiVersion: networking.k8s.io/v1kind: Ingressmetadata: annotations: cert-manager.io/cluster-issuer: my-acme-issuer # 这里要改 labels: app: longhorn-ui name: longhorn-ui namespace: longhorn-systemspec: ingressClassName: nginx-ingress rules: - host: longhorn.k8s.lan http: paths: - backend: service: name: longhorn-ui port: number: 8000 path: / pathType: Prefix tls: - hosts: - longhorn.k8s.lan # 这里改成你需要的域名 secretName: longhorn-tls-secret # SSL的Secret 名称随意只需要添加cert-manager.io/cluster-issuer这一行 他就会自动完成证书申请+挑战+证书部署
还是很强大的
至此局域网内的证书基本就完善了 应该没啥要折腾的了
后续一个小更新:
step-ca默认证书有效期只有24小时 有点太短了其实
我们去修改一下他的configmap
{ "root": "/home/step/certs/root_ca.crt", "crt": "/home/step/certs/intermediate_ca.crt", "key": "/home/step/secrets/intermediate_ca_key", "address": ":9000", "dnsNames": [ "step-ca-server.step-ca.svc.cluster.local", "ca.k8s.lan" ], "logger": { "format": "text" }, "db": { "type": "badger", "dataSource": "/home/step/db" }, "authority": { "provisioners": [ { "type": "ACME", "name": "acme", "forceCN": true, "claims": { "minTLSCertDuration": "1h", "maxTLSCertDuration": "8760h", "defaultTLSCertDuration": "168h" } } ] }}这样的话默认有效期就是7天 足够了
当然你把”defaultTLSCertDuration”改成比如87600h有效期直接10年 也是可以的
但各种服务定期能续签才能说明ACME工作正常 不然啥时候炸了你都不知道(
文章分享
如果这篇文章对你有帮助,欢迎分享给更多人!