我们已经解决了K8s内集群的证书 现在需要解决我们直接跑在虚拟机上的一些服务的证书签名了
之前是自己签名 现在发现可以用根证书+ACME统一签名
因为是根证书 实际上可以在Ingress里设置任何一个域名（即使他已经存在）
甚至可以把baidu换成google（
方案选择
Caddy
我们可以使用Caddy的tls internal去简单的给局域网的服务设置反代
我的打算是Caddy在K8s上运行 我们给他单独分配一个LoadBalanceIP
直接贴上yml
---
apiVersion: apps/v1
kind: Deployment
metadata:
annotations: {}
labels:
k8s.kuboard.cn/name: caddy-deployment
name: caddy-deployment
namespace: caddy-namespace
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 3
selector:
matchLabels:
k8s.kuboard.cn/name: caddy-deployment
template:
metadata:
labels:
k8s.kuboard.cn/name: caddy-deployment
spec:
containers:
- image: caddy
imagePullPolicy: IfNotPresent
name: caddy
ports:
- containerPort: 80
name: http
- containerPort: 443
name: https
volumeMounts:
- mountPath: /etc/caddy/Caddyfile
name: volume-piaii
subPath: Caddyfile
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- name: caddy-volume
persistentVolumeClaim:
claimName: caddy-pvc
- configMap:
items:
- key: Caddyfile
path: Caddyfile
name: caddy-config
name: volume-piaii
---
apiVersion: v1
kind: Service
metadata:
annotations:
metallb.io/ip-allocated-from-pool: default-address-pool
labels:
k8s.kuboard.cn/name: caddy-deployment
name: caddy-deployment
namespace: caddy-namespace
spec:
allocateLoadBalancerNodePorts: true
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
nodePort: 31922
port: 80
protocol: TCP
targetPort: 80
- name: https
nodePort: 30408
port: 443
protocol: TCP
targetPort: 443
selector:
k8s.kuboard.cn/name: caddy-deployment
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: 192.168.0.241
ipMode: VIP
---
apiVersion: v1
kind: ConfigMap
metadata:
name: caddy-config
data:
Caddyfile: |
gitlab.lan {
reverse_proxy 192.168.0.37:80
tls internal
}
以上yml创建了一个Deployment 并且设置了ConfigMap以及PVC存储Caddy的数据
我这里用的是Longhorn
然后Caddy的Service被分配了一个LoadbalanceIP
只需要把.lan域名全部指向Caddy的LoadbalanceIP就可以用了
没问题 启动后 Caddy 会在 /data/pki/authorities/local/ 下创建内部 CA 及根证书 信任Caddy的证书直接能用
这个很简单 那问题来了
我觉得K8s内和局域网内我需要信任两个证书（*.k8s.lan appname.lan） 太麻烦了
Caddy的TLS internal主要是给测试用的
有没有更高级 可自定义性更强的方案呢
自建ACME
我在思考 既然Let’s Encrypt无法为lan域名签证书
那我们在局域网自己搭建一个证书签发机构不就可以了
然后让Traefik之类的服务去连接我们自己的证书机构负责签名
Step-Ca搭建
因为这玩意折腾我一天了 我这里不想写折腾过程了 懒
首先的话我们需要在本地有Stepca的CLI
winget install Smallstep.step-ca
当然你在Linux上也是可以的
apt-get update && apt-get install -y --no-install-recommends curl vim gpg ca-certificates
curl -fsSL https://packages.smallstep.com/keys/apt/repo-signing-key.gpg -o /etc/apt/trusted.gpg.d/smallstep.asc && \
echo 'deb [signed-by=/etc/apt/trusted.gpg.d/smallstep.asc] https://packages.smallstep.com/stable/debian debs main' \
| tee /etc/apt/sources.list.d/smallstep.list
apt-get update && apt-get -y install step-cli step-ca
我们有主要是为了生成初始证书
也可以用init pod去生成 但那个太麻烦 我懒得写yml 望周知（（
然后的话我们自己创建一下证书
# 根证书创建
step certificate create \
"Cainong Root CA" \
certs/root_ca.crt \
secrets/root_ca_key \
--profile root-ca \
--no-password \
--insecure \
--not-after 87600h
# 创建中间证书并用根证书签名
step certificate create \
"Cainong Intermediate CA" \
certs/intermediate_ca.crt \
secrets/intermediate_ca_key \
--profile intermediate-ca \
--ca certs/root_ca.crt \
--ca-key secrets/root_ca_key \
--no-password \
--insecure \
--not-after 43800h
step-ca需要根证书 中间证书 中间证书的私钥
也就是说，step-ca实际上是使用的中间证书给域名颁发证书
所以我们需要把/secret/intermediate_ca_key intermediate_ca.crt root_ca.crt这两个文件传到Master节点上
然后执行
kubectl create secret generic step-ca-certs -n step-ca --from-file=root-ca.crt=certs/root_ca.crt --from-file=intermediate_ca.crt=certs/intermediate_ca.crt
这里只放Deployment的YAML了 Service我相信你会的
apiVersion: apps/v1
kind: Deployment
metadata:
annotations: {}
labels:
k8s.kuboard.cn/name: step-ca-server
name: step-ca-server
namespace: step-ca
spec:
selector:
matchLabels:
k8s.kuboard.cn/name: step-ca-server
template:
metadata:
annotations:
labels:
k8s.kuboard.cn/name: step-ca-server
spec:
containers:
- env:
- name: DOCKER_STEPCA_INIT_NAME
value: Smallstep
- name: DOCKER_STEPCA_INIT_REMOTE_MANAGEMENT
value: 'true'
- name: DOCKER_STEPCA_INIT_DNS_NAMES
value: stepca.k8s.lan
- name: CA_PASSWORD
valueFrom:
secretKeyRef:
key: password
name: step-ca-password
image: 'smallstep/step-ca:latest'
imagePullPolicy: IfNotPresent
name: step-ca-server
ports:
- containerPort: 9000
name: stepca
protocol: TCP
volumeMounts:
- mountPath: /home/step/config/ca.json
name: config
subPath: ca.json
- mountPath: /home/step/certs/intermediate_ca.crt
name: intermediate-ca
subPath: intermediate_ca.crt
- mountPath: /home/step/certs/root_ca.crt
name: root-ca
subPath: root_ca.crt
- mountPath: /home/step/secrets/intermediate_ca_key
name: intermediate-ca-key
subPath: intermediate_ca_key
- mountPath: /home/step/secrets/password
name: password
subPath: password
- mountPath: /home/step/db
name: data
securityContext:
runAsGroup: 0
runAsUser: 0
volumes:
- configMap:
defaultMode: 420
name: step-ca-configmap
name: config
- name: intermediate-ca
secret:
defaultMode: 420
items:
- key: intermediate_ca.crt
path: intermediate_ca.crt
secretName: step-ca-certs
- name: root-ca
secret:
defaultMode: 420
items:
- key: root-ca.crt
path: root_ca.crt
secretName: step-ca-certs
- name: intermediate-ca-key
secret:
defaultMode: 420
items:
- key: intermediate_ca_key
path: intermediate_ca_key
secretName: step-ca-password
- name: password
secret:
defaultMode: 420
items:
- key: password
path: password
secretName: step-ca-password
- name: data
persistentVolumeClaim:
claimName: step-ca-pvc
不难看到我是用kuboard配置的 手写YAML真要命的
然后是configmap
apiVersion: v1
data:
ca.json: |2-
{
"root": "/home/step/certs/root_ca.crt",
"crt": "/home/step/certs/intermediate_ca.crt",
"key": "/home/step/secrets/intermediate_ca_key",
"address": ":9000",
"dnsNames": ["step-ca-server.step-ca.svc.cluster.local", "ca.k8s.lan"],
"logger": {
"format": "text"
},
"db": {
"type": "badger",
"dataSource": "/home/step/db"
},
"authority": {
"provisioners": [
{
"type": "ACME",
"name": "acme",
"forceCN": true
}
]
}
}
kind: ConfigMap
metadata:
name: step-ca-configmap
namespace: step-ca
不出意外step-ca 就能跑起来了
但现在又有个小问题
Ingress与ACME的Service之间通信是用HTTP 这是不允许的
并且ACME要求全程都使用https
也就是说
客户端->Ingress->Service->ACME Pod全部都要用https
也就是说 我们需要先给Ingress申请一个证书 我给ca服务的域名是ca.k8s.lan
回到我们一开始申请证书的地方，我们用根证书先给ca.k8s.lan申请一个先
step certificate create "ca.k8s.lan" ca.k8s.lan.crt ca.k8s.lan.key \
--ca-key secrets/root_ca_key \
--ca certs/root_ca.crt \
--san ca.k8s.lan \
--not-after 87600h \
--insecure --no-password
然后把这个证书传到集群内作为TLS Secret 给Ingress装上
还没完 我们需要解决Ingress解密Https请求 我们让Ingress不解密 直接传给Pod
我们需要给Ingres加上这两个设置
annotations:
nginx.ingress.kubernetes.io/ssl-passthrough: "true"
nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
然后不出意外的话就可以在局域网内给证书签名了 值得一提的是申请证书的设备也要信任我们的根证书才可以
Certmanager
在K8s集群内最好的方式应该还是用Certmanager去负责Ingress的自签名，创建好Issuer之后只需要在Ingress配置里加个标签 Certmanager就会自动帮我们完成http挑战+证书申请+部署
还是很方便的
在Master节点上执行
helm install cert-manager oci://quay.io/jetstack/charts/cert-manager --version v1.19.0 --namespace cert-manager --create-namespace --set crds.enabled=true
然后创建一个Cluster Issuer
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: step-ca
spec:
acme:
server: https://ca.k8s.lan/acme/acme/directory #ACME服务器地址
email: admin@k8s.lan
spec:
acme:
privateKeySecretRef:
name: my-acme-account-key
server: https://ca.k8s.lan/acme/acme/directory
solvers:
- http01:
ingress:
class: nginx-ingress # Ingress类名 负责http挑战
还没完 因为我们是自己创的证书 所以我们需要让Certmanager也信任根证书才行
简单来说就是把root cert挂载到Certmanager的/etc/ssl/certs/里
还没完 挂了之后重启Pod还是不认 我们还需要添加环境变量SSL_CERT_FILE=/etc/ssl/certs/root_ca.crt
以上都完成之后就可以创建一个Ingress试试了
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
cert-manager.io/cluster-issuer: my-acme-issuer # 这里要改
labels:
app: longhorn-ui
name: longhorn-ui
namespace: longhorn-system
spec:
ingressClassName: nginx-ingress
rules:
- host: longhorn.k8s.lan
http:
paths:
- backend:
service:
name: longhorn-ui
port:
number: 8000
path: /
pathType: Prefix
tls:
- hosts:
- longhorn.k8s.lan # 这里改成你需要的域名
secretName: longhorn-tls-secret # SSL的Secret 名称随意
只需要添加cert-manager.io/cluster-issuer这一行 他就会自动完成证书申请+挑战+证书部署
还是很强大的
至此局域网内的证书基本就完善了 应该没啥要折腾的了
后续一个小更新：
step-ca默认证书有效期只有24小时 有点太短了其实
我们去修改一下他的configmap
{
"root": "/home/step/certs/root_ca.crt",
"crt": "/home/step/certs/intermediate_ca.crt",
"key": "/home/step/secrets/intermediate_ca_key",
"address": ":9000",
"dnsNames": [
"step-ca-server.step-ca.svc.cluster.local",
"ca.k8s.lan"
],
"logger": {
"format": "text"
},
"db": {
"type": "badger",
"dataSource": "/home/step/db"
},
"authority": {
"provisioners": [
{
"type": "ACME",
"name": "acme",
"forceCN": true,
"claims": {
"minTLSCertDuration": "1h",
"maxTLSCertDuration": "8760h",
"defaultTLSCertDuration": "168h"
}
}
]
}
}
这样的话默认有效期就是7天 足够了
当然你把”defaultTLSCertDuration”改成比如87600h有效期直接10年 也是可以的
但各种服务定期能续签才能说明ACME工作正常 不然啥时候炸了你都不知道（