通过读取cert-manager指标和K8s Secret定位濒临过期的证书,由LLM推理失效原因并自动触发续期或提PR通知责任人。
TLS 证书过期 Agent 构建指南:在故障前捕获 cert-manager 续期失败、未管理的 Secret 和过期证书
cert-manager 虽然已安装,但证书依然会过期。cert-manager 默认在证书生命周期的三分之二处续期,所以一张 90 天有效期的 Let's Encrypt 证书,会在过期前 30 天进入续期窗口。这段时间内续期可能正在静默失败。失败可分为五类,每类有不同的修复方式。
前两类在 cert-manager 自身对象中可见。后两类则不可见,而它们恰恰是你收到告警的原因——因为没有任何东西在监控它们。Ingress 后端上的过期证书与上游问题表现相同,所以如果你在边缘追逐间歇性错误,ingress-nginx 502 指南涵盖了非 TLS 原因,而这个 Agent 则涵盖 TLS 相关的。
三个来源,按 Secret 名称合并。第一个是 cert-manager 的指标,只读的 Prometheus MCP server 可以回答:
# Certificates expiring within 14 days
(certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 14
# Certificates cert-manager itself says are not ready
certmanager_certificate_ready_status{condition="False"} == 1
指标只能覆盖 cert-manager 管理的证书,所以第二个来源遍历所有 TLS Secret 并在本地解析叶子证书:
# inventory.py — deterministic, read-only, keys never leave this process
import base64, ssl, socket
from datetime import datetime, timezone
from cryptography import x509
from kubernetes import client, config
config.load_incluster_config()
core, net = client.CoreV1Api(), client.NetworkingV1Api()
def parse_leaf(pem: bytes) -> x509.Certificate:
return x509.load_pem_x509_certificate(pem.split(b"-----END CERTIFICATE-----")[0]
+ b"-----END CERTIFICATE-----\n")
def secret_inventory() -> dict:
out = {}
for s in core.list_secret_for_all_namespaces(field_selector="type=kubernetes.io/tls").items:
crt = s.data.get("tls.crt")
if not crt:
continue
leaf = parse_leaf(base64.b64decode(crt))
ann = s.metadata.annotations or {}
key = f"{s.metadata.namespace}/{s.metadata.name}"
out[key] = {
"secret": key,
"managed_by": ann.get("cert-manager.io/certificate-name"), # None = unmanaged
"issuer": ann.get("cert-manager.io/issuer-name"),
"not_after": leaf.not_valid_after_utc.isoformat(),
"days_left": (leaf.not_valid_after_utc - datetime.now(timezone.utc)).days,
"serial": format(leaf.serial_number, "x"),
"sans": [n.value for n in leaf.extensions.get_extension_for_class(
x509.SubjectAlternativeName).value],
}
return out
故意从不读取 tls.key 字段。Agent 的 ServiceAccount 需要对 Secrets 的 get 和 list 权限才能完成这些操作,这是本系列中最敏感的权限,所以 secrets-management 文章完整适用:模型收到的是上面的字典,绝不是 Secret 对象本身。
第三个来源连接到每个 Ingress host,比较所服务证书的序列号与存储的序列号。这是捕获第四类的唯一方式:
def served_serial(host: str, port: int = 443) -> str | None:
ctx = ssl.create_default_context()
ctx.check_hostname, ctx.verify_mode = False, ssl.CERT_NONE # we want the cert, not validation
try:
with socket.create_connection((host, port), timeout=5) as sock:
with ctx.wrap_socket(sock, server_hostname=host) as tls:
der = tls.getpeercert(binary_form=True)
return format(x509.load_der_x509_certificate(der).serial_number, "x")
except (OSError, ssl.SSLError):
return None
def ingress_hosts() -> dict:
hosts = {}
for ing in net.list_ingress_for_all_namespaces().items:
for tls in ing.spec.tls or []:
for h in tls.hosts or []:
hosts[h] = f"{ing.metadata.namespace}/{tls.secret_name}"
return hosts
合并三个来源得到每证书记录:剩余天数、cert-manager 是否管理它、cert-manager 是否认为它就绪、以及所服务证书与存储证书是否匹配。剩余天数超过 21 天且所服务序列号匹配的都会被丢弃。剩下的就是短列表,在大多数集群上它不超过十个条目。
对于短列表中 cert-manager 管理的条目,封装器还会拉取解释失败原因的对象链,因为原因字符串位于其底部:
kubectl get certificate,certificaterequest,order,challenge -n shop -o wide
NAME READY SECRET AGE
certificate.cert-manager.io/shop-tls False shop-tls 88d
NAME STATE DOMAIN REASON
challenge.acme.cert-manager.io/shop-tls-... pending shop.example Waiting for HTTP-01 challenge propagation: wrong status code '404', expected '200'
这个原因字符串是模型收到的最有价值的单一输入,封装器原样传递它。
每张候选证书一次模型调用,强制使用工具 schema 以获得结构化答案:
CERT_TOOL = {
"name": "diagnose_certificate",
"description": "Explain why one certificate has not renewed and choose a remediation.",
"input_schema": {
"type": "object",
"properties": {
"diagnosis": {"enum": [
"acme_http01_unreachable", "acme_dns01_not_propagated",
"acme_rate_limited", "issuer_not_ready",
"unmanaged_secret_expiring", "renewed_not_reloaded",
"served_cert_is_external", "unknown"]},
"action": {"enum": [
"retrigger_renewal", "rollout_restart",
"propose_ingress_fix_pr", "propose_adopt_cert_manager_pr",
"wait_for_rate_limit", "ask_owner", "report_only"]},
"workload": {"type": "string",
"description": "namespace/kind/name to restart. Only for rollout_restart."},
"evidence": {"type": "string",
"description": "2-3 sentences quoting the challenge or order reason, "
"the days-left figure, and the served-vs-stored serial."},
"risk": {"type": "string",
"description": "What breaks if this action is wrong, and why the "
"safer alternative was not chosen."},
},
"required": ["diagnosis", "action", "evidence", "risk"],
},
}
SYSTEM = (
"You diagnose TLS certificate renewals for an SRE team. A 404 or connection "
"refused on an HTTP-01 challenge is an Ingress routing problem: the "
"/.well-known/acme-challenge/ path is not reaching the cert-manager solver "
"pod. Choose propose_ingress_fix_pr. A DNS-01 'not yet propagated' reason "
"under two hours old is normal; choose report_only. A rateLimited order is "
"never fixed by retrying: choose wait_for_rate_limit and say when the window "
"clears. If the Secret is fresh but the served serial is old, the pod read the "
"cert at startup: choose rollout_restart and name the workload. An unmanaged "
"Secret is always ask_owner or propose_adopt_cert_manager_pr, never "
"retrigger_renewal. If the served certificate is not in any Secret at all, "
"TLS terminates outside the cluster: served_cert_is_external, report_only. "
"Use retrigger_renewal only when the Certificate is Ready=True, inside its "
"renewal window, and no Order exists for it."
)
Prompt 中最后一条规则很重要。cmctl renew 是诱人的万能锤子,但对于有失败 challenge 的证书它什么都不做,只会创建另一个失败的 Order,而这会计入 Let's Encrypt 每个主机名每小时五次失败验证的限制。Agent 只在唯一有帮助的情况下才能挥动它:cert-manager simply simply 还没来得及续期,通常是因为控制器在此周期内宕机或重启了。
RBAC 按最小权限 ServiceAccount 文章所述方式构建,将 Agent 不能做什么编码进去,而不是信任 Prompt:
rules:
- apiGroups: [""]
resources: ["secrets"]
verbs: ["get", "list"] # parse tls.crt; tls.key is never read
- apiGroups: ["networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get", "list"]
- apiGroups: ["cert-manager.io"]
resources: ["certificates", "certificaterequests", "issuers", "clusterissuers"]
verbs: ["get", "list"]
- apiGroups: ["cert-manager.io"]
resources: ["certificates/status"]
verbs: ["update"] # what cmctl renew needs, nothing more
- apiGroups: ["acme.cert-manager.io"]
resources: ["orders", "challenges"]
verbs: ["get", "list"]
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets", "daemonsets"]
verbs: ["get", "patch"] # rollout restart annotation only
对 Secrets 没有 create 或 delete 权限,所以 Agent 不能手工粘贴证书进去,也不能删除 cert-manager 会重新生成的证书。对 certificates、issuers 或 ingresses 没有写权限,所以所有配置变更都通过 GitOps-for-agents 路径变成 Pull Request。
执行器增加了三条 RBAC 无法表达的规则:
rollout_restart 受剩余天数限制。低于 7 天时自动运行,因为凌晨 3 点过期的过期证书比下午 3 点受控重启更糟糕。超过 7 天则提交审批,两个序列号并列展示。
retrigger_renewal 每张证书每 24 小时最多一次,在 Agent 自身状态中跟踪。这是 Prompt 无法保证的限速保护。
所有变更都挂在熔断器后面。在无关事故期间重启四个 Deployment 的证书 Agent 使事故更糟,所以 SLO 燃烧门限在这里同样适用。
重启本身与 kubectl rollout restart 发送的补丁相同,只在执行器在行动前立即第二次重新验证序列号不匹配时才应用:
def rollout_restart(ns, kind, name, expected_secret_serial, host):
if served_serial(host) == expected_secret_serial:
return "skipped: served cert already current"
body = {"spec": {"template": {"metadata": {"annotations": {
"certagent.devtocash.com/restartedAt": datetime.now(timezone.utc).isoformat()}}}}}
getattr(apps, f"patch_namespaced_{kind}")(name, ns, body)
一个 40 个命名空间的 EKS 集群,61 个 TLS Secret,其中 38 个由 cert-manager 管理。资产清单产生了六个候选:
shop/shop-tls managed days_left=19 ready=False acme_http01_unreachable -> propose_ingress_fix_pr
api/api-tls managed days_left=27 ready=True renewed_not_reloaded -> rollout_restart (approval)
legacy/portal-tls unmanaged days_left=9 ready=n/a unmanaged_secret_expiring -> ask_owner
admin/grafana-tls managed days_left=3 ready=False acme_rate_limited -> wait_for_rate_limit
edge/www-tls managed days_left=41 ready=True served_cert_is_external -> report_only
ml/jupyter-tls managed days_left=6 ready=True renewed_not_reloaded -> rollout_restart (auto)
shop 的失败是因为一个新的基于路径的 Ingress 规则,用 404 遮盖了 storefront 的 ACME challenge 路径;Agent 开的 PR 把 /.well-known/acme-challenge/ 路由加回到它上方。grafana 证书那一周被 Helm 升级重新创建了六次,每次升级一个 Order,触发了重复证书限制;诚实的答案是等两天,Agent 给出了日期。legacy portal 原来是 2024 年承包商手工粘贴进去的 Secret,有效期两年,直到 Agent 询问才有人知道。www 证书在集群前面的 CloudFront 发行版上,从未在范围内。两次重启、一个 PR、一条 Slack 消息、一次等待,没有证书过期。
所服务序列号检查只对 Agent 能从集群内部通过 443 访问的 host 有效,也只对解析到用户访问相同位置的 host 有效。split-horizon DNS 或内部专用 Ingress class 会悄无声息地将这些 host 从第四类检测中移除,你应该把不可达 host 的数量作为独立指标记录。
未管理 Secret 扫描告诉你证书即将过期,但不告诉你是谁负责。没有服务目录将命名空间解析到团队,ask_owner 退化为共享频道的一条消息,而证书过期通知历来都是这样石沉大海的。
第五类基本上无法触及。Webhook caBundles、kubelet serving 证书和服务网格 CA 根证书各自按自己的时间表过期,而升级就绪 Agent 是 webhook 检查的更好场所,因为那才是它咬人的时候。这个 Agent 在无聊的大多数上赚取它的价值:那些 cert-manager 应该续期却静默失败的证书。