用LLM Agent自动化TLS证书过期巡检

TLS 证书过期 Agent 构建指南：在故障前捕获 cert-manager 续期失败、未管理的 Secret 和过期证书

cert-manager 虽然已安装，但证书依然会过期。cert-manager 默认在证书生命周期的三分之二处续期，所以一张 90 天有效期的 Let's Encrypt 证书，会在过期前 30 天进入续期窗口。这段时间内续期可能正在静默失败。失败可分为五类，每类有不同的修复方式。

前两类在 cert-manager 自身对象中可见。后两类则不可见，而它们恰恰是你收到告警的原因——因为没有任何东西在监控它们。Ingress 后端上的过期证书与上游问题表现相同，所以如果你在边缘追逐间歇性错误，ingress-nginx 502 指南涵盖了非 TLS 原因，而这个 Agent 则涵盖 TLS 相关的。

第一步：资产清单是代码，而非 Prompt

三个来源，按 Secret 名称合并。第一个是 cert-manager 的指标，只读的 Prometheus MCP server 可以回答：

# Certificates expiring within 14 days
(certmanager_certificate_expiration_timestamp_seconds - time()) / 86400 < 14

# Certificates cert-manager itself says are not ready
certmanager_certificate_ready_status{condition="False"} == 1

指标只能覆盖 cert-manager 管理的证书，所以第二个来源遍历所有 TLS Secret 并在本地解析叶子证书：

# inventory.py — deterministic, read-only, keys never leave this process
import base64, ssl, socket
from datetime import datetime, timezone
from cryptography import x509
from kubernetes import client, config

config.load_incluster_config()
core, net = client.CoreV1Api(), client.NetworkingV1Api()

def parse_leaf(pem: bytes) -> x509.Certificate:
    return x509.load_pem_x509_certificate(pem.split(b"-----END CERTIFICATE-----")[0]
                                          + b"-----END CERTIFICATE-----\n")

def secret_inventory() -> dict:
    out = {}
    for s in core.list_secret_for_all_namespaces(field_selector="type=kubernetes.io/tls").items:
        crt = s.data.get("tls.crt")
        if not crt:
            continue
        leaf = parse_leaf(base64.b64decode(crt))
        ann = s.metadata.annotations or {}
        key = f"{s.metadata.namespace}/{s.metadata.name}"
        out[key] = {
            "secret": key,
            "managed_by": ann.get("cert-manager.io/certificate-name"),   # None = unmanaged
            "issuer": ann.get("cert-manager.io/issuer-name"),
            "not_after": leaf.not_valid_after_utc.isoformat(),
            "days_left": (leaf.not_valid_after_utc - datetime.now(timezone.utc)).days,
            "serial": format(leaf.serial_number, "x"),
            "sans": [n.value for n in leaf.extensions.get_extension_for_class(
                x509.SubjectAlternativeName).value],
        }
    return out

故意从不读取 tls.key 字段。Agent 的 ServiceAccount 需要对 Secrets 的 get 和 list 权限才能完成这些操作，这是本系列中最敏感的权限，所以 secrets-management 文章完整适用：模型收到的是上面的字典，绝不是 Secret 对象本身。

第三个来源连接到每个 Ingress host，比较所服务证书的序列号与存储的序列号。这是捕获第四类的唯一方式：

def served_serial(host: str, port: int = 443) -> str | None:
    ctx = ssl.create_default_context()
    ctx.check_hostname, ctx.verify_mode = False, ssl.CERT_NONE   # we want the cert, not validation
    try:
        with socket.create_connection((host, port), timeout=5) as sock:
            with ctx.wrap_socket(sock, server_hostname=host) as tls:
                der = tls.getpeercert(binary_form=True)
        return format(x509.load_der_x509_certificate(der).serial_number, "x")
    except (OSError, ssl.SSLError):
        return None

def ingress_hosts() -> dict:
    hosts = {}
    for ing in net.list_ingress_for_all_namespaces().items:
        for tls in ing.spec.tls or []:
            for h in tls.hosts or []:
                hosts[h] = f"{ing.metadata.namespace}/{tls.secret_name}"
    return hosts

合并三个来源得到每证书记录：剩余天数、cert-manager 是否管理它、cert-manager 是否认为它就绪、以及所服务证书与存储证书是否匹配。剩余天数超过 21 天且所服务序列号匹配的都会被丢弃。剩下的就是短列表，在大多数集群上它不超过十个条目。

对于短列表中 cert-manager 管理的条目，封装器还会拉取解释失败原因的对象链，因为原因字符串位于其底部：

kubectl get certificate,certificaterequest,order,challenge -n shop -o wide

NAME                                  READY   SECRET          AGE
certificate.cert-manager.io/shop-tls  False   shop-tls        88d

NAME                                          STATE     DOMAIN         REASON
challenge.acme.cert-manager.io/shop-tls-...   pending   shop.example   Waiting for HTTP-01 challenge propagation: wrong status code '404', expected '200'

这个原因字符串是模型收到的最有价值的单一输入，封装器原样传递它。

第二步：诊断调用

每张候选证书一次模型调用，强制使用工具 schema 以获得结构化答案：

CERT_TOOL = {
    "name": "diagnose_certificate",
    "description": "Explain why one certificate has not renewed and choose a remediation.",
    "input_schema": {
        "type": "object",
        "properties": {
            "diagnosis": {"enum": [
                "acme_http01_unreachable", "acme_dns01_not_propagated",
                "acme_rate_limited", "issuer_not_ready",
                "unmanaged_secret_expiring", "renewed_not_reloaded",
                "served_cert_is_external", "unknown"]},
            "action": {"enum": [
                "retrigger_renewal", "rollout_restart",
                "propose_ingress_fix_pr", "propose_adopt_cert_manager_pr",
                "wait_for_rate_limit", "ask_owner", "report_only"]},
            "workload": {"type": "string",
                "description": "namespace/kind/name to restart. Only for rollout_restart."},
            "evidence": {"type": "string",
                "description": "2-3 sentences quoting the challenge or order reason, "
                               "the days-left figure, and the served-vs-stored serial."},
            "risk": {"type": "string",
                "description": "What breaks if this action is wrong, and why the "
                               "safer alternative was not chosen."},
        },
        "required": ["diagnosis", "action", "evidence", "risk"],
    },
}

SYSTEM = (
    "You diagnose TLS certificate renewals for an SRE team. A 404 or connection "
    "refused on an HTTP-01 challenge is an Ingress routing problem: the "
    "/.well-known/acme-challenge/ path is not reaching the cert-manager solver "
    "pod. Choose propose_ingress_fix_pr. A DNS-01 'not yet propagated' reason "
    "under two hours old is normal; choose report_only. A rateLimited order is "
    "never fixed by retrying: choose wait_for_rate_limit and say when the window "
    "clears. If the Secret is fresh but the served serial is old, the pod read the "
    "cert at startup: choose rollout_restart and name the workload. An unmanaged "
    "Secret is always ask_owner or propose_adopt_cert_manager_pr, never "
    "retrigger_renewal. If the served certificate is not in any Secret at all, "
    "TLS terminates outside the cluster: served_cert_is_external, report_only. "
    "Use retrigger_renewal only when the Certificate is Ready=True, inside its "
    "renewal window, and no Order exists for it."
)

Prompt 中最后一条规则很重要。cmctl renew 是诱人的万能锤子，但对于有失败 challenge 的证书它什么都不做，只会创建另一个失败的 Order，而这会计入 Let's Encrypt 每个主机名每小时五次失败验证的限制。Agent 只在唯一有帮助的情况下才能挥动它：cert-manager simply simply 还没来得及续期，通常是因为控制器在此周期内宕机或重启了。

第三步：执行与硬限制

RBAC 按最小权限 ServiceAccount 文章所述方式构建，将 Agent 不能做什么编码进去，而不是信任 Prompt：

rules:
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]                        # parse tls.crt; tls.key is never read
  - apiGroups: ["networking.k8s.io"]
    resources: ["ingresses"]
    verbs: ["get", "list"]
  - apiGroups: ["cert-manager.io"]
    resources: ["certificates", "certificaterequests", "issuers", "clusterissuers"]
    verbs: ["get", "list"]
  - apiGroups: ["cert-manager.io"]
    resources: ["certificates/status"]
    verbs: ["update"]                             # what cmctl renew needs, nothing more
  - apiGroups: ["acme.cert-manager.io"]
    resources: ["orders", "challenges"]
    verbs: ["get", "list"]
  - apiGroups: ["apps"]
    resources: ["deployments", "statefulsets", "daemonsets"]
    verbs: ["get", "patch"]                       # rollout restart annotation only

对 Secrets 没有 create 或 delete 权限，所以 Agent 不能手工粘贴证书进去，也不能删除 cert-manager 会重新生成的证书。对 certificates、issuers 或 ingresses 没有写权限，所以所有配置变更都通过 GitOps-for-agents 路径变成 Pull Request。

执行器增加了三条 RBAC 无法表达的规则：

rollout_restart 受剩余天数限制。低于 7 天时自动运行，因为凌晨 3 点过期的过期证书比下午 3 点受控重启更糟糕。超过 7 天则提交审批，两个序列号并列展示。

retrigger_renewal 每张证书每 24 小时最多一次，在 Agent 自身状态中跟踪。这是 Prompt 无法保证的限速保护。

所有变更都挂在熔断器后面。在无关事故期间重启四个 Deployment 的证书 Agent 使事故更糟，所以 SLO 燃烧门限在这里同样适用。

重启本身与 kubectl rollout restart 发送的补丁相同，只在执行器在行动前立即第二次重新验证序列号不匹配时才应用：

def rollout_restart(ns, kind, name, expected_secret_serial, host):
    if served_serial(host) == expected_secret_serial:
        return "skipped: served cert already current"
    body = {"spec": {"template": {"metadata": {"annotations": {
        "certagent.devtocash.com/restartedAt": datetime.now(timezone.utc).isoformat()}}}}}
    getattr(apps, f"patch_namespaced_{kind}")(name, ns, body)

一个 40 个命名空间的 EKS 集群，61 个 TLS Secret，其中 38 个由 cert-manager 管理。资产清单产生了六个候选：

shop/shop-tls          managed   days_left=19  ready=False  acme_http01_unreachable      -> propose_ingress_fix_pr
api/api-tls            managed   days_left=27  ready=True   renewed_not_reloaded          -> rollout_restart (approval)
legacy/portal-tls      unmanaged days_left=9   ready=n/a    unmanaged_secret_expiring     -> ask_owner
admin/grafana-tls      managed   days_left=3   ready=False  acme_rate_limited             -> wait_for_rate_limit
edge/www-tls           managed   days_left=41  ready=True   served_cert_is_external       -> report_only
ml/jupyter-tls         managed   days_left=6   ready=True   renewed_not_reloaded          -> rollout_restart (auto)

shop 的失败是因为一个新的基于路径的 Ingress 规则，用 404 遮盖了 storefront 的 ACME challenge 路径；Agent 开的 PR 把 /.well-known/acme-challenge/ 路由加回到它上方。grafana 证书那一周被 Helm 升级重新创建了六次，每次升级一个 Order，触发了重复证书限制；诚实的答案是等两天，Agent 给出了日期。legacy portal 原来是 2024 年承包商手工粘贴进去的 Secret，有效期两年，直到 Agent 询问才有人知道。www 证书在集群前面的 CloudFront 发行版上，从未在范围内。两次重启、一个 PR、一条 Slack 消息、一次等待，没有证书过期。

所服务序列号检查只对 Agent 能从集群内部通过 443 访问的 host 有效，也只对解析到用户访问相同位置的 host 有效。split-horizon DNS 或内部专用 Ingress class 会悄无声息地将这些 host 从第四类检测中移除，你应该把不可达 host 的数量作为独立指标记录。

未管理 Secret 扫描告诉你证书即将过期，但不告诉你是谁负责。没有服务目录将命名空间解析到团队，ask_owner 退化为共享频道的一条消息，而证书过期通知历来都是这样石沉大海的。

第五类基本上无法触及。Webhook caBundles、kubelet serving 证书和服务网格 CA 根证书各自按自己的时间表过期，而升级就绪 Agent 是 webhook 检查的更好场所，因为那才是它咬人的时候。这个 Agent 在无聊的大多数上赚取它的价值：那些 cert-manager 应该续期却静默失败的证书。