# Cert expiry monitoring for certbot-managed custom domains

> **Created on GitHub as [#848](https://github.com/aimclear/amsoil-dlp/issues/848) on 2026-05-12.** This file is the archived draft used to create that issue.

## Summary

`Dealer.certExpiresAt` is populated when certbot provisions an SSL certificate for a custom domain, but the value is never compared against the current date by any code in the application. There is no cron job that monitors cert expiry, no alert when a cert nears expiration, and `scripts/ssl-manager.sh` has no `renew` command - meaning the app has zero visibility into whether certbot's systemd timer is actually renewing certificates on schedule. Expiring certificates on certbot-managed domains can silently fail to renew, after which Apache continues serving the expired certificate until someone manually notices that a dealer's site is broken.

## Background - current state (audited 2026-05-12)

**Where `certExpiresAt` is written:**
- `app/api/dealer/custom-domain/activate/route.ts` (initial provision via certbot)
- `app/api/admin/custom-domains/[domain]/regenerate/route.ts` (admin manual regenerate)

Both write the value returned from `provisionCertificate()` in `lib/certbot-api.ts`, which shells out to `scripts/ssl-manager.sh provision <domain>`. The shell script extracts the expiry date via `openssl x509 -enddate` and returns it.

**Where `certExpiresAt` is read:**
- `app/api/admin/custom-domains/route.ts:86-92` - calculates `daysUntilExpiry` for display in the admin UI list, but takes no action based on the value.
- Nothing else.

**No comparison to current date exists anywhere in the codebase.** No `findMany({ where: { certExpiresAt: { lt: ... } } })` query exists. No `if (certExpiresAt < now)` branch exists.

**No cron monitors cert expiry:**
- `docs/CRON_JOBS.md` lists existing scheduled tasks (`cleanup-expired-registrations`, `backup-to-remote.sh`) - no cert monitoring is listed.
- `app/api/cron/` contains no cert-related route.
- `.github/workflows/` contains no scheduled cert checks.

**No app-level renewal:**
- `scripts/ssl-manager.sh` exposes `provision`, `revoke`, `status`, `verify-dns` - but **no `renew` command**.
- Renewal implicitly relies on certbot's built-in systemd timer (`certbot.service` / `certbot.timer`), which is installed by default on Debian/Ubuntu via the certbot package - but this assumption is not documented and not verified anywhere in the codebase.

## Blast radius

**All custom domains are certbot-managed** (A-record + Let's Encrypt; there is no Cloudflare-for-SaaS path in use). If the systemd timer fails for any reason - disabled accidentally, package upgrade issue, Let's Encrypt rate-limit hit, DNS validation failure - the certificate expires. Apache continues to serve the expired certificate from `/etc/letsencrypt/live/{domain}/fullchain.pem`. Browsers display a TLS warning. The dealer's site is broken from the user's perspective until someone notices and manually regenerates the cert. Every custom domain is exposed to this gap.

## Partial mitigation already in flight

The Google Search Console indexing initiative ([design spec](../specs/2026-05-12-gsc-indexing-design.md)) adds a preflight check that runs before any sitemap submission or Indexing API ping. The preflight does an HTTPS HEAD against the dealer's canonical URL; Node's `https` module rejects invalid certificates by default, so an expired cert causes the preflight to throw with an SSL error. The worker then:
- Pauses GSC submissions for that dealer
- Auto-logs a `DealerNote` to the Activity & Notes feed
- Surfaces "Last preflight error: SSL certificate expired" in the dealer detail modal

This means **once GSC indexing ships, expired certs for active dealers will surface as an automatically-logged dealer issue within ~24 hours** (next daily-audit cycle). It is not a substitute for proactive monitoring - it's a backstop that catches the symptom after the fact.

## Proposed scope of this issue

1. **Add a cert expiry health-check route**
   - `GET /api/admin/custom-domains/cert-health` (admin-only via `requireAdmin()`)
   - Returns a list of certbot-managed custom domains grouped by expiry buckets: `<30 days`, `<14 days`, `<7 days`, `expired`.

2. **Add a cron route for proactive alerting**
   - `POST /api/cron/cert-expiry-audit` (Bearer `CRON_SECRET`, matching the existing cron pattern in `app/api/cron/cleanup-expired-registrations/route.ts`)
   - Runs daily.
   - Queries `Dealer.certExpiresAt` for all dealers with certbot-managed custom domains.
   - For each dealer crossing an alert threshold (30 / 14 / 7 / 1 days, or already expired): writes a `DealerNote` (`isAutomatic: true`, `noteType: 'internal'`) and triggers a Resend email to the ops alias.
   - Deduplicates alerts (one alert per threshold per dealer, not one per audit run).

3. **Update `scripts/ssl-manager.sh`**
   - Add a `renew` command that calls `certbot renew --quiet` and reports success/failure exit code.
   - Optionally invoke this from a server crontab as an explicit backup to the systemd timer.

4. **Document the renewal mechanism**
   - Update `docs/PRODUCTION_SERVER_SETUP.md` with the assumption that `certbot.timer` is active.
   - Add a verification step to the deployment checklist: `systemctl status certbot.timer` should show enabled and active.
   - Document the new `cert-expiry-audit` cron in `docs/CRON_JOBS.md`.

5. **Tests**
   - Unit test cert-health query logic (expiry buckets).
   - Integration test for the audit route (Bearer auth, dedup, alert thresholds).

## Out of scope

- Replacing Apache with a different reverse proxy. Far separate initiative.
- Dealer-facing notification of impending cert expiry. Could be added later; for now, ops-team alerts are sufficient.

## References

- SSL audit transcript: surfaced during brainstorming for the GSC indexing design spec.
- Files touched in audit: `lib/certbot-api.ts`, `scripts/ssl-manager.sh`, `app/api/admin/custom-domains/route.ts`, `app/api/admin/custom-domains/[domain]/regenerate/route.ts`, `app/api/dealer/custom-domain/activate/route.ts`, `app/api/dealer/custom-domain/route.ts`, `docs/CRON_JOBS.md`, `docs/PRODUCTION_SERVER_SETUP.md`, `docs/CLOUDFLARE_PRODUCTION_DEPLOYMENT.md`.