Daily Engineering Intelligence

Technology Newspaper

← Back to Home

Engineering Intelligence β€” Wednesday

October 21, 2026

πŸ”₯ What Changed

A light Wednesday, led by the story behind yesterday's urgent Go release: eight eMudhra intermediate CA keys anyone could recover, so patching trust stores beyond Go matters too. Also SpiceDB v1.57.0's backend-correctness fixes and Datadog's deploy-time supply-chain policy engine.

Holds: Google Cloud's europe-west2-b power event (Oct 20, 11:38–19:30 IST; services across the region saw errors; Google has promised an analysis). Go x/net/idna Punycode CPU DoS #82374 (CVE-2026-97044, public track; the x/net CL is still under review and not in a tagged release). Kafka 4.4.0 still has no announcement (site PR #925 open). grpc-go v1.85 is still unreleased. The Go #81957 GC backports (#82189 β†’ 1.26.11, #82190 β†’ 1.27.4) and #82297 are still open. #82177, #82026, #81759 and #82272 are unchanged. Still waiting on GitHub's root-cause analysis for Oct 19 and on the AWS European Sovereign Cloud exercise (Sat Oct 24). Also seen: Go #82365 (http2 GOAWAY state), DeepSeek DeepInfer, PyMuPDF 2.0.

🚨 The CA behind Go 1.27.3: eight eMudhra intermediate keys were trivially recoverable β€” patch trust stores beyond Go too

Yesterday's urgent Go release distrusted eight unnamed intermediates. They're now public: eMudhra's EM DV / OV / EV TLS CA - G3B and G3C (six TLS CAs) plus two code-signing CAs, all from one April 30, 2025 key ceremony. Mozilla's Dennis Jackson found that the eight P-384 private keys were integers below 256, so anyone could recover them from the public certificates almost instantly. A researcher reported this on Oct 17 with signed proof for all eight keys. Chrome blocked the keys via CRLSet that evening and Firefox via OneCRL; Apple followed on Oct 19 and Microsoft on Oct 20. eMudhra revoked the intermediates on Oct 18 (reason cACompromise) and Sectigo revoked its cross-signs on Oct 19. No TLS certificate from these CAs appears in Certificate Transparency logs, so clients that enforce CT (Chrome, Firefox, Safari, Edge) couldn't have been attacked. For clients that don't check CT, nobody knows whether the keys were abused. eMudhra says one of the intermediates was in production and none issued end-entity certificates. Chrome removes emSign ECC Root CA - G3 from its root store around Oct 27 (PDT), and NSS 3.131.1 removes it as well. On the Go issue, Roland Shoemaker explained that the release notes were deliberately vague so attackers wouldn't get a head start. Filippo Valsorda added that Linux trust stores have no way to propagate revocation, so the only option was hard-coding the keys.

Why you care: Impact: any TLS client that isn't a browser and still trusts these chains can be impersonated (MITM), and that includes service-to-service calls, webhooks and OIDC/JWKS fetches. Who is affected: Go binaries not rebuilt on 1.27.3 / 1.26.10, and every runtime or container that ships its own CA bundle. Patched so far: Debian, Ubuntu, Alpine and Fedora ca-certificates, certifi 2026.10.19, rustls-webpki 0.103.16 and cloudflared 2026.10.2; Node.js and Bun are still pending. Action: rebuild Go services on the fixed release and update the ca-certificates package in your base images (Go 1.25 and older depend on the system pool). Check the CA bundles you use to verify mTLS client certificates. Test the path each application actually uses (the incident site has test hosts; a passing curl doesn't prove your service is safe). If new certificate errors appear, don't fix them by turning off verification. Links: tracker + patched packages Β· Google Β· Cloudflare Β· go.dev/issue/82345.
Read Mozilla's incident bug 2083449 β†’

SpiceDB v1.57.0: resumable bulk deletes, a pgx context bug that broke healthy connections, and drain-before-shutdown

Released Oct 21, 00:43 IST. New: an experimental full relation cache serves a hot object#relation from memory, empty answers included. DeleteRelationships can now be resumed with a cursor: on CockroachDB each batch continues from a primary-key cursor, so batches stop rescanning the MVCC tombstones earlier batches left behind. A new spicedb datastore delete-relationships CLI requires every filter flag, using <any> explicitly, so a forgotten flag can't widen a delete. Fixes worth learning from: the Postgres pool warmed up on a context that was cancelled as soon as startup returned. pgx reacts to a cancelled context by setting an immediate deadline on the connection, and those connections still entered the pool, so queries failed with i/o timeout against a healthy database. Retry backoff grew without limit, so 15 retries meant 14 minutes; it's now capped at 10 s. On SIGTERM, SpiceDB now reports NOT_SERVING on its gRPC health check and holds the listener open before draining. --dispatch-upstream-timeout had been ignored (it was hardcoded to 60 s). Cancelled LookupResources/LookupSubjects streams returned success with incomplete results, and those results could be cached. When a memory limit couldn't be read (as on ECS), the cache fallback was 256 KiB instead of 256 MiB. Postgres limited deletes now locate rows by ctid.

Why you care: SpiceDB is on the board under Auth/IAM, and Zanzibar-style authorization is a growing ask in job listings. Even if you never run it, the fixes are general Go backend lessons: never hand a background goroutine a context you're about to cancel, put a cap on exponential backoff, treat a cancelled stream as an error and never as a partial success, and fail readiness checks before you drain on Kubernetes. One caveat: the release is built with Go 1.26.9 (go.mod), which predates the x509 fix in item 1.
Read the SpiceDB v1.57.0 release notes β†’

πŸ“š Worth Your Time

Datadog Turnstile: checking signed supply-chain evidence for every deploy

Oct 20. An image signature only proves the image hasn't changed since it was signed. It says nothing about a malicious commit, a poisoned dependency or a compromised build runner. Turnstile records each stage as a signed in-toto attestation (commit safety, PR approval, build provenance, SBOM, vulnerabilities, deployment), and each attestation is keyed to the next stage. Attestations are signed keylessly with short-lived keys bound to an OIDC workload identity (a CI job, a Kubernetes service account), so trust follows identities instead of long-lived keys, and everything a compromised producer signed can be invalidated. OPA/Rego policies are scoped per environment and roll out from audit to warn to block. Each decision is stored as a signed SLSA Verification Summary Attestation that other systems can reuse. The policies themselves ship as TUF-signed OCI artifacts. The data lives in DynamoDB with a per-process LRU cache in front. Across about 1M evaluations in August: 92.8% policy cache hit rate, p50 36 ms, p99 481 ms. Caveat: it's an internal system, and no code is published.

Why you care: A strong Auth/IAM-meets-platform design: workload identity, policy as code, signed decisions, and a deploy-time check that has to stay fast. The cache key design (artifact digest + policy ID + binary versions + environment) is also a reusable pattern for any authorization service that sits in the request path.
Read on the Datadog engineering blog β†’