Daily Engineering Intelligence

Technology Newspaper

← Back to Home

Engineering Intelligence β€” Saturday

October 24, 2026

πŸ”₯ What Changed

A normal Saturday: a Go runtime GC stall that hits kube-apiserver on 192-core amd64, cert-manager closing a private-key copy hole (plus Cilium's out-of-band Go bump), and a big nats.go robustness release. For the weekend: OpenAI's Habitat storage deep dive and Murat's HPTS Days 2–3 notes.

Holds: Kafka 4.4.0 (website PR #928 still open, no announcement) and 4.3.2 (PRs #927/#929 open). Go #82374 x/net/idna (no x/net tag after v0.61.0, no point release yet). grpc-go v1.85 is still unreleased. The Go backports #82189, #82190 and #82297 are still open (#82297 was independently confirmed on 1.26.10). No root-cause analyses yet from GitHub for Oct 21, Oct 22 or the new Oct 23 incidents. Google Cloud's final europe-west2 report is still to come. The AWS European Sovereign Cloud exercise is today and hasn't been reported yet. Also seen: chi XFF spoof GHSA (fixed Oct 12), Go #82460, #82450 and #82455, Fly.io SBD.

Go #82472: on 192-core amd64, heap-triggered GC marks stretch 3–4x because of lfstack contention (kube-apiserver goroutines get stuck in mark assists)

Filed Oct 24, 03:53 IST; labelled compiler/runtime; no CL yet. The reporter runs kube-apiserver on 192-core AMD Genoa/Turin hosts with GOMAXPROCS unset, GOGC=400, 325k goroutines and 116–140 GB live heaps. GC cycles forced by the 2-minute timer took about 5.5 s on average. Cycles triggered by heap growth, which also enlist idle-mark workers on every core, took 14.8–20.6 s, and 18 of 20 marks ran past 9 s. During those marks, allocating goroutines (3–5 GB/s) sat parked in gcParkAssist, stalling etcd watches and the decrypt/decode path. In a 60 s profile, getempty β†’ lfstack.pop β†’ Cas64 took 33% of CPU-seconds, mostly reached from goroutine stack scanning. Porting arm64's existing lfstack backoff (CL 654895) to amd64 fixed it: 0 of 9 heap-triggered cycles stalled, marks took 5.0–7.2 s, and idle-mark CPU fell from a median of about 1,900 to about 430 CPU-s per cycle. A pause value of 8 (about 143 ns on Genoa) worked best. arm64's value of 128 is about 18x longer in wall time on amd64. The issue includes a standalone reproducer (r7a.48xlarge, 500k parked goroutines, 32 GB live heap). Longer term, #81304 would make the procyield argument mean the same thing on every architecture.

Why you care: Big-core nodes with large heaps and hundreds of thousands of goroutines are exactly where control planes and Go data services live. If you see multi-second p99 spikes that line up with heap-triggered GCs on 96+ cores, this is a candidate cause. Until a fix lands, compare mark times of heap-triggered vs timer-forced cycles in your GC traces (GODEBUG=gctrace=1).
Read golang/go#82472 β†’

cert-manager v1.21.3 closes a hole where a status writer could copy any Secret's private key; Cilium ships an out-of-band Go 1.26.10 release

cert-manager v1.21.3 (Oct 24, 02:17 IST). Before this release, a principal that could write the certificates/status subresource but couldn't read Secrets could point status.nextPrivateKeySecretName at any Secret in the namespace and have its private key copied into spec.secretName. The requestmanager and issuing controllers now only use a next-private-key Secret that carries the cert-manager.io/next-private-key label and is owned by the Certificate. The keymanager always generates its own Secret names and records a NotOwned warning event when the field points elsewhere (#9368). The release also fixes an ACME ARI renewal loop: a stale ARI window left over from the previous certificate kept triggering re-issuance until the CA rate limit was hit (#9292). It moves to Go 1.26.10 and drops emSign ECC Root CA - G3 from the distroless images. The notes say "All users should upgrade"; no GHSA has been filed. Cilium 1.20.4 / 1.19.10 / 1.18.16 (Oct 23, 20:21 IST) is an out-of-band security release for CVE-2026-111943 (Go 1.26.10). It also fixes a 1.20.3 regression where the Gateway API implementation mishandled BackendTLSPolicy, causing high Cilium CPU usage and failure to forward traffic. CiliumEnvoyConfig now only redirects Services in its own namespace, as documented (release notes).

Why you care: RBAC on /status subresources is often granted loosely to controllers and CI bots. Here, write access to status was enough to exfiltrate keys within the namespace. Upgrade cert-manager and audit who can write certificates/status. If you're on Cilium 1.20.3 with Gateway API BackendTLSPolicy, upgrade to 1.20.4.
Read the cert-manager v1.21.3 release notes β†’

nats.go v1.55.0: about 30 robustness fixes for deadlocks, leaks and stalls in reconnect, drain and JetStream consumers

Oct 23, 21:14 IST. 47 changes, more than half from 18 external contributors. Fixes include: ForceReconnect() called mid-reconnect ignored ReconnectWait and dialed in a tight loop; a partial write that left a truncated message on the wire closed the connection for good instead of reconnecting; pull Consume() and Messages() deadlocked on Stop() or Drain() while buffered messages were still being delivered; the Messages() iterator leaked goroutines after the connection closed; pull consumers with StopAfter stalled for about 30 s waiting for the missed-heartbeat check; ordered-consumer Fetch() and Next() blocked forever after ErrOrderedConsumerConcurrentRequests, and FetchNoWait() restarted from the beginning of the stream; ordered Consume() used the 15 s pull heartbeat instead of 5 s, so a stall took about 30 s rather than 10 s to reset; PublishAsync() exceeded WithPublishAsyncMaxPending() under concurrent calls; headers weren't counted against pending byte limits, so slow consumers went unreported; and services stopped on async errors from unrelated subscriptions. It also adds FetchMaxBytes(), Conn.RemoteAccount() and ErrStreamMessageExceedsMaximum.

Why you care: If you run NATS/JetStream from Go, several of these are exactly the hangs that show up as "the consumer stopped and nothing logged" during deploys and failovers. It's worth upgrading and re-running your reconnect and drain tests.
Read the nats.go v1.55.0 changelog β†’

πŸ“š Worth Your Time

OpenAI Habitat Part II: running Cosmos DB at 20M QPS β€” from shared to single-tenant containers and back, CDC-invalidated caches, Kafka CDC

Oct 23, 22:30 IST. Habitat serves more than 500 PB across 40 regions; its largest Azure Cosmos DB account holds tens of PB and serves more than 20M QPS. Throughput limits are actually enforced per physical partition, not per logical key, so niche features became noisy neighbours that hurt login and signup. "Weak partitions": each split halves per-partition throughput as storage grows. Moving to per-use-case containers brought launch-day SEVs (1K RU/s in load tests vs 1M at launch). OpenAI went back to a shared container by default with hot-key throttling, QoS request tiers and a new serverless mode built for them (10K RU/s per physical partition, splits driven by RU usage), and reports no provisioning or noisy-neighbour incidents since. A 2024 failover took hours, and the failback caused a gateway connection stampede. For reads (over 90% of traffic): tiny latency-critical datasets like auth metadata are replicated to all 40 regions; a Valkey lookaside cache is invalidated globally within seconds from the Cosmos change feed, TAO-style; and the service was rewritten from Python to Rust. Together these took global read p50 from about 120 ms to about 10 ms. CDC handles millions of changes per second over 100+ Kafka topics, with DLQs against head-of-line blocking and batched checkpoints. Next up is an in-house exabyte-scale store built on RocksDB Cloud and Raft.

Why you care: An honest single-tenant vs multi-tenant decision record at extreme scale, plus a concrete CDC-driven cache-invalidation design that beats picking a TTL. It applies directly to any Kafka-backed backend.
Read OpenAI's post β†’

Murat Demirbas: HPTS 2026 Days 2–3 β€” S3 is a million HDDs, Apple cuts Cassandra storage from 6.2x to 1.6x, retry is still an open problem, and "can it CashDB?"

Oct 23, 22:15 IST (Thursday's Day 1 notes are now followed by Days 2–3). Viktor Leis argues S3 is essentially a million magnetic disks: about 10 ms and 80 MB/s per request, and Iceberg on S3 manages about 1 transaction per second. Caching, log-structuring and large objects are rules inherited from spinning disks, so he makes the case for an SSD-based object store. Scott Andreas (Apple) takes Cassandra's physical-to-logical storage ratio from 6.2x to 4.2x with witness replicas, where mutation tracking replaces Merkle-tree repair, then to 3.6x, 2.4x and finally 1.6x with a satellite data centre. Peter Alvaro explains why retry storms and metastability remain unsolved, and Aleksey Charapko presents the MESSI simulator. Alex Miller tours about 20 quorum and leaderless protocols with the test "can it CashDB?": Accord yes, RIOT probably, EPaxos no. Cloudflare is working to put QuePaxa into production, and SurrealDB is putting TAPIR into production. Also: Tesla's time-series database on S3, Parquet and DuckDB; Hellerstein on commitments, sealing and CALM; and durable execution vs the end-to-end argument.

Why you care: A dense map of where storage and consensus research meets production in 2026. The S3-is-HDD framing and the witness-replica numbers are directly useful when you size replicated stores or object-storage-backed systems.
Read Murat's notes β†’