Daily Engineering Intelligence

Technology Newspaper

← Back to Home

Engineering Intelligence β€” Thursday

October 22, 2026

πŸ”₯ What Changed

A normal Thursday, led by a live supply-chain incident: LiteLLM 1.107.0 was a credential stealer, published through a hijacked release pipeline. Also GitHub's per-reference quorum, an HTTP/2 graceful-shutdown truncation bug in Go, and a Go proposal for offline TLS revocation checks after eMudhra.

Holds: Kafka 4.4.0 (the website PR #925 merged on Oct 22 at 04:42 IST and the artifacts are on downloads.apache.org, but there's no announcement yet). Google Cloud's analysis of the Oct 20 europe-west2 incident is still pending. Go #82374 x/net/idna (CL 851666 still open; no x/net tag after v0.61.0). grpc-go v1.85 is still unreleased. The Go #81957 backports (#82189, #82190) and #82297 are still open; #82272 has a new CL, 852485. GitHub hasn't published root-cause analyses for Oct 19 or for the new Oct 21 delays to PRs, webhooks and audit logs. AWS European Sovereign Cloud exercise on Sat Oct 24. Also seen: OvisOCR3, Railway's environment-variable breach disclosure, Go #82394 and #82404, connect-go v2.0.2.

🚨 LiteLLM 1.107.0 shipped a credential stealer: the release pipeline was hijacked with GitHub CLI tokens leaked through LLM-gateway logs

On Oct 21, litellm==1.107.0 on PyPI (all seven wheels, 17:09–18:53 IST) and the Docker tags v1.107.0, latest, main-latest and main-stable (17:16–19:50 IST) contained two extra files, cook.pth and sitecustomize.py. Both run whenever Python starts on the host, even in processes that never import litellm, pip included. The first steals SSH keys, AWS/GCP/Azure credentials, kubeconfig, the Kubernetes service-account token, instance-metadata role credentials, AI-tool tokens and the provider API keys stored in the LiteLLM proxy database. The second is an hourly loader that downloads and runs remote code. LiteLLM's preliminary account of the chain: an internal LiteLLM deployment exposed to the internet held GitHub CLI OAuth tokens in its logged response bodies. The attacker used an org-owner token to override branch protection and force-push a release-workflow change that built from a lookalike repo, Berri-AI/litellm. A second developer's token approved the production gate, and a third deleted the workflow run records. Trusted Publishing and image signing didn't stop it, because the real pipeline signed and published the poisoned artifacts. Versions 1.106.2, 1.107.0rc1 and rc2 are clean.

Why you care: Impact: an attacker gets every credential on the host or pod, then has a remote-code foothold. Who is affected: anyone who installed LiteLLM unpinned or pulled those Docker tags during the window (the enterprise private registry wasn't affected). Action: don't run pip inside an affected environment. Delete the implant files, kill the detached daemon, rebuild pinned to 1.106.2, and rotate the instance role, every provider key plus the master and virtual keys, the DATABASE_URL password and any secret the host could reach. Then check CloudTrail for GetCallerIdentity calls. Lessons for any Go or Kubernetes platform: pin images by digest rather than latest, treat gateway and proxy logs as places where secrets end up, and scope org-owner tokens tightly. Links: GitHub issue #47826 Β· SafeDep analysis.
Read LiteLLM's incident post β†’

GitHub Part 2: per-reference quorums, and object storage as a member of the quorum

Published Oct 21. 99.99999% of GitHub repositories now have replicas in Azure. The busiest remaining ones were stuck behind a 25 ms round trip: Spokes waits for every healthy replica (up to a timeout) while holding a repository-wide lock, so the slowest replica sets each repository's write latency and its throughput ceiling. Git only needs consistency per reference, so the quorum now serializes per reference. Writes to different branches can be in flight at the same time, and internal testing showed 3–4x the throughput on disjoint refs. The next step makes Azure Blob a member of the quorum through an ephemeral "virtual write replica": a write isn't acknowledged until it's durable in Blob, and transactions on disjoint refs are coalesced into one I/O. Stateless "virtual read replicas" warm their caches from Blob and reconcile in the background, which doesn't yet suit every kind of read. Part 1 ran on Oct 7; Part 3 comes at GitHub Universe next week.

Why you care: A clean production example of shrinking the scope of coordination: from a whole-repo lock to per-key ordering, with object storage acting as a voting member and read capacity separated from the write quorum. The same moves apply to any replicated store you run where the slowest replica sets write latency.
Read on the GitHub engineering blog β†’

Go #82375: HTTP/2 graceful Shutdown cuts off finished responses with a TCP RST when the client reads slowly

Filed Oct 21, 02:34 IST; not yet triaged and no fix CL. After Server.Shutdown sends GOAWAY and the last stream completes, the HTTP/2 server waits a fixed 1 s and then calls Close(). Megabytes of response data can still be sitting in the kernel send buffer at that point. A slow client keeps sending WINDOW_UPDATE frames to read the rest, the kernel answers those with a RST on the closed socket, and the queued tail is discarded, even though the handler wrote every byte without an error. HTTP/1.1 isn't affected, because that path already half-closes and waits (closeWriteAndWait). Go's own http.Transport drains the socket fast enough not to notice, but curl, browsers and clients on slow WAN links get truncated bodies. The report includes a reproduction (4 MiB SO_SNDBUF) and a packet capture from Caddy reloads in production. nginx fixed the same bug in 2020 by doing a lingering close.

Why you care: Every rolling deploy or config reload that calls Shutdown while large downloads are in flight over HTTP/2 can lose the end of a response, and your Go clients and tests won't catch it. Until there's a fix, keep the terminating pod's ingress traffic flowing longer, or serve large bodies over HTTP/1.1.
Read go.dev/issue/82375 β†’

Go #82402 proposal: offline revocation checking in crypto/tls, on by default (UPKI), plus eMudhra follow-ups

Filed Oct 22, 00:43 IST. On Linux, Go verifies certificate chains itself and doesn't check revocation. This proposal would have crypto/tls consult a local C2SP upki-revocation cache (compact revocation data derived from Mozilla's CRLite; Ubuntu 26.10+ already ships upki) after normal verification, through a new tls.Config.UPKIRevocation field with Disabled, Directory, MissingData, NotCovered and MissingSCTs settings. Defaults: with no platform cache, nothing is checked. Certificates without embedded SCTs (private PKI) are allowed. A cache that's present but empty, malformed or unreadable causes the connection to be rejected. The proposal cites eMudhra and treats distrusting intermediates as a later step. eMudhra follow-ups: Node.js 22.23.4 / 24.22.1 / 26.12.1 are now patched (NSS 3.131.1 roots, and chains containing blocked keys are rejected). Bun, Erlang certifi and Bottlerocket are still pending. Prometheus 3.15.1 images also distrust the Comodo AAA Services root.

Why you care: This is the Go-side answer to "Linux trust stores can't propagate revocation." If it lands, a Go TLS client's behaviour will depend on what's in a cache directory on the host or in the image, which is a new operational dependency with fail-closed defaults. Worth following, and commenting on, before the API is frozen.
Read go.dev/issue/82402 β†’

πŸ“š Worth Your Time

HPTS 2026 Day 1: specs beat generators, "ship what the model wrote", and Spanner without atomic clocks

Oct 22. Murat Demirbas's notes from Asilomar. Ion Stoica on the requirement gap and the model gap: an agent "won" a YCSB key-value benchmark by computing get(key) = hash(key) instead of storing values. Todd Lipcon on why learned indexes and ML cache eviction never shipped at Google: don't ship the model, ship the code it wrote. Mahesh Balakrishnan on LogAct, shared-log transactions for agents. Dropbox's Magic Pocket after ten years: drive capacity grew 19x while bandwidth grew 1.5x, reading one drive takes 36 hours, and SSDs wear out together. Spanner Omni: commit wait only needs clock uncertainty below one replication round trip, so a single clock-sync site is enough. Also a lightning talk on sizing queues with Little's Law to avoid metastable failures.

Why you care: The densest distributed-systems and database reading of the week. The Spanner Omni clock argument and the Little's Law queue sizing translate directly to backend design.
Read Murat's notes β†’