Incident on Oct 20, 11:30β17:40 IST (6 h 10 m); preliminary report posted Oct 22. A grid disruption hit the facility hosting europe-west2-b. The UPS batteries took the load, but both the primary and the backup PLCs (the automation controllers) failed to start the diesel generators and transfer the load to them. When the batteries ran out, hosts, network switching and storage in -b lost power. Google names two reasons the impact reached europe-west2-a and -c: the sudden surge of redirected queries, HA database failovers and compute saturated the remaining capacity, causing admission queues, slow DB failovers and 500/503 errors; and regional control planes made synchronous validation calls across zones to metadata and catalog components in the dead zone, which timed out. BigQuery, Looker, GCE, GKE, Regional Persistent Disk and Cloud SQL degraded in the unaffected zones. Around 40 products are listed, including Managed Service for Apache Kafka, Pub/Sub and Spanner. A full RCA will follow. Also out: GitHub's root-cause analysis for Oct 19. An Actions service lost its connections to cache servers in one region and every server reconnected at once, overloading a shared data-centre-to-region network link. The connection-rate limiter meant to prevent this was still in shadow mode (logging only). It's now enforced, with backoff and per-service caps on the shared link (GitHub status).
π₯ What Changed
A normal Friday led by postmortems: Google Cloud explains how losing one zone degraded europe-west2-a and -c, and GitHub's retry-storm RCA shows the same pattern. Also: Istio and Envoy distrust eMudhra (and eMudhra is leaving public TLS), Echo and Go reverse-proxy auth-boundary fixes, and the OCRia-1 document parser.
Holds: Kafka 4.4.0 (website PR #928 still open, no announcement yet). Go #82374 x/net/idna (fixed in x/net, but it still needs vendoring into the 1.26/1.27 branches, and there's no x/net tag after v0.61.0). grpc-go v1.85 is still unreleased. The Go backports #82189, #82190 and #82297 are still open; #82026 has a disputed cause and a candidate patch. GitHub hasn't published root-cause analyses for Oct 21 or for the Oct 22 Actions Results incident. Google Cloud's final europe-west2 report is still to come. AWS European Sovereign Cloud exercise on Sat Oct 24. Murat's HPTS Day 2 notes aren't up yet. Also seen: Fly.io SBD, Go #82420 and #82416, etcd patch releases.
Istio 1.31.3, 1.30.7 and 1.29.10 (advisory dated Oct 21). In 1.30 and 1.31, istiod, istioctl and pilot-agent are built with Go 1.26.10, which rejects chains containing the compromised keys. ztunnel moves to rustls-webpki 0.103.16, and the ubuntu and distroless images drop emSign ECC Root CA - G3. 1.29 reached end of life on Oct 15 and gets this exceptional release, but it's built on Go 1.25 and only gets the updated CA bundle. Envoy has no built-in list of distrusted keys and doesn't check revocation unless a CRL is configured, so sidecars and gateways depend on the CA bundle in the image. You're exposed if you use a DestinationRule with tls.mode: SIMPLE and no caCertificates/credentialName (typical for ServiceEntry egress), a RequestAuthentication jwksUri, or XDS_ROOT_CA=SYSTEM. Mesh mTLS isn't affected. If you can't upgrade, pin caCertificates or set caCrl. Envoy 1.40.1 / 1.39.5 / 1.38.8 (Oct 23, 04:50β07:39 IST): the Docker images no longer trust the compromised CAs, and Envoy Mobile drops the AAA Certificate Services and emSign ECC Root G3 roots. Argo CD 3.3.18 is a distrust-only release. eMudhra (Oct 22): the board is voluntarily withdrawing from the global SSL/TLS certificate business, which it puts at about 1% of revenue. It says the affected CAs were revoked within 14 hours and issued no end-entity certificates, and that its G1 RSA roots will keep serving customers during the transition. Also in Istio 1.31.3: istiod no longer fails to start when its self-signed CA certificate expired while the deployment was scaled to zero.
Echo advisories published Oct 22, 22:03 IST. GHSA-cxf3-9h37-7rr8: Echo's Proxy middleware switched to a raw tunnel as soon as a client requested a WebSocket upgrade, before checking whether the upstream accepted it. If the upstream answered with an ordinary HTTP response and kept the connection open, every later request on that connection went straight upstream, skipping Echo's router and any route-level authentication or authorization middleware. Affects all v4 up to 4.16.0 and v5 up to 5.4.0; fixed in 4.16.1 / 5.4.1, which route upgrades through Go's reverse proxy. GHSA-h53f-rm2w-w7rg (CVSS 5.3): in v5, ExtractIPFromRealIPHeader checked the header's value against the trusted ranges instead of checking the direct peer, so RealIP() could be spoofed. v4.16.2 / v5.4.2 (Oct 23, 07:24 IST) add that a BasicX scheme no longer reaches the Basic-auth validator and that CSRF rejects empty tokens. Go #82431 (filed Oct 23, 01:57 IST; NeedsFix; CL 852965): httputil SetURL and NewSingleHostReverseProxy don't clear URL.Opaque. A request line like GET a:b:/path HTTP/1.1 therefore forwards the opaque path, and over HTTP/1.1 possibly the authority, ignoring the target's base path. The Go team classifies it as a bug and hardening fix, not a CVE.
httputil.ReverseProxy and enforce auth only at that layer, upgrade Echo now, and until Go ships its fix, reject non-origin-form request targets.
Oct 22. A 4B vision-language model that now powers LlamaParse. On ParseBench it scores 87.4 overall, against LightOnOCR-3-4B 75.3, Chandra-OCR-2 74.8 and Mistral OCR 4.1 67.5. It also scores 85.1 on charts (output as HTML data tables) versus 66.1 for the next model, and 83.4 versus 67.9 on formatting. On third-party benchmarks the lead is small: olmOCR-bench 83.5 vs 82.3 (rescored at 1,288 px rendering), PulseBench-Tab 84.1 vs 82.4, fr-bench-pdf2md 72.1 vs 71.4. All of these were run by the vendor, and ParseBench is LlamaIndex's own benchmark. Trained on 100k+ pages with SFT followed by RL with verifiable rewards. According to the technical report, a 0.69B DFlash-2 drafter gives about 4,700 tokens/s for a single request on one B200 and about 10.2 pages/s at 64 concurrent requests. It's hosted only, with no open weights yet. The open alternatives are still OvisOCR3 and Nemotron 3.5 Parse.
π Worth Your Time
Oct 23, 03:49 IST. The post explains why cgroups can't fence the last-level cache, memory bandwidth or softirq time, and how the HTB traffic-shaping lock ends up acting as a noisy neighbour itself. Stateful stream processors suffer most, because falling behind means minutes of checkpoint restore. Netflix weighed Firecracker/Kata (needs bare metal or nested virtualization, no managed service, no GPUs), gVisor (I/O overhead, effectively single-vendor) and plain EC2 (hands the ops burden back to teams). It chose one right-sized EC2 instance per task that is still a Titus host. Opt-in is by allowlist. The federation layer decides eligibility and allowed instance families, the autoscaler launches the smallest fitting instance just in time, and the scheduler enforces one task per node with pod anti-affinity, a taint/toleration and a non-preemptible priority class, with no defragmentation. Warm pools stop paying off once there are many instance shapes. Isolation is measured with a throughput- and latency-efficiency benchmark rather than asserted. Moving a few noisy workloads onto their own nodes also lets the shared fleet pack more densely.