Daily Engineering Intelligence

Technology Newspaper

← Back to Home

Engineering Intelligence β€” Wednesday

October 28, 2026

πŸ”₯ What Changed

A busy Wednesday that closes several holds. Go: x/net v0.62.0 fixes two x/net/html CVEs, and go-jose has a JWK CPU DoS with no fix on v3. Kafka 4.3.2 fixes two silent data-loss bugs. Tailscale patches SSH and serve paths that reached root-only sockets. grpc-go v1.85 is out. Google's final europe-west2 report explains why the generators never started. For reading time: Uber on performance-normalized CPU for stateful workloads.

Also patched: confluent-kafka-go v2.16.1 raises its go directive to 1.26.10 for the eMudhra distrust, but Go 1.27.0–1.27.2 satisfy that directive and are still vulnerable, so you need 1.27.3 or later (broker TLS through librdkafka isn't affected). Grafana 13.3.0/13.2.5/13.1.9/13.0.12/12.4.14 are security releases (CVE-2026-100105, -88858, -88967 on every line; no details published yet). Atlantis 0.47.1 fixes GHSA-rjh2-4xgw-p2hj (high): a workspace name from a PR's atlantis.yaml reached a shell unquoted, giving RCE with the Atlantis service account's cloud and VCS credentials. eMudhra: Microsoft removed six eMudhra G3 roots and an AAA Certificate Services cross-sign from the Windows trust list (out-of-band update, 10-26) and from Edge (10-27). Chrome's emSign ECC Root CA – G3 removal was scheduled for 10-27. Go #82472 (Oct 24): fix CL 854865 adds lfstack/spanSet backoff on amd64; in A/B tests 18 of 20 heap-triggered GC cycles stalled without it and 0 of 9 with it. It's still in review.

Holds: Go 1.27.4/1.26.11 (not pre-announced; 23 and 20 open issues); Go #82374 x/net/idna (not in v0.62.0); #82497 and #82547 unchanged; Bun eMudhra update; GitHub root-cause analyses for 10-21/22/23/26 and a new 10-27 incident (12:25–13:31 IST, 404s for GitHub App server-to-server tokens). Alternates: OpenBMB MiniCPM-OCR 1.0 and the Jev OCR quality judge (OCR ran yesterday; waiting for independent evals), MCP Python SDK GHSA-w4fh (bearer tokens accepted without an audience check; fixed in mcp 1.30.0/2.2.0), OTTL v1 stable, Go #82615 (HTTP/2 WINDOW_UPDATE) and #82649 (ReverseProxy trailers). Closed: the x/net/html release, Kafka 4.3.2, grpc-go v1.85, and the final europe-west2 report.

Go security bumps: golang.org/x/net v0.62.0 fixes two x/net/html CVEs, and go-jose has an RSA-JWK CPU DoS with no fix on v3

(a) This is the x/net/html release pre-announced for 10-27. CVE-2026-78661 (GO-2026-6860, #82224): html.Parse/ParseFragment take quadratic time when coalescing adjacent text nodes, so a long input of repeated a</x> runs is enough. CVE-2026-97045 (GO-2026-6866, #82622): html.Render checks whether a text node's ancestors make it literal, but it stopped at the nearest foreign (SVG/MathML) ancestor. A subtree fostered under <mtext> via <mglyph>/<malignmark> therefore re-parses in a different namespace. That's the classic parseβ†’renderβ†’parse mutation-XSS gap for HTML sanitizers. v0.62.0 also carries last week's ipv4/ipv6 bounds fixes. The x/idna Punycode fix (#82374) is still not in it. (b) go-jose GHSA-mgcc-cww5-fp2q (CVE-2026-116587, high): JWK parsing passes an unbounded RSA modulus and primes to crypto/rsa PrivateKey.Validate(), which is cubic-time on Go 1.24+. One crafted key, for example in an embedded jwk header passed to ParseSigned, burns CPU. Fixed in go-jose/v4 v4.1.6. v3 (≀3.0.5) has no patched version. Go 1.27.4/1.26.11 still aren't pre-announced, though #82536 (the ML-DSA ClientHello fix) is now cherry-pick approved.

Why you care: Run go get golang.org/x/net@v0.62.0 in anything that parses or re-renders untrusted HTML, such as sanitizers, scrapers and email/HTML previewers. go-jose sits under many OIDC/JWT stacks. Check go mod why github.com/go-jose/go-jose/v3 in particular: v3 has no fix, so the only remedy is moving to v4.1.6.
Read the golang-announce post β†’

Apache Kafka 4.3.2 is out with two data-loss fixes: tiered storage could delete in-retention remote data after a leader election, and a clean restart could make time retention delete recent segments

About 60 fixes (announce thread), and both data-loss fixes are also in 4.4.0. KAFKA-20732 (Critical; 3.9.0–4.3.0): after a leader election, highestOffsetInRemoteStorage is in-memory only and starts at -1. The copy task and the size-retention task both start immediately. If expiration runs first, the whole local log is counted on top of the remote size, so the partition looks roughly twice as big as it is. The broker then deletes the oldest remote segments and advances logStartOffset, which replicates to followers. Consumers see OFFSET_OUT_OF_RANGE for data that was still within retention.bytes. KAFKA-20979 (Major; 4.1.0, 4.3.0): on clean shutdown, the time index is fsynced at its 10 MB preallocated size and then truncated without an fsync. After a restart, the zero-filled tail reads as timestamp-0 entries, and time-based retention deletes segments that should have been kept. Also: several Streams StreamThread-death fixes (KAFKA-20808/20827/20721), a premature state-dir cleanup fix (KAFKA-20805), and CVE-driven dependency bumps (jackson-databind, lz4-java, log4j, Jetty).

Why you care: If you run tiered storage with size-based retention on 3.9–4.3, every leader election (rolling restarts included) is a chance to silently lose remote data, so upgrade to 4.3.2 or 4.4.0 before your next roll. KAFKA-20979 hits ordinary clean restarts on 4.1+. Neither shows up as an error, only as offsets that are gone.
Read the Kafka 4.3.2 announcement β†’

Tailscale v1.104.4 fixes the "Tailworm" findings: SSH forwarding and serve could reach root-only sockets, including tailscaled's own LocalAPI

A researcher published "Tailworm" on 10-24, two hours after sending it to Tailscale. Four bulletins. TS-2026-014: Tailscale SSH validated Unix-socket forwarding paths lexically but acted as root. ssh -L could dial root-only sockets (the Docker socket, tailscaled's LocalAPI), ssh -R could unlink and replace root-owned sockets, and a TOCTOU race defeated the allowlist. Forwarding now runs as the authenticated user. Mitigation: TS_SSH_DISABLE_FORWARDING=1. TS-2026-015: window-change messages and env appends caused unbounded memory growth that survived disconnect, ending in a tailscaled OOM. TS-2026-016: serve's TCP forwards to Unix backends dialed as root, so raw bytes could reach the LocalAPI (full node takeover), and the symlink guard only ran at config reload rather than on every dial. TS-2026-017: HTTP/2 streams on one connection shared one serve context, so Tailscale-App-Capabilities could bleed between mounts. Exploitation needs a foothold on an authorized peer with SSH allowed by policy. Tailscale calls it lateral movement, not a worm.

Why you care: If you use Tailscale SSH or serve to reach boxes, upgrade to 1.104.4, then tighten SSH policy and use check mode for sensitive hosts. The bug classes are also a useful Go review checklist for any root daemon acting for a user: lexical path checks followed by privileged syscalls, guards checked at config time instead of use time, and per-request state stored on a per-connection context under HTTP/2.
Read Tailscale's findings and fixes β†’

grpc-go v1.85.0 ships: control-frame throttling now counts only peer-induced frames, plus HTTP/2 stream-ID and END_STREAM correctness fixes

Behaviour changes worth knowing about. Control-buffer throttling now counts only control frames the peer caused. The default GRPC_GO_EXPERIMENTAL_CONTROL_BUFFER_THROTTLE_LIMIT goes from 100 to 10,000, and client connections are closed if reads stay throttled for 10 s (#9563). Stop() now force-terminates even while GracefulStop() is waiting (#9394). The server always advances the highest stream ID on HEADERS: before, a truncated HEADERS frame allowed stream-ID reuse, against RFC 7540 Β§5.1.1 (#9414). HEADERS with END_STREAM now terminate the stream; before, handlers could hang (#9391). Requests missing both :authority and Host get HTTP 400 (#9365). Other fixes: xDS header matchers with non-lowercase names (#9353), retries use the LB-chosen :authority (#9277), and truncated messages return io.ErrUnexpectedEOF instead of io.EOF (#9204). New APIs: metadata.ValueFromOutgoingContext (#9282) and the effective authority in stats.OutHeader (#9296).

Why you care: It's a routine bump, but two changes affect shutdown and error handling. If your shutdown path calls GracefulStop with a timeout and then Stop, the force-close now actually works. Code that matched io.EOF on truncated streams needs to accept io.ErrUnexpectedEOF.
Read the grpc-go v1.85.0 notes β†’

Google's final report on the europe-west2-b outage: the generator controller watched voltage on a locked-out feeder, so the generators never started

The site was temporarily on a single feed (feeder B opened and locked out for repairs after a 10-14 generator test). At 06:53 BST an external cable fault dropped feed A. The PLCs' undervoltage sensing was wired on the line side of the isolated B breaker. Both controllers saw live upstream voltage, concluded the utility was healthy, and never started the generators. The 5-minute UPS ran out at 07:00, and the system also inhibited remote transfer. Staff manually switched to the standby utility, and data halls had power again at 08:20. Recovery then took much longer. Unclean-shutdown storage recovery ran from 08:50, and responders deliberately held services back until 11:06. Traffic was re-admitted at 11:45, and GCE was 99.5% healthy at 13:08, 6 h 08 m in total. Zones a and c were hit by demand surges and cross-zone control-plane calls to the dead zone, and "some failovers were unsuccessful due to capacity constraints and difficulty in replicating the underlying storage mechanism". The impact list includes GKE, Pub/Sub, Managed Service for Apache Kafka, Cloud SQL, AlloyDB and Spanner. Actions: fix the PLC logic to check both voltage and breaker position, validate PLC configs globally by 15 Nov, freeze maintenance at similar sites, and practise the emergency procedure twice per shift.

Why you care: The health check measured the wrong point: voltage being present didn't mean the breaker was closed. That's the same failure as a readiness probe that checks a port rather than the dependency behind it. For your own architecture, the real lessons are about the recovery: losing one zone overloaded the other two, and regional control planes depended on the dead zone. Multi-zone only works if the surviving zones have headroom and nothing regional calls into the lost zone.
Read the final incident report β†’

πŸ“š Worth Your Time

Uber: "not all cores are created equal" β€” normalized compute units for stateful workloads cut Kafka cores by about 8%

Odin, Uber's stateful platform (about 5M cores), gives each host type a performance factor, with an older generation as the 100 baseline. For Kafka: newest cloud ARM 127, on-prem gen 20 135, gen 21 151, newest cloud x86 196. CPU usage is converted to normalized units, the cluster target is computed from the hottest node (because any follower can become leader), and the result is converted back to physical cores per host. Synthetic benchmarks weren't accurate enough, so factors are now re-estimated daily from production usage for each technology (Kafka and Cassandra differ). Across about 1.4M auto-scaled cores, the Kafka utilization spread fell from 8 to 3 percentage points (the hottest host type was 50% hotter than the coldest, now about 10%) with about 8% fewer Kafka cores. A leader-follower workload that degraded whenever it wasn't on the fastest hardware stopped doing so.

Why you care: Kubernetes CPU requests are absolute core-time, so the same Kafka or Postgres pod is over-provisioned on new nodes and under-provisioned on old ones. Stateful leaders can't be load-balanced away from that difference. This is a concrete design for mixed ARM/x86 node pools, and it explains why your leader pods get slower after rescheduling.
Read Uber's post β†’