Daily Engineering Intelligence

Technology Newspaper

← Back to Home

Engineering Intelligence β€” Tuesday

November 3, 2026

πŸ”₯ What Changed

A busy Tuesday with six items. The Linux TCP remote-root bug from last week now has fixed kernels on RHEL, and Ubuntu has fixes in its -proposed pocket. Red Hat and Ubuntu both say MPTCP isn't required to trigger the bug, and Red Hat has published an nftables stopgap. One external contributor used an AI-assisted fuzzer to file ten Go compiler wrong-code bugs in a single day, including one on arm64 that dates back to go1.21. AWS Lambda can now consume self-managed Kafka as a queue using share groups, and Temporal's Go SDK team has proposed native deterministic execution for Go. The two long reads are Cloudflare's Kafka-fed DNS build pipeline and OpenAI's rebuilt Git backend.

Also seen: go-jose v4.1.8 (11-03 00:51 IST) adds ML-DSA signing; still no v3 fixes. Temporal 1.32.2/1.31.5/1.30.9 (11-03 ~00:00 IST) ship "security vulnerabilities" fixes, which are dependency bumps to Go 1.26.10, x/net v0.60.0 and go-jose v4.1.6. That is not v4.1.7, so the 10-31 JWE-header fixes aren't included. ValidateWorkerDeploymentVersionComputeConfig now requires namespace Write access. nats-server v2.15.3-RC.1/v2.14.10-RC.1 (Go 1.27.3) fix wildcard characters inside a subject token causing subject-matching collisions (#8765), consumer info reporting uncommitted assignments as success, and R1 scale-up stalling under writes; they also add TLS verify: optional. Go #82770: the HTTP/2 client keeps sending requests on a connection whose RST_STREAM health PING is never answered (e.g. a dropped NAT mapping) when SendPingTimeout is 0, so set ReadIdleTimeout/SendPingTimeout on long-lived clients. Go #82813/#82814: the HTTP/1 server keeps reading requests after an unhijacked 101 response, and the HTTP/2 transport accepts 101. Both were reported as security issues, and Go is treating them as non-security bugs. Postgres 19 correction: the RC1 announcement (10-15) already set RC2 for 11-12 and GA for 11-19. Kafka Monthly Digest (10): KIP-1381 proposes letting brokers disable compression codecs for client writes, citing native-code risk.

Holds: Go 1.27.4/1.26.11 (not pre-announced; go.dev/dl still 1.27.3/1.26.10; now also carrying the miscompile backports and #82771); Go #82717 MPTCP default (no comments since 10-31 21:15 UTC); CVE-2026-98432 Ubuntu -updates, Ubuntu cloud kernels and OpenShift; x/oauth2 CL 854305 (still NEW, last update 10-29); Go CL 857381 (#82756), CL 857344 (#82777); go-jose v3 fixes; Kafka 3.9.3; Go #82785 review; GitHub RCAs for 10-21/22/26 and 10-28 to 10-30; Northflank root cause; Dapr CVE/GHSA IDs; Postgres 19 RC2 11-12 / GA 11-19. Alternates: TreasuryQA Bench (arXiv 2610.38924, agents + MCP + OCR over scanned Treasury archives; OCR scout), Quantiphi "From Text to Vision", WorkOS breach notice (CRM/Salesforce/Snowflake data, attributed to an AI-driven pentest framework; seen only as a customer email posted to HN, nothing on workos.com yet, unverified), MinerU 4.1.4, Ankit Rana "R + W > N".

CVE-2026-98432 update: RHEL 8.10/9.8/10.2 kernels fixed 11-01, Ubuntu 22.04/24.04/26.04 fixes only in -proposed (11-02), OpenShift still unfixed; Red Hat and Ubuntu say MPTCP is not required

Red Hat released Critical kernel errata on 11-01: RHEL 8.10 kernel-4.18.0-553.182.1.el8_10 (RHSA-2026:93414; RT kernel in RHSA-2026:93415), RHEL 9.8 5.14.0-687.64.1.el9_8 (RHSA-2026:93412) and RHEL 10.2 6.12.0-211.72.1.el10_2 (RHSA-2026:93413). Third-party trackers report AlmaLinux and Oracle RHCK builds landing on 11-02. Red Hat's bulletin RHSB-2026-016 says OpenShift 4 (RHCOS nodes) is affected, interim guidance is "being evaluated", and "Multipath TCP is not a required condition for this flaw… nodes in their default configuration, where MPTCP is disabled, remain affected." It publishes a stopgap nftables table that drops inbound IPv4 TCP packets carrying IP options (ip protocol tcp ip hdrlength > 5 counter drop, hook input, priority -310). Red Hat says the rule "reduces exposure… but cannot fully prevent it", and that on OpenShift it doesn't filter pod-to-pod SDN traffic. Ubuntu's 11-02 post: fixed linux packages are in -proposed for 22.04 (5.15.0-203.213), 24.04 (6.8.0-149.149) and 26.04 (7.0.0-43.43), and fixed in 26.10 (7.3.0-10.10). Ubuntu will move them to -updates after testing on more than 2,000 certified machines. 16.04 to 20.04 fixes are pending (Ubuntu Pro). Ubuntu also says MPTCP-disabling mitigations exist only because Anthropic's unreleased exploit uses MPTCP, and that "security updates are the only appropriate mitigation". Cloud and HWE kernels (linux-aws, linux-gcp, …) each have their own tracker rows.

Why you care: This changes what to do on the Ubuntu and RHEL hosts that were still waiting last week. Patch RHEL-family nodes now. On Ubuntu LTS nodes, decide whether to take the -proposed kernel (Ubuntu documents the apt install -t <codename>-proposed path) or wait for -updates. Until a node is patched, Red Hat's IP-options rule is the vendor-backed version of the packet filter Damien Neild suggested. Keep the Go-side GODEBUG=multipathtcp=0 from 10-30, but both vendors now say plainly that it only narrows the one known exploit. EKS/GKE/AKS managed node images follow their own kernel streams, so check the cloud-kernel row, not just linux.
Read Red Hat RHSB-2026-016 β†’ Also: ubuntu.com β†’

An external contributor using an AI-assisted fuzzer filed ten Go compiler wrong-code bugs in one day, including an arm64 bug present since go1.21; 1.27/1.26 backport issues are open for nine

Contributor s-celles filed ten cmd/compile miscompilation issues on 11-02 (#82763, #82767, #82772–#82777, #82782, #82786), each with a minimal reproducer and a pointer to the faulty SSA rewrite rule. #82777 says the bugs were "found with a metamorphic fuzzer (constant vs. run-time operands). Assisted by AI." The one most server teams could hit is #82777 (arm64, go1.21.13 through 1.27.3): when a conditional select's flags fold to a constant, the CSINV rule rewrites ^y to boolean Not, which lowers to XOR $1, so ^y becomes y ^ 1 (f(0,2) returns 3 instead of -3). Fix CL 857344 is in review, and Jorropo called it "a misscompilation" and opened backports #82800/#82801. The rest are on other architectures or narrower: arm bits.TrailingZeros64(0) returns 32 (go1.16+, #82786), riscv64 int8(x - (-x)) >> 8 compiles to 0 (go1.26 regression, #82773), s390x signed/unsigned compares (#82774), ppc64 truncation (#82775), mips/mipsle shifts (#82772, #82776), ppc64 power10 ReverseBytes16 (#82782), and two constant shift-composition rules (#82763; #82767, a tip regression from prove-pass changes). Separately, release-blocker #82771: under -race, a data race whose reading side runs in a channel timer's sendTime crashes the process with SIGSEGV instead of a race report (go1.26.9 through 1.27.3; fix CL 857721, backports #82783/#82784).

Why you care: This continues yesterday's theme. Fuzzers paired with LLMs are now finding long-standing wrong-code bugs in mature compilers, several a day, and the next Go point releases will carry them along with the 12 queued CVEs. Practical exposure is low on amd64 but real on Graviton and other arm64 fleets, since the trigger is ordinary code once inlining makes a condition constant. There's no workaround to apply, so plan to take Go 1.27.4/1.26.11 promptly. If you see an arm64-only test failure involving bitwise NOT inside a conditional, check #82777 before debugging your own code. If go test -race jobs in CI segfault in timer code, that is #82771, not your test.
Read Go issue #82777 β†’

AWS Lambda can now consume self-managed Kafka and Confluent Cloud as a queue, using Kafka share groups, so parallelism is no longer capped at partition count

Self-managed Kafka event source mappings (your own clusters or managed ones such as Confluent Cloud) now accept ConsumptionMode=Queue. Queue mode uses Kafka share groups (GA in Apache Kafka 4.2) instead of the consumer-group protocol. Several event pollers can then read the same partition at once, each running up to 10 concurrent invocations, with no changes to producers, topics or function code. The trade-off: queue mode does not preserve partition ordering, and AWS says it's for workloads that don't need ordering. It requires a cluster with share groups enabled (Kafka 4.2+), it's available in every commercial Region that supports self-managed Kafka ESM, and provisioned mode still bills Event Poller Units.

Why you care: This is the first big managed consumer built on share groups (KIP-932), and it handles the usual "one slow message blocks its partition" and "over-partition to get more consumers" problems without touching producers. For event-driven Go services the design rule still applies: queue semantics mean at-least-once delivery with no ordering, so handlers must be idempotent and keyed (dedupe on message id in the same transaction as the side effect). Kafka 4.4 added share-group DLQs (KIP-1191), which makes this more usable for poison messages. Don't switch an ordered topic, such as per-entity state changes, to Queue mode.
Read the AWS announcement β†’

Temporal's Go SDK engineers propose runtime/bubble: host-controlled, reproducible execution of native goroutines, channels, select and timers, built on the testing/synctest machinery

GitHub users @yuandrew and @Quinn-With-Two-Ns (Temporal Go SDK) propose an opt-in API with no new syntax. bubble.New(options, fn) creates a paused bubble, and b.Step(Activation{Now, Deliver}) delivers host inputs and sets logical time, then runs until the goroutines are quiescent, complete or fault. Step returns a State with Status (Quiescent/Completed/Faulted), RootDone, LiveGoroutines and TimerChanges. Timers fire only when the host authorizes them through Delivery.FireTimer, and advancing time alone doesn't fire them. b.Close() and b.Discard() handle cleanup. They give two use cases: deterministic simulation testing for Go distributed systems (replaying a failing interleaving of late messages, timeouts and crashes), and Temporal's workflow replay. Temporal's Go SDK currently ships its own replacements for goroutines, channels, select and timers to get determinism, and the authors want to build on native primitives instead. The proposal is brand new, with no comments or committee review yet.

Why you care: If accepted, Go would get a supported foundation for FoundationDB/TigerBeetle-style deterministic simulation, which today requires rewriting code against a custom scheduler. Both OpenAI's Pando (below) and Jepsen's ParticleDB report this week show how much that kind of testing catches. It also explains why Temporal workflow code can't use plain go, channels or time.Sleep. Read it for the design, and watch whether the proposal review group engages; it is early, and the restrictions inside a bubble matter.
Read Go proposal #82785 β†’

πŸ“š Worth Your Time

Cloudflare's DNS Zone Builder: a Kafka-fed pipeline that now picks the smallest provably safe rebuild, cutting peak memory 85% and p99 spikes from 14.6 to 1.21 minutes

Alex Fattouche describes the pipeline from a DNS change to Cloudflare's edge: database β†’ Kafka events β†’ DNS Kafka Aggregator β†’ Zone Builder β†’ Quicksilver KV. It handles about 700 record changes per second, and some zones hold over 20 million records. Zone-setting events used to trigger full rebuilds that held whole zones in memory. A new planner chooses one of five build types (resources only, resources + metadata, apex + metadata, purge, full). The rule: a targeted build must write keys byte-for-byte identical to a full build, and if the planner is unsure it does a full build ("the planner never turns uncertainty into less work"). Plans are re-checked against a fresh replica read just before writing, and a replica that is behind is rejected. A unified scheduler merges work for the same zone, and merging only ever adds work. It keeps the DNS work separate from the Kafka messages awaiting acknowledgement and uses soft commits: builds can finish out of order, but the committed offset advances only when every earlier message has finished, so a restart replays work instead of losing it. Rollout used equivalence tests, a side-by-side full build for every targeted build, and a mirror Zone Builder fed by the production Kafka stream that writes to a separate Quicksilver; a comparer samples 20% of zones (0.31% transient diff rate). Results: 92.95% of builds avoid a full rebuild, memory fell from 49β†’7 GiB average and 79β†’12 GiB peak, consumption now tracks production instead of catch-up bursts of up to 1,550 msg/s, and median delay fell from about 7.7 to about 3.3 s.

Why you care: These are well-explained patterns for any Go/Kafka consumer that rebuilds derived state: event payloads that say what changed, "escalate on uncertainty" planning, per-key coalescing, ack tracking kept separate from work, out-of-order completion with in-order commits, and a production-traffic mirror with a diffing comparer before cutover. It also has a useful aside on how coarse histogram buckets inflate p90/p99 estimates.
Read Cloudflare: targeted zone builds β†’

OpenAI's Pando: a Git-compatible backend that drops repository-wide consistency for per-ref compare-and-swap and causal consistency, serving about 1 PB of packs a day

Agent-driven development pushed OpenAI's monorepo reliability below 90% at times. Pando stores each push's packfile as an immutable blob, records a "ref version" (new tip, pack, dependencies) in a horizontally scaled metadata store, and guards only the ref pointer with a compare-and-swap. Pushes to unrelated branches never contend, and a single ref is serializable with read-your-writes. Readers get causal consistency: a multi-branch clone can see foo and bar at different points in time. The post compares this design with modeled three-phase-commit replicas (disks and quorum round trips on the write path) and a shared WAL manifest in blob storage (a CAS bottleneck, plus replicas falling behind on reads), with an interactive simulation of 1,000 writers. Reads reuse stored packs through Git's packfile-uri, sent from regional caches with over 99% hit rate. Pushes advertise only trunk, so packs can be reused (median under 20 KiB, p99 under 3 MiB). LSM-style compaction (L0β†’L4 covers) doubles as a skip list, so fetch planning is O(log Ξ”). It was built in Rust in 10 weeks, largely by Codex agents, with a Git conformance suite, deterministic simulation testing, and Lean/TLA+ models used to find bugs. It has run in production since April: median push latency is about 5x lower, and it has served 57 TB in one hour. Upstream Git patches are in progress.

Why you care: This is a clear worked example of choosing the consistency boundary: provide only the guarantee users rely on (per-ref ordering), and the write path scales horizontally. The same question comes up when designing an event store, an outbox or a config service, where you have to decide whether you need a global order or only an order per key. Read the 3PC vs WAL vs per-key-CAS section and the compaction-as-index trick. Treat the "built by agents" claims as vendor-reported, but note that the verification they relied on was conformance tests, DST and formal models, which is the lesson Jepsen drew from ParticleDB.
Read OpenAI: Scaling Git β†’