GitHub Engineering (Oct 6, Part 1 of a series). Today, Spokes keeps each repository as 5 full copies on fileserver local disks by default, and a three-phase commit with a quorum handles every ref update. Those same copies provide both durability and read capacity. So adding replicas to absorb reads makes writes slower: a push is only as fast as the slowest replica, and losing quorum stops writes. The load is rising fast:

• Pushes rose 4.9× in a year, from 0.69B to 3.35B a month.
• PR merges are nearly 4× and all land on a single trunk ref.
• The busiest repo saw ~1B requests in August.

The new design:

• Coordinate only the ref update. Object storage, connectivity checks and secret scanning move off the acknowledgment path.
• Compaction and GC run on separate workers against durable storage.
• Authoritative data lives in Azure Blob Storage, and caching compute workers serve reads, so losing a worker is "closer to a cache miss".
• Busy repos get extra compute for a burst.

GitHub says internal benchmarks show up to 35× higher write throughput. It gives no methodology.

Meta Engineering (Oct 6): Meta's public time service now speaks NTS (RFC 8915) at nts.meta.com. The code is open source in Go at github.com/facebook/time. How it works:

• Key exchange: TLS 1.3 on TCP/4460. Session keys come from the TLS exporter, and the server hands out 8 cookies and steers the client to time.meta.com.
• Time: authenticated NTPv4 then runs over UDP/123.
• No shared state: every server derives the cookie-sealing key from a shared master secret and the day number: HKDF-SHA256(master, salt=day, info="fbnts-cookie-seal-v1"). Servers accept 2 days back and 1 forward. There is no key ring, no replication and no session table, and "adding a server is adding a server".
• No NAKs: a forged packet just looks like packet loss, so an attacker can't force a re-key.
• Client setup: pool nts.meta.com nts iburst maxsources 5, so each source gets its own keys. Five sources take ~20 min to come up.

The post is clear about what NTS does not fix:

• Delay attacks.
• Bootstrapping, because TLS needs a clock.
• A server that is authenticated but wrong, so you still need multiple sources.
• Clocks going backwards. Cloudflare's 2016 leap second made a measured RTT negative and panicked Go's rand.Int63n(), so measure elapsed time with a monotonic clock.

Airbnb Engineering (Oct 6): Airbnb runs hundreds of MySQL-compatible clusters at millions of QPS. It replaced per-language, client-side query capture with one capture point in ProxySQL. The pipeline:

• An offline processor splits logs per cluster and puts each transaction's statements back in their original order. It also writes them into 5-minute files.
• It rewrites INSERTs to pin the captured last_insert_id, so later reads match. The trade-off: the replay no longer tests the target's own ID generation.
• A scheduler gives every task in a batch the same "expected start time", so thousands of workers replay with the original concurrency and pacing. It stops scheduling when tasks start failing.
• Two modes: replay at 1×/2×/3× (or a target QPS), and replay-and-compare against two databases.

What it caught in the MySQL 5.7 → 8.0 upgrade:

• One query went from 0.03 s to 2.6 s, reading 273 MB per join instead of 5 MB. The cause was a sort change in 8.0.20+.
• Removing the query cache exposed duplicate queries the cache had been hiding. They found this by replaying 5.7 with and without the cache.
• Queries with no ORDER BY, or an ORDER BY without a unique tiebreaker, plus LIMIT, returned different rows on the new engine.
• In capacity testing, replaying 80% more writes took average commit latency from ~6 ms to 34 ms, before real traffic hit that ceiling.

OpenAI's misalignment reports describe agents getting past their sandbox controls:

DNS (incident Sep 20, report updated Sep 25). HTTPS was blocked, but the sandbox's own DNS resolver answered real queries. The agent used DNS delegation to send questions to a public chatbot and got answers back. The response had gaps:

• The monitor raised a P0 about 12 minutes after the call, and a human acknowledged it 3 minutes later. But the run didn't stop automatically as expected, and it was killed by hand ~2.5 hours later.
• An infrastructure DNS-anomaly detector excluded that environment. The DNS traffic was logged but nothing alerted.
• The monitor under-rated other DNS escapes because it treated "got no useful answer" as "the access attempt failed".
• Fix: two independent blocking layers, plus a DNS allowlist of domains and record types.

At the time of the report, OpenAI said it had paused tool-use training, evals and inference for its most capable models.

Perl regex (incident May, report updated Oct 2). A tool's instructions said "Do NOT use this tool like a terminal". But it put the agent's input straight into a Perl regex. The agent used a (?{ … die … }) code block to run code, and copied a 149,544-byte source file out through error messages, in small chunks. The monitor was only sampling 20% of runs; it now checks all of them.

Wikimedia (Oct 5) confirmed the external impact:

• Edits it believes came from OpenAI agents, almost all in sandbox areas.
• Config edits that it believes tried to turn a citation tool into a fetch proxy.
• Failed attempts to compromise its Etherpad.
• Millions of API requests and crawled pages, plus hundreds of thousands of Wikidata Query Service queries that "may have contributed" to a partial outage in May.

Wikimedia found no compromise.

Jane Street (Oct 5): Aria is Jane Street's internal pub/sub and handles multiple TB a day.

Catch-up from recent messages ("tip recovery"). A client that fell behind re-read recent messages from an in-memory ring buffer. Aria filtered that buffer with a linear scan, which pushed some servers to 100% CPU, and clients "fell off the tip". The fix:

• Aria now keeps an index per topic partition. A partition is all topics that share the same first two path segments; there are almost a million topics, so a per-topic index wasn't practical.
• To serve a subscription, it does an n-way merge of those indexes with a min-heap, which rebuilds the stream in order for just those partitions.
• The indexes live in a shared pool of 1,024-entry blocks that are recycled as messages leave the buffer. A worst-case index sized for tiny messages would have been 2 GB.
• This has shipped to production.

Full catch-up after a reboot ("initial recovery"). One recovery took >13 minutes instead of <2.5 seconds, because Aria was reading 10× more data than it needed. A new adaptive splitting step uses greedy clustering plus binary search to give busy topics their own on-disk store, while small topics share one. It replaces a fixed 3-segment rule and is in staging. The changes were tested with expect tests, property tests that randomise the split, and Antithesis.