Daily Engineering Intelligence

Technology Newspaper

← Back to Home

Engineering Intelligence β€” Monday

October 26, 2026

πŸ”₯ What Changed

A quiet Monday. One catch-up postmortem worth reading for the attack chain: how PingCAP's breach went from a single GitHub account to a month of quiet cloud data exfiltration. For reading time: Marc Brooker's trace-backed animations of etcd and Postgres tipping into metastable failure.

Reminder: Chrome removes the eMudhra root emSign ECC Root CA – G3 on Tuesday Oct 27, morning Pacific time (evening in India). Okta and Auth0 are dropping it for outbound hooks too. Holds: Kafka 4.4.0 (artifacts are on downloads.apache.org, but the website PR #928 is still open and there's no announcement) and 4.3.2. Go #82374 x/net/idna (x/net is still v0.61.0). grpc-go v1.85 is still unreleased. The Go backports #82189, #82190 and #82297 are still open. Go #82497 (MaxConcurrentStreams) is now marked NeedsInvestigation, with no fix yet. Still waiting on GitHub's root-cause analyses for Oct 21–23, Google Cloud's final europe-west2 report, and AWS's write-up of the Sovereign Cloud exercise. Also seen: Go #82504, a Transport retry regression on tip only (not in any release), and PlanetScale on shard-group design.

PingCAP / TiDB Cloud breach: a compromised GitHub account led to keys in Git history and plaintext Terraform state, then eight GCP service accounts and a month of data exfiltration

Between Sep 9 and Oct 7 an attacker used stolen service-account credentials to get into PingCAP's internal systems. PingCAP noticed on Oct 7 and says that by Oct 14 all identified compromised credentials were invalidated and the access paths blocked (incident report, Oct 19; CEO note, Oct 23). NebuSec's investigation (Oct 21) lays out the chain. The attacker got in through a compromised GitHub account that exposed private repos. Service-account private keys sitting in Git commit history and plaintext Terraform state, plus a plaintext Jira API token and a bot key in code, let them pivot into Google Cloud (eight service accounts), Jira and the internal chat used for TiDB alerts. Customers' AWS keys that had been pasted into Jira support tickets led on to S3 buckets holding TiDB diagnostic data. BigQuery EXTRACT jobs wrote 6,458 tables to Cloud Storage as .json.gz (Sep 23–24), and the attacker downloaded them with rclone (Oct 2–5). What was exposed: names and emails, org/project/cluster IDs, SQL editor text and errors, contracts, pricing and billing, and encrypted TiDB Cloud API-key values (the encryption key was not exposed). There's no evidence that data stored in customers' TiDB databases was accessed, or that the attacker ran commands or stayed in production. Four vulnerabilities found in the stolen source code were fixed and retested by Oct 14. The attacker group (FulcrumSec) claims 10.6 TB, 413 repos and 40K+ active API keypairs, per InfoQ (Oct 25). None of that is verified.

Why you care: Nothing here needed a zero-day. Every hop was a long-lived credential sitting somewhere it shouldn't be: Git history, Terraform state, a ticket. Customer data left through ordinary cloud APIs (BigQuery export, GCS reads), and nobody noticed for a month. Worth a 15-minute check on your own estate: run gitleaks/trufflehog over full Git history, look for unencrypted remote state, find service-account JSON keys you could replace with workload identity, and set alerts for bulk BigQuery/S3 exports. If you use TiDB Cloud, replace your management API keys and rotate anything you ever gave their support team.
Read NebuSec's investigation β†’

πŸ“š Worth Your Time

Marc Brooker, "Animations for System Builders": real traces showing etcd and Postgres tipping into metastable failure, and Raft losing writes to a lying fsync

Published Oct 23. A short post with animations built from traces captured in a real test environment. (1) A Raft cluster can lose writes if even one member's storage lies about fsync(). (2) Postgres 16 in a MicroVM easily handles 20 queries/s while its working set sits in the page cache. After a load spike, the running queries (12 MB each) push data out of cache, so queries are served from disk and run slower, and the system stays in that slow state after the spike. (3) etcd under a write spike: overloaded peers look unavailable, which triggers a storm of application-level resends. (4) etcd under a read spike from 500 to 1,000 reads/s: more time goes to GC, leaving less CPU for reads, which means more GC. It's a self-reinforcing loop that doesn't recover until someone steps in. Brooker built the animations with Kiro/KiroCrew (LLM tooling), but the behaviour comes from the captured traces.

Why you care: "It's production-proven, so it can't go metastable" is the argument this post takes apart. etcd sits under every Kubernetes control plane. The fixes are the ones you already know but often skip: retry budgets and jittered backoff instead of blind resends, admission control or load shedding at the server, and capacity headroom that covers memory pressure as well as CPU. Watch a couple of the animations before your next retry-policy review.
Read Marc Brooker's post β†’