Skill packs › Skill

Fleet Uptime Monitor

Probe the BlitzAdmin/LSS fleet for HTTP 5xx outages and draft (never send) an alert on a fleet-wide failure. Use for scheduled fleet uptime checks or when asked “are our sites up / is the fleet down”.

Skill file fleet-uptime-monitor.md · last updated Aug 2, 2026

Catch a fleet-wide outage within the run interval instead of by luck. This exists because on 2026-07-21 ~150 client sites returned HTTP 500 and nothing noticed until an unrelated daily job tripped over it. It generalizes the single-site philmershon-uptime-check to the whole fleet.

Hard rules: read-only probing; never send email — drafts only; healthy = ONE line, verbose only when something is actually broken; never write a date/status you didn’t verify.

Step 1 — Build the probe set (union TWO rosters — never trust one)

Read both on the LOCAL Mac via Desktop Commander and probe the union. A single roster is a single point of blindness: on 2026-07-27 the credential cache was found 111 days stale, which had been silently hiding 18 live client sites (incl. piotrzawislak.com, leopohlmann.com, matspohlmann.com) from every uptime probe since they launched.

Roster	Path	Field	Caveat
Credential cache	/Users/dennisyu/Documents/Claude/blitzadmin_site_cache.json	sites[] where status ∈ (active,installed)	Written by refresh-site-cache, which needs an interactive BlitzAdmin browser token — goes stale silently whenever nobody logs in
Publishing targets	/Users/dennisyu/Documents/Claude/Projects/Personal Brand Website/Fleet-Publishing-System/targets.json	targets[]	No token required; this is the roster that keeps moving

• Report the age of BOTH files in every close-out line — healthy runs included. A roster that stopped updating is itself the incident; it must never be able to rot invisibly again. Flag any file >30 days old explicitly.
• Never silently fall back. If one roster is missing/unreadable, probe the other and say which one is gone. If both are gone, probe the 3 anchors alone and say so loudly.
• Probe set = the 3 ALWAYS-anchors plus a random sample of ~15 domains from the union (rotate the sample each run):
• ALWAYS anchors: localservicespotlight.com, blitzmetrics.com, dennisyu.com (blitzmetrics.com + dennisyu.com sit on different IPs and act as controls — if they’re up but a shared-host cohort is down, it’s a host-level fault, not the whole internet).

Step 2 — Probe real HTTP status (free, local curl — NOT DataForSEO units)

Run on the LOCAL Mac via Desktop Commander start_process (never the sandbox). The status CODE is what matters — the 2026-07-21 fatal returned a fully-rendered HTML body WITH a 500, so “did a body come back” is the wrong test: ` for u in <domains>; do printf “%s ” “$u”; curl -sS -o /dev/null -w “%{http_code}\n” –max-time 20 “https://$u/”; done ` Record domain → front-page status.

Step 2b — Cloak probe: EVERY domain, EVERY run, ungated (never behind a 5xx)

A status code cannot see a cloaker. Probe every domain from Step 2 with a second identity and an impossible-path .xml canary:

` GB=’Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)’ curl -sS -o /dev/null -w “%{http_code} %{size_download}\n” -A “$GB” –max-time 15 “https://$u/” curl -sS -w “\n%{http_code} %{size_download} %{content_type}\n” -A “$GB” –max-time 15 \ “https://$u/blitz-canary-9174-do-not-create.xml” `

The canary MUST return 404. A 200 on a path that has never existed means something is intercepting requests in front of WordPress. Alert on either:

• canary body contains product/category, manufacturer-site, or product-similar-image, or
• the Googlebot body is large and its <title> is something the site does not sell (compare Googlebot <title> to the browser <title> — a mismatch is the tell).

Ignore a status split alone: 202/200 with a ~169-byte body and the site’s real title is a CDN bot challenge, not a compromise (two false positives, 2026-07-27).

An EMPTY <title> is not evidence — never score it clean. If either title is "", the discriminator is unusable and the answer is UNRESOLVED, not clean. On a vhost where every PHP route 500s, both bodies are 0 bytes, both titles are "", and "" == "" scores as a match. That is how owenhemsath.com was recorded “clean (titles match, 0 spam markers)” for three days while serving Googlebot a 533 KB spam sitemap (2026-08-01).

Step 2c — robots.txt shape + backdoor path (payload-independent; MANDATORY on the full union)

Run fleet_robots_sweep.py. It exists because the canary and the title fallback both go blind on a vhost whose PHP is dead, and because the spam-marker list is payload-family-specific — the owenhemsath payload carried zero of the three known markers.

Flag on the SHAPE of /robots.txt, never on a payload signature (shape survives a new payload):

• more than 2 Sitemap: directives (a normal WordPress robots.txt has one), or
• any query-string sitemap form (/?sitemapindex.xml, /?sitemap531.xml), or
• any sitemap naming a non-WordPress .php file (/buy.php?sitemap462.xml).

Then probe the advertised path with both identities: an impossible path answering 200 with >50 <loc> is a cloak regardless of which UA sees it.

robots.txt is a static file served by Caddy’s file_server, so it still answers 200 on exactly the vhosts where PHP is dead — it is readable precisely where every other detector fails.

Known backdoor filename: buy.php, with per-site sitemap numbers (462, 602). It is NOT on the 7/31 remediation hunt list (about.php, radio.php, index.php, content.php, lock360.php, admin.php, wp-login.php).

Re-scope whenever you write a new detector. Never carry the old scope number forward — “1 of 137” was reported truthfully every 3 hours while the real figure was 3 of 137.

Why this is Step 2b and not part of Step 3. It was written on 2026-07-27 as a Step-3 addition — gated behind “non-healthy” — and that gate made it dead code. On 2026-07-29 the three infected sites had their HTTP 500s repaired while the injector was left running. They began returning 200 to humans, so they never reached Step 3 again, and this monitor reported them healthy every 3 hours for a day and a half while site:philmershon.com filled with counterfeit-goods listings. The 500 was the disguise; removing the 500 removed the only signal a status-code monitor could see. A check that can only fire on an already-broken site cannot catch a cloaker on a working one — see the standing rule a check that cannot fail is not a check.

Verification after any claimed cleanup is this one command, not the homepage: the canary must 404.

Force-verify list — probe these every run regardless of sampling: philmershon.com, dylan-haugen.com, theathletespotlight.com. These are the 2026-07-27 infection cohort. They stay on this list after they clear, because a clear does not mean safe: this injector reinfects. dylan-haugen.com was verified canary-404 at 2026-07-31T23:35Z and was reinfected by 2026-08-01 ~04:11Z — the cleanup held about 4.5 hours. philmershon.com cleared 7/31T23:35Z, then threw a 500 across every PHP route within ~95 minutes, then recovered again by 08-01T04:40Z. Do not read a green result on any of these three as closed; read it as clean at the timestamp of that probe. Removal from this list requires a human decision after the root cause and persistence mechanism are identified on-box — not a passing canary.

Step 2d — web shell markers on every body, ungated (structural; fleet_shell_sweep.py)

Run fleet_shell_sweep.py. fleet_probe.probe() also scores every body it fetches, anywhere in the system, so this cannot be skipped by forgetting to call it.

Flag on the controls a file manager needs in order to BE a file manager — ≥2 distinct of: function picked(, name="getupload", name="get_url", name="dpath", Current Path :, /wp-sites/live/. Exactly one marker is PARTIAL — a trigger to LOOK, never a verdict.

Never infer identity from a byte count. azuifeachor.com’s shell was prepended in front of the real 251,647-byte homepage; the 00:45Z run compared 251,647 → 9,968 and concluded the big number was the clean page and the shell had just arrived. Both bodies carried all six controls. Read bodies.

A negative-control fixture must come from a different, known-clean subject. Using the site under investigation as the “clean” side is a check that cannot fail — that is how the infected 251,647-byte capture was nearly enshrined as the definition of clean.

Layer discriminators (run all three; two must PASS): a static file served by Caddy’s file_server (robots.txt, a theme/core asset) — clean means the injection is not at the vhost layer; /wp-cron.php — clean means it is not an INI auto_prepend_file; /wp-json/ — poisoned means it is inside the WordPress bootstrap, and means the REST API is emitting invalid JSON to every integration. A cached poisoned copy on disk (WP Rocket’s cached@<epoch> footer decodes to the generation time) dates the compromise and must be purged after removal, never before.

Step 3 — Deep-probe anything non-healthy

Healthy = 200, or 301/302 to the same site. For any domain returning 4xx/5xx (or a timeout), deep-probe to classify the layer. Always probe all six — the two CONTROLS are what make the diagnosis, and skipping them produced a wrong call on 2026-07-27:

Probe	Why
/	the symptom
/wp-json/ and /wp-login.php	main-query vs. non-main-query routes
a static asset (theme CSS or /wp-content/uploads/)	is Caddy/the web server alive
/wp-cron.php — CONTROL	runs wp-load.php, which loads every active plugin + the theme’s functions.php. If anything fataled at load, this 500s too. 200 here disproves “WordPress/PHP is fataling.”
a deliberately nonexistent URL — CONTROL	must return 404. A nonexistent URL has no content to render and no template to fail in, so a 500 here means the HTTP status pipeline itself is broken, not rendering.

Also dig +short <domain> for the IP, so you can spot a shared-host pattern, and record whether the failing response carries the origin’s own headers (via: Caddy, content-type).

Interpretation:

• /wp-cron.php 5xx and static asset 200 → WordPress/PHP really is fataling at load (the 2026-07-21 signature: a network-activated plugin hooking parse_request/wp/template_redirect).
• /wp-cron.php 200 but / and the nonexistent URL both 5xx → NOT a plugin fatal. Every plugin loads fine; the request dies during PHP shutdown / response completion, after content is generated. wp-cron.php escapes it by die()ing early under DOING_CRON. Suspects, in order: a stale persistent object-cache drop-in (wp-content/object-cache.php) pointing at a dead Redis/Memcached; PHP-FPM segfaulting at shutdown (grep -iE 'SIGSEGV|signal 11'); a fatal in a registered shutdown hook; index.php left chattr +i by a prior incident response (check lsattr — a +i file cannot be overwritten even as root, so a “we restored it” step fails silently).
• Failing response has no via: Caddy / no content-type while healthy responses on the same host do, and HEAD returns 200 while GET returns 4xx → an edge/proxy layer in front of the origin is blocking; the request never reaches the site. WAF / bot-mitigation / firewall rule. Fix is in the host or CDN panel — there is nothing to fix inside WordPress.
• Everything including the static asset fails → server/network-level (host or DNS), not application. If it’s one or a few sites, resolve the A-record first: an off-fleet IP means that domain isn’t pointed at us (see the NET-by-breadth learning below).
• Only /wp-json/ or a store path failing → partial breakage, still urgent.

Never report a site healthy on the front page alone. A 500 can return a fully-rendered HTML body — philmershon.com looked perfect in a browser for five days while every crawler and integration saw a 500. Status code is the signal; the nonexistent-URL 404 is the confirmation.

Step 3b — Re-probe the open watchlist, every run

Read /Users/dennisyu/Documents/Claude/Projects/Personal Brand Website/fleet-watchlist.json (create it if absent: {"open": []}). Every domain in open[] gets the full Step 3 deep probe on every run, regardless of sampling — these are known-broken sites with a human on the hook.

• Still broken → one line per site in the close-out with its day-count. Do not re-draft the same alert; escalate the day-count instead.
• Recovered (front page 200/301 and nonexistent URL 404 and, for anything with a cloak signature, the Step-2b canary returning 404) → say so explicitly, note how long it was down, and remove it from open[].
• Never clear a cloak entry on uptime alone. philmershon.com, dylan-haugen.com and theathletespotlight.com came back to 200 on 2026-07-29 with the injector still live; a status-only recovery test would have closed all three. They stay on the watchlist until the canary 404s.
• Any domain that triggers Step 5 gets added to open[] with {"domain", "since", "signature", "owner_thread"}.

This is the point of the watchlist: recovery gets reported automatically, so nobody has to chase status. Boomerang is then only chasing human answers, not “is it back up yet.”

Step 4 — Severity

• All healthy → Step 6 (one line).
• 1 site 5xx → single-site alert.
• ≥3 sites 5xx, OR a shared-IP cohort down while a co-hosted/control anchor is 200 → FLEET-WIDE OUTAGE (the 2026-07-21 pattern). Escalate: expand the sample to ~30 domains to estimate blast radius.
• Any Step-2b cloak hit → P1, outranks every uptime finding, even if every site returns 200. A cloaked site is worse than a down site: it damages the client’s index while looking fine to them. Sweep the full union roster (not the sample) with the canary to establish scope, capture the canary body + both homepage bodies + headers as evidence, and escalate to the web team. Never attempt to clean it over REST — the injector sits in front of PHP, an application password cannot reach it, and poking it destroys evidence.

Step 5 — On any failure: draft + record (NEVER send)

1. Write an incident file INCIDENT-fleet-<YYYY-MM-DD>.md in /Users/dennisyu/Documents/Claude/Projects/Personal Brand Website/ in the style of INCIDENT-fleet-500-2026-07-21.md: the down/healthy table, the layer diagnosis, the shared-IP finding, likely cause, and the next human step (log into wp-admin and bisect network-activated plugins; or read wp-content/debug.log).
2. Stage a Gmail DRAFT to the owner address configured for this agent (connector create_draft, never send) — never a hardcoded address; this skill ships in downloadable packs: subject ⚠️ Fleet uptime — <N> sites returning <code> (<date>); body = the domain→status→layer table, the one-line diagnosis, blast-radius estimate, and the note that download zips under /wp-content/uploads/ may still serve even while pages are dark (QR-to-zip unaffected; QR-to-page dark).
3. Verify the draft exists (list drafts) and report its id.

Step 6 — Healthy close-out (the usual case)

One line only, and it MUST carry both roster ages so a frozen roster can’t hide behind a green result:

Fleet healthy — 18/18 probed 200/301, 18/18 canary 404, shell 0/<N> read <R>/<N> (anchors + 15 of <N> union), checked <time>. Rosters: cache <D>d, targets <D>d.

The canary count is not optional in this line. If it is missing, the run did not check for cloaking and the green result only covers uptime — say that explicitly rather than printing a bare “healthy”.

Neither is the shell count, and it must carry its read coverage as a fraction. A scope number is a property of the instrument as much as of the world: report shell 0/137 (read 128/137), never a bare 0. Domains that were never read are not clean — 2026-08-02 printed “clean 136” while 77 were never fetched. Say which detector produced each number.

Append ⚠️ <roster> <D>d stale for any roster over 30 days. Nothing longer. Do not draft anything — a stale roster is reported here and alerted by refresh-site-cache itself; drafting every 3h would just train you to ignore it.

---

Registered as a Cowork skill 2026-07-25 (fleet-uptime-monitor) and wired to the fleet-uptime-monitor scheduled task (every 3h). Generalizes philmershon-uptime-check; both should call this one rather than re-deriving the probe logic.

Updated 2026-07-27: probe set now unions the credential cache with targets.json, and every close-out reports both roster ages. Cause — a healthy 18/18 run was passing while the credential cache sat 111 days stale, so 18 live client sites had never once been probed. Green results are only trustworthy if you also know the list was current.

Updated 2026-07-30: Step 2b (cloak probe) is now ungated and runs on every domain every run, and the healthy close-out must report a canary count alongside the status count. Cause — the cloak check written on 2026-07-27 was placed inside Step 3, which only runs on non-healthy sites. When the three infected sites’ HTTP 500s were repaired on 7/29 the injector was left running, so they returned 200, skipped Step 3, and this monitor called them healthy every 3 hours. The 09:52 run on 7/30 went further and cleared two of them from the watchlist as RECOVERED on a status-only test. Step 3b now forbids clearing a cloak entry on anything but a 404 canary.

Updated 2026-07-28: /wp-cron.php and a nonexistent-URL-expects-404 are now MANDATORY control probes in Step 3, and the interpretation table branches on them. Cause — this monitor diagnosed philmershon.com and dylan-haugen.com as a plugin-load fatal; /wp-cron.php returning 200 disproved that, and the real fault was shutdown-phase. Added Step 3b: an explicit watchlist (fleet-watchlist.json) of open incidents that is deep-probed on every run, so recovery is reported automatically rather than chased by email.

Learned in the field

Appended automatically by the self-improvement loop (Skill-Learnings/): dated lessons from real runs. Newest at the bottom.

<!– learning:2026-07-21-monitor-false-positives-are-security-failures –> July 21, 2026 (from: sigrun-security-monitor daily run)

A monitor that alerts on something harmless every single day has already failed, because the next real alert arrives in a channel nobody reads. The sigrun.com security monitor scanned rendered HTML for spam words with a plain substring test. On July 20, 2026 the /my-dashboard/ token gateway shipped, embedding an AES-GCM envelope as one unbroken 1,036,296-character base64 run. Random base64 spelled “pORn” inside it. The monitor alerted on July 20 and again on July 21 — four consecutive ALERT lines in security/log.md, every one of them our own page, on a service whose entire purpose was catching the injection a client had found before we did.

Rules for any content-scanning monitor:

1. Strip data before scanning prose. Any unbroken run of 200+ base64 characters is a token, an

inline font, or a data: URI — never text. Substitute it out, then scan.

1. Split markers by class. Domains and CSS hiding patterns (“znaki.fm”, “1xbet”,

“position:absolute;left:-9”) contain punctuation that cannot occur inside base64, so a raw substring test is safe. Plain English words (“casino”, “viagra”, “porn”) need a leading word-boundary against the stripped text.

1. Require a leading boundary, not a trailing one. Injected spam is nearly always inflected —

“casinos”, “pornography”, “viagra-online”. A trailing boundary makes the monitor miss the real attack while still flagging the noise.

1. Keep collision-prone words out of the word list entirely. “escort” fires on legit prose

(“escorted”); if such a term is needed, express it with punctuation as a raw marker.

1. Pin both directions in tests. Every fix to a detector needs a test that the false positive stays

silent AND a test that a genuine injection still fires, including one page that contains both a token blob and real spam. security/test_monitor.py, 16 assertions, runs in under a second.

Diagnose before you escalate: the run’s first move was to locate the exact offending byte offset and print its surrounding context, which showed the match sitting inside our own <template id="smb-vault">. Drafting the alert email the runbook called for would have sent operations chasing our own code. Learned July 21, 2026.

<!– learning:2026-07-26-classify-diffs-dont-just-diff –> July 26, 2026 (from: sigrun-security-monitor daily task)

A monitor must CLASSIFY changes, not just diff them

Applies to any monitoring or audit skill that compares today’s state to a stored baseline — security monitors, fleet audits, SERP monitors, delta audits, Wikidata audits.

What happened. The sigrun.com security monitor fired 16 ALERT lines. All 16 were one thing: WP Engine’s managed auto-updater moved eight plugins forward by exactly one release overnight (Elementor 4.1.5→4.2.0, Yoast 28.0→28.1, and six more). The monitor did a plain set difference on {plugin, version, status} objects, so every bump printed twice — once as the new version “CHANGED/ADDED”, once as the old version “REMOVED.”

Worse than the noise: nothing in the design ever cleared it. The baseline only moves when a human runs --baseline, so the same 16 lines would have printed every morning indefinitely. That is the exact failure that let the July 14, 2026 injection sit unnoticed until a client found it — the alerts were there, nobody was reading them.

The rule. A diff tells you something changed. A monitor’s job is to say whether that change is expected. Any monitor whose baseline can only be refreshed by hand will eventually be refreshed by nobody, and then it is decoration. So for every field you diff, answer up front: what does a legitimate change to this field look like, and can I verify it automatically?

• Verifiable and expected → INFO, and advance the baseline for that one field only.

Plugin version went up and api.wordpress.org confirms that exact version is a published release of that exact slug? Routine. Log it, move the row, stay silent.

• Unverifiable → ALERT. An upstream lookup that failed means we do not know, which is

not the same as fine. Never let a failed check fall through to a pass.

• Expected-shaped but wrong → ALERT loudly, and name the risk in the message. A

downgrade is the tell — that is how a patched vulnerability gets reintroduced. A version string upstream never published is a tampered plugin folder.

• Never softened → the fields that carry the actual threat. New/removed items, status

flips, and anything user- or permission-related stay strict-diff with no exceptions.

Two traps to design against.

1. Auto-advancing the baseline is only safe when the advance is earned. Advance one row

per verified item, never “accept everything I saw today.” Otherwise the first real intrusion silently becomes the new normal.

1. A poisoned wave. Real tampering will arrive hidden inside a legitimate-looking batch.

Test it: take a valid 8-plugin update wave, smuggle in one downgrade and one rogue slug, and assert that both still fire and only the clean seven advance. If your classifier handles a batch as a batch instead of per-item, this is where it breaks.

Receipts: classify_plugins() plus 31 new offline tests (upstream lookup stubbed, so the suite never depends on the network); pre-change baseline retained; the run went from 16 ALERT to exit 0 with 8 INFO lines, then fully silent on the next run. Same underlying lesson as the July 21, 2026 pORn false positive, one layer up: false positives are a security failure, not cosmetics — fix the detector, not the symptom, and pin it with a test in both directions. Learned July 26, 2026.

<!– learning:2026-07-27-a-site-going-down-can-masquerade-as-resolved –> July 27, 2026 (from: weekly-fleet-hub-audit (Fleet-Publishing-System))

Diff on signals, not list membership — a site going down looks exactly like a gap being “resolved”

Week-over-week gap diffs compare list membership in fleet-audit-gaps.json. When philmershon.com and thyrellpardjo.com went unreachable this week they dropped out of needs_hub, which a naive diff reads as “resolved” — when in fact the site got worse, not better.

Rule: before reporting anything as RESOLVED, check that the subject is still reachable. A domain leaving a gap list while its homepage status is non-200 is a regression, not a fix. Cross-check every “resolved” against homepage_up from the same run, and report it as a regression when the site is down.

Same shape as the standing rule “verify the replacement text, not just the absence of the old text”: absence of a problem marker is not evidence of the good outcome. Whenever a check is written as “X is no longer present,” ask what else could remove X, and assert the positive condition instead. Learned July 27, 2026.

<!– learning:2026-07-27-classify-net-failures-by-breadth –> July 27, 2026 (from: weekly-fleet-hub-audit (Fleet-Publishing-System))

A single-site NET result means that domain points off-fleet — not that our network is intercepting

The standing gotcha says an SSL CERTIFICATE_VERIFY_FAILED (NET) means our own network is intercepting traffic (captive portal, e.g. attwifi), and that those sites must never be scored as down. That is correct only when NET hits many sites at once.

On July 27, 2026 exactly one of 91 sites returned NET: thyrellpardjo.com. Checking it directly showed it was not our network at all —

• thyrellpardjo.com A-record → 66.96.162.133 (not the fleet IP 34.199.192.119)
• TLS handshake serves a certificate for a different hostname
• every other fleet site on 34.199.192.119 returned 200 in the same pass

The domain simply is not pointed at our fleet host (moved, expired-and-parked, or never cut over).

Refined rule: classify NET by breadth, then verify.

• NET on many/most sites → our network. Renormalise, never call sites down.
• NET on one or a few sites → resolve the A-record and compare against the fleet IP

before concluding anything. Off-fleet IP + hostname-mismatch cert = a DNS / cutover problem for that domain, and it belongs in the infra queue, not the “ignore, it’s our wifi” bucket.

Neither bucket is a guess: one A-record lookup and one certificate check decide it. Learned July 27, 2026.

<!– learning:2026-07-27-normalise-http-header-case-at-the-boundary –> July 27, 2026 (from: weekly-fleet-hub-audit (Fleet-Publishing-System, impact_fleet.py))

Never read a metric out of dict(resp.headers) — case-insensitivity is lost

urllib‘s resp.headers is an email.message.Message, whose .get() is case-insensitive. dict(resp.headers) is a plain dict and is not. Our fleet host returns WordPress pagination headers title-cased as X-Wp-Total, so every hdr.get("X-WP-Total", default) lookup missed and fell through to its default:

` dict(resp.headers).get(“X-WP-Total”) -> None resp.headers.get(“X-WP-Total”) -> 1682 actual keys: [‘X-Wp-Total’, ‘X-Wp-Totalpages’] `

total_posts collapsed to len(posts) = 10 (the per_page cap) on all 91 sites and posts_30d to 0 on all 91. The fleet reported “0 sites posting in 30 days” while seankelly.io had published 1,028 posts in that window. Content velocity contributed a constant to every Business Impact Score since the 2026-07-19 launch, and the content-dispatch queue was flagging active sites as dormant. It scored wrong for 8 days.

Standing rule for every collector we write: if a value is read out of an HTTP response header, normalise the case at the boundary. Never dict(resp.headers). Applies equally to requests (its own CaseInsensitiveDict is safe — but dict(r.headers) throws that away), http.client, and urllib. Fix shipped as a CIHeaders(dict) class that lowercases keys on ingest and on lookup (get, __getitem__, __contains__); post-fix verification: seankelly.io 1,682 total / 1,028 in 30d.

Companion rule — a defaulted metric must be distinguishable from a real zero. hdr.get(key, 0) returning 0 for “header absent” is indistinguishable from a site that genuinely posted nothing. Uniform values across an entire fleet are the tell. Any collector that produces a fleet-wide constant (0 everywhere, or the page-size cap everywhere) should fail loudly rather than score: if >90% of rows return the identical value for a metric that should vary, print a [WARN] line instead of publishing it. Learned July 27, 2026.

<!– learning:2026-07-28-a-control-probe-beats-a-confident-diagnosis –> July 28, 2026 (from: fleet-uptime-monitor scheduled run)

Name the probe that would DISPROVE your diagnosis, and run it before you send

This monitor found philmershon.com and dylan-haugen.com returning 500 on /, /wp-json/, /wp-login.php and /wp-content/uploads/, with theme CSS at 200. It concluded “WordPress/PHP is fataling at plugin load or core boot” and wrote that into an incident file and an alert. The reasoning was clean and the evidence was real. It was still wrong, and it would have sent whoever picked it up hunting for a plugin load error that does not exist.

One probe settles it: /wp-cron.php returned 200. That path runs wp-load.php, which loads every active plugin and the theme’s functions.php. If anything fataled at load, it would 500 too. It didn’t — so nothing is fataling at load. The request dies later, during PHP shutdown / response completion, after the content is already generated. wp-cron.php escapes it by die()ing early under DOING_CRON.

The second control is even cheaper: request a URL that does not exist. It must return 404. A nonexistent URL has no content to render and no template to fail in, so a 500 there means the HTTP status pipeline itself is broken — not rendering, not a plugin. Both fleet sites returned 500 on a garbage URL.

The generalisable rule. A diagnosis built only from failing probes is a hypothesis wearing evidence as a costume. Every failing probe is consistent with many causes; what discriminates between them is a probe you predict will pass. So before writing a cause into a report:

1. State the diagnosis as a prediction — “if it’s a load-time fatal, then X must also fail.”
2. Find the cheapest X that the diagnosis says must fail but a rival diagnosis says will pass.
3. Run it. If X passes, the diagnosis is dead — do not ship it with hedging.

Corollaries that fell out of the same run:

• A 5xx can carry a fully-rendered body. philmershon.com served a complete homepage — hero,

six live posts from MySQL, testimonials, footer — under its 500 for five days. It looked perfect to every human who visited. Only crawlers, monitors and integrations saw the failure. Any check that asks “did content come back” passes here. That is precisely how it ran unnoticed to day five. (It then degraded to zero bytes an hour after the first report — so re-probe before restating a body observation, too.)

• Missing headers are evidence. Four other sites returned 403 with **no via: Caddy, no

content-type, no alt-svc* — while healthy *and 500 responses from the same host all carried them. The origin’s fingerprint being absent proves the response was manufactured by an edge layer in front of the origin and the request never arrived. HEAD on the same URL returned 200. Read what a response doesn’t have, not just its code.

• Same code ≠ same incident. Nine sites were down in three unrelated ways (shutdown-phase 500,

edge 403, off-fleet DNS) plus one deliberate maintenance 503. Bucketing them by status code would have sent one owner down three wrong paths.

Fixed in this skill: /wp-cron.php and a nonexistent-URL-expects-404 are now mandatory controls in Step 3, the interpretation table branches on them, and Step 3b re-probes an explicit watchlist of open incidents every run so recovery is reported automatically instead of chased by email. Learned July 28, 2026.

<!– learning:2026-07-28-a-checker-with-a-literal-in-it-cries-wolf –> July 28, 2026 (from: skill-pack-propagation daily run, July 28, 2026 (verify_directory_and_somba.py))

A checker with a literal in it eventually checks the wrong thing

The verifier written this same run failed twice against surfaces that were perfectly healthy:

1. It asserted the live bundle was Sigrun-Agent-Pack-1.zip. A second publish cycle two hours

later made it -2, and the checker reported four failures on a correct set of pages.

1. It then checked every date against one constant, Jul 28, 2026. Three formats render across

these surfaces — the directory abbreviates the month, the members’ “Kept current” badge spells it out (July 28, 2026), and the top bar uses 28 Jul 2026. One more false failure.

Both are the July 26 lesson (“verify the format each surface actually renders”) arriving from a new direction: the thing that went stale was the verifier, and a verifier does not get verified by anything.

Rules:

1. Derive expectations from the artifact of the run you are verifying — read the upload

manifest, the VERSION.txt, the config — never from a value you typed while writing the check.

1. A checker that cries wolf is worse than no checker. After two false alarms nobody reads

the third, and the third is the real one. Treat a false failure as a P1 on the checker.

1. Check per surface, not per system. If two pages render a date differently, that is two

checks. Collapsing them into one constant guarantees one of them is wrong.

1. When a check fails, prove the SUBJECT is broken before you touch it. Both failures here

would have “fixed” a healthy page had the message been taken at face value.

Learned July 28, 2026.

<!– learning:2026-07-28-a-user-agent-without-its-escort-is-still-a-robot –> July 28, 2026 (from: skill-pack-propagation daily run, July 28, 2026)

Sending a full Chrome User-Agent is necessary but no longer sufficient to get past a Cloudflare-fronted WordPress site. On July 28, 2026 the REST POST to blitzmetrics.com/task-library-dashboard returned 403 with a “Just a moment…” interstitial while the POST to blitzmetrics.com/build-agents — same site, same app password, same full UA — returned 200. The obvious reads (expired app password, WAF hates this page, transient rate limit) were all wrong.

Isolate before theorizing, and isolate with a no-op probe: POST the page’s own current content back to it, unchanged, so the request is real but the page cannot be damaged. Run it twice each way. Result was deterministic — UA-only 403, 403, full browser header set 200, 200. That rules out a cooldown, which a single retry never can: a retry that succeeds after a pause is indistinguishable from a fix that works.

The rule: a browser UA arriving without the headers a real browser sends beside it is itself a bot signature. Send the escort — Accept, Accept-Language, sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform, Sec-Fetch-Dest/Mode/Site, plus Origin and Referer on writes. Put them in one constant applied to every request, not to the one page that failed; the next surface to trip the rule should cost nothing. In republish_daily.py this is BROWSER_HEADERS.

Two corollaries worth carrying:

• A 403 is not an authentication failure. Verify auth with GET /wp-json/wp/v2/users/me

before touching credentials. A working GET plus a failing POST means the body or the headers, never the password.

• For uptime probing, a bot-shaped request can manufacture an outage. A monitor that

probes with a bare UA can collect 403s from a perfectly healthy site and report a fleet-wide failure. Probe with the same full header set, and treat a 403 carrying a challenge interstitial as inconclusive — never as down, never as up.

<!– learning:2026-07-29-a-check-that-cannot-fail-is-not-a-check –> July 29, 2026 (from: redirect-chain audit, July 29, 2026)

A check built on a branch that can never run reports perfect health forever

Two scripts written the same day counted redirect hops like this:

try: r = urllib.request.urlopen(req) # HEAD return hops, r.status except urllib.error.HTTPError as e: # <– catch the 301 here …follow Location, hops += 1

That except can never fire. urlopen installs HTTPRedirectHandler by default and follows 301/302 transparently, returning the FINAL response. So the counter returned 0 hops for every URL on earth — including one we had already proved by hand was a double 301. The first audit printed “34 rules · 0 chained” and read as good news.

Fixing the counter turned that into: 6 chains on one site, 18 on another, one self-redirecting rule with 12,190 hits looping forever, and the busiest rule on the site (829,576 hits) throwing away a trailing slash and buying a second hop on every single request.

Rules:

1. Before trusting a checker, make it fail on purpose. Feed it a case you KNOW is bad. If it

passes, the checker is broken — not the world. Both scripts now call a selftest() that fetches a URL known to redirect and aborts the whole run if it measures zero hops.

1. Any library call with “convenience” behaviour is a checker’s enemy. Following redirects,

retrying, normalising, caching: all of it hides the exact signal a verifier exists to see. Measure at the lowest level that still answers the question.

1. A try/except around a network call deserves the same scrutiny as the happy path. An

except clause that cannot be reached is dead code that looks like diligence.

1. Zero is a suspicious answer. Zero orphans, zero chains, zero errors on the first run of a

brand-new check almost always means the check is not wired to anything. Confirm one true positive exists before believing the zero.

Learned July 29, 2026.

<!– learning:2026-07-30-a-repaired-symptom-is-not-a-repaired-incident –> July 30, 2026 (from: fleet cloak re-probe during a seankelly.io article wave)

A repaired symptom is not a repaired incident — and never gate a detector behind the symptom

Applies to any monitor whose deep checks are conditional on a shallow signal.

What happened. On 2026-07-27 three fleet sites were found serving product-spam to Googlebot while returning HTTP 500 to humans. A cloak check was written the same day and filed under Step 3, which by design only runs against sites that are already non-healthy. On 7/29 the 500s were repaired and the injector was not. All three began returning 200, so they stopped qualifying for Step 3, and this monitor reported them healthy on every 3-hour run for a day and a half. The 7/30 09:52 run then cleared two of them from the watchlist as RECOVERED, on the strength of / 200 + wp-cron 200 + ghost-URL 404 — all true, all irrelevant to a cloaker. By the time anyone re-probed by hand, site:philmershon.com returned ten counterfeit-goods listings and zero real pages.

Rules:

1. A detector must not depend on the condition it is meant to survive. If the only path to the

cloak check is “site is down”, then fixing the downtime disables the cloak check. Run detectors unconditionally on every target, every run.

1. **The visible symptom is often the attacker’s disguise, and removing it is a regression in

observability.** Going from 500 to 200 made these sites look better and made us blinder. Treat “the symptom went away without anyone explaining the mechanism” as an open question, not a close.

1. Define a positive clear condition per signature, and never accept a weaker one. Uptime clears

an outage. Only a 404 on a path that never existed clears a cloak. Write the clear condition into the watchlist entry so a later run cannot invent an easier test.

1. **Trust a third party’s “it’s fixed” for the part they touched, and re-verify the part they

didn’t.** The uptime repair was real and worth thanking someone for. It said nothing about the injector.

1. Scope before escalating. A full-roster canary sweep (193 domains, ~4 chunked calls) turned

“three sites” into a bounded fact rather than an assumption, which is what makes an escalation actionable instead of alarming.

Learned July 30, 2026.

<!– learning:2026-08-01-a-check-that-cannot-pass –> August 1, 2026 (from: fleet-uptime-monitor scheduled run)

A check that cannot PASS is the same bug as a check that cannot FAIL — and the probe engine must self-test before it is allowed a verdict

This run’s first pass reported 33 of 33 domains unreachable (000). Read at face value that is a total fleet outage, far worse than 2026-07-21, and the draft would have gone out saying so.

It was entirely false. The probe engine read curl’s status line with

-w ‘@@META %{http_code} %{size_download} %{content_type}’ re.search(r’@@META (\d+) (\d+) (\S*)\s*$’, out)

A real content type contains a space — text/html; charset=UTF-8. (\S*) matched text/html;, \s*$ then failed against the remaining charset=UTF-8, the whole match failed, and the code fell through to its ('000', 0, '') default. Every healthy site scored 000. The only domains that parsed correctly were the four edge-403 sites, because a zero-byte 403 carries an empty content-type — so the sites that were genuinely broken were the only ones reported accurately, and the ones that were fine all looked dead. One serial curl disproved it in a single command.

This is the 2026-07-29 lesson (a check that cannot fail is not a check) arriving inverted. Both are the checker breaking rather than the world, and both are invisible because the output is a plausible number rather than an exception.

Rules:

1. Run a self-test before every verdict, not once when the script was written. selftest() now

fetches a control anchor and a URL known to 404, and sys.exits without emitting a verdict if the control is unreachable. A monitor that cannot reach a known-good site has learned nothing about the fleet and must say so instead of reporting.

1. Pin the checker in both directions on live data. The parser fix was verified against a known

200, a known 404, and a known 500 before the sweep ran. A parser validated only against failures is how this bug survived — the failing cases were the ones it handled correctly.

1. **A default value returned by a failed parse must never be indistinguishable from a real

measurement.** ('000', ...) on a regex miss looks exactly like a network timeout. Either raise, or tag the value as PARSE_FAIL so it can never be aggregated into a status count.

1. A fleet-wide uniform result is a checker smell, not a finding. 33/33 identical is the same

tell as the 2026-07-27 “0 posts on all 91 sites” header-case bug. Before believing any fleet-wide constant, reproduce ONE row of it by hand with the simplest possible tool.

1. Never let a convenience wrapper shorten the diagnosis. The confirming test was

curl -sS -o /dev/null -w "%{http_code}" https://dennisyu.com/ typed straight into a shell — no parser, no threads, no script. Keep one such command in the runbook for exactly this moment.

Corollary, from a second self-inflicted false positive in the same run. The follow-up gap-checker ran redirect-following probes in a ThreadPoolExecutor while every worker wrote to the same /tmp/_f.body. Bodies interleaved, and seven healthy sites appeared to serve each other’s <title> — blitzgifts.com titled “Cecelia Leger”, matspohlmann.com titled “Tommy Mello”. Cross-site title bleed is precisely the cloak signature, so this nearly became a seven-site P1. Re-running serially with per-call temp files showed every canary title matching its own homepage. Any parallel checker must give each worker its own output path, and a title-comparison result from a run with shared state is inadmissible. When a detector fires, prove the subject is broken before you write it down — here, the second run cost ninety seconds and saved a false escalation. Learned August 1, 2026.

<!– learning:2026-08-01-a-detector-blind-spot-is-a-scope-lie –> August 1, 2026 (from: fleet-uptime-monitor 16:45Z scheduled run)

When a detector goes blind, it does not report “blind” — it reports CLEAN. And the scope number it produces becomes a lie you repeat every three hours.

owenhemsath.com had been on the watchlist for three days as a plain uptime failure — “whole-vhost 500”. It was also serving Googlebot a 533 KB spam sitemap with 3000 <loc>, every URL pointing at the client’s own domain, <lastmod> set to tomorrow, regenerated per request. Three separate detectors reported it clean, every three hours, for three days:

1. The canary needs a 404 it can never get. Every PHP route on that vhost 500s, so the canary

returned 500 and the site was filed cloak-check-degraded — correctly — and handed to the fallback.

1. The fallback compared two empty strings. It compares the Googlebot <title> to the browser

<title>. Both were "", because both responses were 500/0 bytes. "" == "" evaluated true and the row was written as “clean (titles match, 0 spam markers).”

1. The spam-marker list is payload-specific. This payload contains zero occurrences of

product/category, manufacturer-site, product-similar-image. Different payload family from dylan-haugen’s. The content test could not have fired even on a readable body.

Each failure was individually reasonable. Together they produced a confident green.

Rules:

1. A comparison whose operands can both be empty is a check that cannot fail. If either side of a

discriminator is null/empty/zero-length, the answer is UNRESOLVED — never clean. This is the 2026-07-29 lesson (a check that cannot fail is not a check) arriving through a third door, after the 2026-08-01 inverted form (a check that cannot pass). All three are the checker breaking rather than the world, and all three are invisible because the output is a plausible verdict rather than an exception.

1. **A degraded detector must escalate, not fall back to a weaker test and then report the weaker

test’s answer as if it were the strong one.* “Cloak-check-degraded” was the honest signal and it was *already being computed — then it was overwritten by a fallback that could not fail. When the primary signal is unavailable, the correct output is a different DETECTOR, not a softer threshold.

1. Every content-signature list is payload-specific and will age out. Prefer detectors that key

on STRUCTURE, which survives the attacker changing payloads: an impossible path answering 200; a robots.txt advertising seven sitemaps; a sitemap naming a non-WordPress .php file. The three spam markers were harvested from one payload family in July and were useless against the August one.

1. Find the surface that is still readable when the usual one is dead. PHP was dead on this

vhost, so every PHP-served probe was uninformative. robots.txt is a static file served by Caddy’s file_server — it answers 200 precisely where PHP-based detection goes blind, and on this site it was the attacker’s own beacon, listing the backdoor by name.

1. Re-scope whenever you write a new detector; never carry the old number forward. “Scope

unchanged — 1 of 137″ was true of the canary and false of the fleet. The new robots/backdoor detector, selftested to fire on two known-infected sites and stay silent on three known-clean ones, returned 3 of 137 on the same roster in the same hour. A scope figure is a property of the instrument as much as of the world, and reporting it without naming the instrument is how a reassurance becomes a lie.

1. A wrong diagnosis hides its own evidence. “Whole-vhost failure — even static files 5xx” was an

artifact of the probe path: the static probe used a jquery file that does not exist on that install, so the miss fell through to PHP and 500’d. One static file that does exist — robots.txt — returned 200 with an etag, proving the web server was alive and only WordPress was dead. The wrong diagnosis had also been the reason nobody looked at robots.txt: if the whole vhost is dead, there is nothing to read. Re-test the premise of a diagnosis before you inherit it, especially one that implies “there is nothing here to look at.”

Receipts: fleet_robots_sweep.py (new, selftested both directions before it was allowed a verdict); fleet_union_canary.fallback() now returns UNRESOLVED on an empty title and was regression-tested against the exact site it previously cleared; fleet_probe._paths() finally moved from a shared _seq counter to uuid4 per call — that hardening had been documented on 2026-08-01T10:04Z but never written to disk, so the race that manufactured the seven-site false cloak alert was still live in the file. A fix recorded in a note and not in the code is not a fix. Learned August 1, 2026.

<!– learning:2026-08-01-a-new-detectors-first-false-positive-is-waiting-in-the-subject –> August 1, 2026 (from: fleet-uptime-monitor 19:04Z scheduled run)

The structural detector you just wrote has its own false positive, and the subject manufactures it for free

fleet_robots_sweep.py was written at 16:40Z to key on STRUCTURE rather than payload signatures, precisely because the marker list had aged out. Correct call. Three hours later it produced its first misleading row: azuifeachor.com‘s advertised backdoor path /buy.php?sitemap602.xml, recorded at 16:45Z as “no live payload”, came back 200 / 140,608 B to both identities. On the face of it that is owenhemsath’s signature — the same planted robots.txt, the same buy.php filename, now apparently serving something.

It is the client’s own WordPress 404 template, returned with a 200 status. Hand-probing settled it in ninety seconds:

` /buy.php?sitemap602.xml GB & BR -> 200 140,470 B title “Page not found – Azu Ifeachor” /blitz-canary-9174-do-not-create.xml GB & BR -> 200 140,470 B same template, same byte count /definitely-not-a-real-path-8823/ GB & BR -> 404 147,192 B same template, correct status body contains 17 references to azuifeachor.com/wp-content, 0 <loc>, 0 spam markers `

A soft-404 catch-all that returns 200 for file-shaped paths. Every impossible path on that vhost answers 200 with the same bytes, so the detector’s positive signal — an impossible path answering 200 — is generated by the subject’s own misconfiguration, on a site that happens to also carry a real beacon. Scored on status alone it becomes a second P1 and the escalation stops being believed.

Rules:

1. A structural detector still needs a content discriminator on the positive branch. “Impossible

path answers 200″ opens the question. What the body is answers it: a spam sitemap carries <loc> entries and links off-site; a soft-404 carries the client’s own wp-content URLs and their own page title. fleet_robots_sweep.py already gates its backdoor_hit on >50 <loc>, which is why it did not fire — the row was still flagged on robots shape, correctly, and the number was the thing that could mislead a reader. Write the discriminator into the REPORT, not just the code.

1. Budget for the new detector’s first false positive on the day you ship it. Every detector

written so far has produced one within a week: substring spam matching (2026-07-21), plugin-version diffing (2026-07-26), shared temp files (2026-08-01). A brand-new detector’s first surprising result deserves a hand probe before it deserves a draft.

1. Two independent detectors agreeing is what makes a RECOVERY reportable. dylan-haugen.com was

called recovered this run only because the canary passed all four written clear conditions AND the robots/backdoor sweep — which had flagged it at 16:40Z — stopped flagging it at 19:11Z. Compare 2026-07-30, when a single status-only test cleared two sites with the injector still live. On the alerting side we already require a probe that would disprove the diagnosis; the clearing side needs the same standard, because a wrong clear is silent and a wrong alert is loud.

1. Resolve every UNRESOLVED inside the run that produced it. Six came out of the canary pass; four

were resolvable by hand in minutes (an edge-403 that blocks the canary too; a SeedProd coming-soon splash with a catch-all route; a bot-mitigation split serving Googlebot zero bytes — the wrong direction for a cloak; a soft-404 redirect to the homepage). An UNRESOLVED carried across a run boundary decays into next run’s assumed-clean, which is the 2026-08-01 empty-title bug wearing a different hat.

Receipts: INCIDENT-fleet-2026-08-01.md sections R–X; evidence-2026-08-01/1900utc/; watchlist clear_condition_rule extended with “a 200 on an impossible path is a trigger to LOOK, not a verdict.” Learned August 1, 2026.

<!– learning:2026-08-02-a-clean-verdict-has-a-shelf-life –> August 2, 2026 (from: fleet-uptime-monitor 22:43Z scheduled run)

A clean verdict has a shelf life, and on a site with a planted artifact it is minutes

azuifeachor.com was examined twice in one run. At 00:28Z it served its real 251,647-byte WordPress homepage and its advertised backdoor path returned the site’s own 404 template; the soft-404 control probe was run correctly and the verdict written was “dormant beacon, no live payload” — true, and obsolete eight minutes later. At 00:36Z the same URL served a 9,968-byte unauthenticated PHP web shell — directory browser, arbitrary file upload, remote-fetch-by-URL, delete, read, no login — with last-modified: 00:32:56Z. The attacker was working on the box while the monitor was running.

It was caught only because the deep probe happened to re-fetch the homepage twenty minutes after evidence capture and the byte count moved from 251,647 to 9,968. That was luck. The whole point of this skill is to stop catching things by luck.

The mistake was not the soft-404 test, which was sound. It was the category: a site with a confirmed attacker-planted robots.txt had been filed as dormant rather than compromised, mechanism unknown. “No payload right now” was recorded as “no payload.”

1. An unexploited foothold is not an absent one. This is 2026-07-30 (*a repaired symptom is

not a repaired incident*) inverted. A planted artifact is a receipt for access that was never revoked; the correct state is compromised, dormant — never clean. Anything with a planted artifact is a live incident until a human proves the access is gone.

1. Re-probe every flagged domain at the END of a run, not only at the start. A long run is a

window an attacker can act inside. Cheap: one GET per flagged domain. This run’s flagged set was two.

1. A large byte-count swing on a known-compromised host is a first-class signal. No parsing

needed — 251,647 → 9,968 on the same URL in twenty minutes is the entire finding.

1. Read what a shell leaks, then stop touching it. The breadcrumb disclosed

/wp-sites/live/<domain>/public — the layout for every vhost on the host. That is worth recording. Interacting further is not: the upload/delete/fetch controls are the attacker’s tooling, and using them destroys evidence. GET the homepage, capture, escalate.

Corollary — a fix that lives in a run script, not the shared engine, is not a fix. 307/308 were dropped from the healthy-redirect set, so an ordinary apex→www redirect scored as an outage. That exact bug had been found, fixed, and written into probe_engine_note at 08:35Z the same day — but the note was prose and the fix was in a script that got rewritten from scratch at 22:46Z, so it came straight back sixteen hours later. This is the 2026-08-01 rule a fix recorded in a note and not in the code is not a fix, one level up. is_front_healthy() now lives in fleet_probe.py and is pinned by test_fleet_probe.py (24 offline assertions, no network, sub-second, both directions).

Second corollary — a detector’s request volume is part of its design. A rewrite that correctly fixed a hardcoded backdoor literal also raised per-domain volume ~4× (6 paths × 2 identities × 2 tries). The origin stopped answering partway through, 77 of 137 robots.txt came back unreadable — including an anchor its own selftest had read four minutes earlier — and the summary printed “clean 136”, counting 77 never-read domains as clean. That is the 2026-08-01 blind-spot lesson reproduced by the file written to fix it, within seven hours. Two tells were free: the unreadable list was alphabetically contiguous (real breakage does not sort), and a hand curl disproved it in one command. Compare requests-per-domain against the version you are replacing, and ship the throttle in the same change.

Third corollary — “readable” must mean a body was parsed, never “the socket did not error.” A 301, a 500 and a zero-byte 200 are all non-reads wearing a status code. 24 of 137 sat in that bucket, so honest coverage was 106/137, not 134/137 — and three cache-masked sites were scored clean on an empty 500 body. This is the empty-operand bug for the third time (empty <title>, 2026-08-01; unreachable except, 2026-07-29). Following one redirect hop recovered 15 domains and confirmed 2 as no robots.txt exists (a definitive negative), reaching 128/137 with 0 newly suspicious.

Finally — incident files are APPEND-ONLY. This run wrote INCIDENT-fleet-2026-08-01.md in rewrite mode and destroyed sections A–X from five earlier runs that day. No git, no snapshot, no backup. The 19:04Z learning above cites “sections R–X” as its receipts and those citations are now dangling. The substance survived only because those runs also mirrored their findings into fleet-watchlist.json, which is an argument for continuing to write structured state as well as prose. Never rewrite an incident file; append a dated section. Learned August 2, 2026.

<!– learning:2026-08-02-a-negative-control-drawn-from-the-subject-cannot-fail –> August 2, 2026 (from: fleet-uptime-monitor 04:45Z scheduled run)

The infected site was almost enshrined as the definition of “clean” — and a byte count was doing the work a body should do

This run set out to re-create a detector, not to find anything. The web-shell detector that found azuifeachor.com at ~00:35Z had lived in a /tmp script and was gone by 04:00Z — a whole DETECTOR lost to “a fix that lives in a run script is not a fix”, within twenty-four hours of that rule being written into this file. Only its output survived. So the markers were moved into fleet_probe.py and pinned with an offline test in both directions.

Choosing the fixtures is where it got interesting. The obvious negative control was the artifact the 00:45Z run had labelled “the site’s REAL 251,647-byte WordPress homepage, served at 00:28Z, eight minutes before the shell appeared.” The test failed: that body carries all six shell controls at offsets 16–1150, with the genuine <title>Azu Ifeachor…</title> page concatenated after them. The shell was already being served at 00:28:40Z.

The earlier run had compared 251,647 → 9,968 bytes and read the big number as “the real page” and the drop as “the attacker planting the shell during our run”. Decoding WP Rocket’s own footer settles it: the 00:28Z capture says cached@1785571469 = 2026-08-01T08:04:29Z, matching its own last-modified header — so the poisoned response was already generating page-cache entries ~16½ hours earlier. The byte drop was the cache turning over, not an attacker typing. Public exposure was 20+ hours, not minutes.

Rules:

1. A negative control drawn from the subject under investigation is a check that cannot fail.

If the “clean” side of a comparison comes from the site you are investigating, a positive result redefines clean to include the compromise. Controls come from a different, known-clean subject. This is the empty-operand family (empty <title> 2026-08-01; unreachable except 2026-07-29; unreadable-counted-clean 2026-08-02) arriving through a fourth door.

1. Size is not identity. A byte count cannot distinguish “the real page” from “the real page with

a shell stapled to its front”. Any detector whose positive branch rests on a length comparison is measuring the wrong quantity. Read the body.

1. Writing the test is how you find out the evidence was misread. Nothing was re-probed to

discover this — the bytes had been on disk for four hours. The test simply asked a question of them that no one had asked. Budget for fixture-writing as investigation, not bookkeeping.

1. Timestamps inside the payload beat timestamps in your notes. cached@<epoch>, last-modified

on a cached artifact, and mtimes date a compromise independently of when a monitor noticed it. Decode them before writing a timeline; “found at” is not “started at”.

1. A detector that only sees one layer must say so in the same sentence as its number. The

robots/backdoor sweep reported azuifeachor.com: beacon only, 0 live payload — correct, and structurally blind to the file manager that same vhost was serving on every PHP route. Three detectors ran this hour and returned 1, 2 and 0 of 137. None of them is “the” number.

1. Re-derive the layer, don’t inherit it. “Planted static file in the docroot” was inherited from

the prior run; a docroot file cannot prepend itself to /wp-json/. Three probes that had to pass relocated it: static files clean (not Caddy — unlike the 7/27 cohort), /wp-cron.php clean (not an INI auto_prepend_file), /wp-json/ poisoned (inside the WordPress bootstrap — and therefore the REST API is emitting invalid JSON to every integration, which nobody had noticed because the status code was 200).

Receipts: fleet_probe.SHELL_MARKERS + shell_verdict() scoring every body the engine fetches; fleet_shell_sweep.py (new, live selftest in both directions, refuses to emit a scope number if the control anchor is not clean-and-readable); fleet_sweep.py no longer re-implements the healthy set inline (its bare code test scored an off-site 301 as healthy — archiepadley.com would have been reported healthy while open on the watchlist); fleet_probe.__main__ no longer advances the rotation cursor, which had made the sample it printed differ from the sample probed by 15 domains; test_fleet_probe.py 45 offline assertions, both directions, sub-second. Learned August 2, 2026.

<!– learning:2026-08-02-an-escalation-recorded-is-not-an-escalation-made –> August 2, 2026 (from: fleet-uptime-monitor 10:04Z scheduled run)

A draft ID written into state is not a draft. The run that wrote “a day-count is not an escalation” then failed to escalate its own P0.

At 10:20Z the previous run found the most severe thing this monitor has ever measured: 67 of 137 client sites publicly serving /wp-config.php.bak with DB credentials *and all four auth salts. It wrote the finding into fleet-watchlist.json and into the incident file, both ending Draft r3479164600282298524 staged, NOT sent.

That draft does not exist. A full listing of every draft newer than one day returns ten drafts and no config-leak draft at any timestamp; the newest fleet draft was the azuifeachor P1 from 00:43Z. So a P0 credential exposure sat for an hour believing itself escalated, seen by nobody.

Step 5.3 of this skill already said to prevent exactly this — “verify the draft exists (list drafts) and report its id.” It was skipped, so a fabricated ID entered state unchallenged and was then repeated by the next reader as established fact.

The irony is the lesson. That same 10:20Z run wrote never_escalated_2026_08_02: “a watchlist day-count is not an escalation. Carrying a site correctly for three days is not the same as telling anyone.” It then did the identical thing one level up, to its own worst finding, in the same file. Writing a rule about a failure mode does not immunise you against it; only a check that runs does.

Rules:

1. Verify every escalation against the mailbox, never against the note claiming one was made.

The write and the confirmation must come from different sources. create_draft returning an ID is the write; list_drafts finding it is the confirmation. One without the other is a claim.

1. A state file is not evidence of an action; it is a claim about one. This is the

2026-08-01 rule a fix recorded in a note and not in the code is not a fix, applied to escalation. Same family as the empty-operand bugs: the output is a plausible sentence rather than an exception, so nothing surfaces.

1. A self-reported timestamp ahead of the file’s own mtime is an integrity smell. The

watchlist said updated: 10:20Z while its mtime was 09:43Z. Free to check, and it is how “staged at 10:20Z” acquired credibility it had not earned. Compare claimed timestamps to mtimes when reading inherited state.

1. Re-scope every carried number when the ROSTER changes, not only when the DETECTOR changes.

The existing rule covers new detectors. The 09:30Z roster fix grew the union 137 → 174, which silently invalidated the 67/137 scope in the other direction. Re-running gave 68 of 174 (read 159/174) and surfaced brendanking.ca — a site the stale pending_dns filter had hidden since April, leaking the whole time, never once probed. An instrument’s scope is a property of the population as much as of the instrument.

1. The structural detector’s false positive arrived on schedule, again. digitalsocialhour.com

flagged ROBOTS_BEACON_ONLY on a bare count of 4 sitemap directives — with zero query-string forms, zero .php sitemaps, all four sitemaps on its own host, 6,395 legitimate on-domain <loc>, behind Cloudflare and not even on our IP. A podcast site with a video sitemap. Cleared by hand in two minutes inside the run that produced it. Genuine scope is 2 of 174, not 3. A count alone is not evidence — the positive branch needs the content discriminator, and the discriminator belongs in the REPORT, not only in the code.

Receipts: INCIDENT-fleet-2026-08-02.md Section E (appended; A–D verified intact afterwards); evidence-2026-08-02/1015utc/; watchlist keys escalation_integrity_2026_08_02_1105Z, P0_config_leak_2026_08_02_1105Z, detector_false_positive_2026_08_02_1105Z; corrected draft r-7159869211732415826, staged 11:01:57Z and verified by listing before being written down. Learned August 2, 2026.

---

Other skills: ai-search-visibility · boil-the-ocean · client-access-checklist · client-relationship-cadence · content-agent · content-factory

The full run order is on the skill-pack directory. Every skill here is one task from the Task Library — the library is the catalogue of what can be done; a pack is the subset you install; an agent is who runs it.