Chris Butcher, Python automation

Broken-scraper triage checklist

The problem

A scraper broke and the cause is unknown. The output is empty, partial, duplicated, or stuck, and rerunning it only repeats the failure.

The artefact

A six-section checklist worked top to bottom: selector drift, empty fields, pagination loops, rate limits, login walls, and output checks. Stop at the first section that explains the symptom, fix that one thing, rerun, and check the output again. The checklist and a worked sample report are copied under files on this page.

Input and output sample

Sample report, from the HN defect:

Symptom: 30 rows extracted, submitter column empty on every row,
test suite passing.
Section 2 (empty fields): one column at 100 percent empty while
rows exist means selector problem, not data problem.
Section 1 (selector drift): the .hnuser class is gone from the
live page, so the selector matched zero elements.
Fix: structural selector a[href^="user?id="], plus a test that
fails when the selector stops matching.

What can break

The fix or design choice

The checklist orders the checks so the cheapest explanation wins first, and the sample report shows the path taken on a real defect: section 2 caught the symptom, section 1 named the cause. Green tests against a fixture prove the code matches the fixture. Only a live-page check proves it matches the site.

Verification log

Verified 2026-11-19

Files

Price shape

Paid scraper diagnostic $150 to $300, triage only, with a written acceptance check: selectors that still match, a sample CSV, and a short failure log. If the sample matches your columns, the full scraper quote follows in writing.

Start here

Email the target URL and describe the symptom: empty fields, partial rows, duplicates, or a stuck run. I reply with the diagnostic scope and price.