HN scraper defect: passing tests, empty column
The problem
A scraper can pass every test and still ship broken data. The Hacker News front-page scraper extracted the top 30 stories with rank, title, domain, points, submitter, and comment count. The test suite passed. The submitter column in the output CSV was empty on every row.
The artefact
The full pipeline, its CSV output, and the verification log live on the artefact store. The triage write-up is copied here as sample-triage-report.md.
Input and output sample
Input: the live Hacker News front page, top 30 stories.
Output before the fix: 30 rows, submitter empty on all 30. Output after the fix: 30 rows, submitter populated on every row.
What can break
- Selector drift: the submitter selector relied on a CSS class,
.hnuser, that is no longer present on live Hacker News - Silent empty fields: the selector matched zero elements, every row wrote an empty string, and nothing raised an error
- Stale fixtures: the mocked tests passed because the test fixture still carried the old class, so the suite verified the code against a page shape the live site no longer serves
The fix
Replaced the class selector with a structural attribute selector, a[href^="user?id="], which matches the submitter link by its URL pattern instead of a presentation class. Added tests that fail when the submitter selector stops matching, so the next drift breaks the suite instead of the output.
Verification log
- 2026-11-12: reran against the live page, 30 rows, submitter populated on every row, 6 of 6 pytest tests passing
- 2026-11-19: defect facts rechecked against the artefact store copy
Verified 2026-11-19
Files
Price shape
This is the failure a paid diagnostic is built to find. Triage runs $150 to $300 with a written acceptance check: selectors that still match, a sample CSV, and a short failure log. Full scraper builds start at $300.
Start here
Send the target URL and the exact columns you need. I reply with the fixed quote and the first sample plan.