← Sam Mozer
LLM market research pipeline
Prediction-market communities talk constantly about strategies and tools. This pipeline reads that chatter and turns it into a structured, queryable dataset.
- Role
- Solo project
- Model
- Llama 4 Scout on Workers AI
- Stack
- Workers · Queues · R2 · D1 · Hono
code on GitHub soon
Architecture
Cron every 30 minutes
│
▼
scheduler decides which sources are due, based on cadence rules
│
▼
Queue scrape jobs · batches of 10 · 5 consumers at a time
├─ Reddit scraper 7 subreddits, public .json endpoints
├─ Polymarket markets Gamma API
└─ Polymarket leaderboard Data API
│
├─► R2 raw responses, archived as-is
└─► D1 posts, comments, markets
│
▼
Workers AI · Llama 4 Scout (JSON mode)
│ extracts strategies, tools, and markets from interesting posts
▼
D1 rollups ──► Hono query API ──► local CLI
What I built and why
- Queues instead of one long cron job. Each source becomes its own job, so a slow or failing scraper doesn't block the others, retries happen per job, and concurrency is a config setting instead of code.
- Raw data in R2, structure in D1. Keeping the untouched responses means I can re-run extraction with a better prompt or model later without scraping again.
- LLM extraction with guardrails. Llama 4 Scout runs in JSON mode so its output can go straight into D1. It kept labeling exchanges, sportsbooks, and wallets as "tools," so I added a post-processing filter for the false positives it surfaced most often.
- A small query layer. A Hono API and a local CLI query the extracted strategies, tools, and markets without opening the database.