erikfiala/google-maps-scraper: self-hosted Playwright scraper that discovers businesses on Google Maps

Self-hosted Playwright scraper that discovers businesses on Google Maps, checkpoints to JSONL per category, optionally extracts emails from websites, and exports a CSV of the fields you enable.

No Monid, Apify, Bright Data, or other paid scrape APIs.

Quick start

npm i && npx playwright install chromium
# edit config/scraper.json — toggle fields + categories
npm run scrape -- --states all
npm run enrich
npm run export

Default category is marketing-agency (enabled in config). Pilot scrape uses CA + NY unless you pass --states all or a custom list.

Configuring output fields

Edit config/scraper.json → fields. Each key is true / false.

Behavior

• CSV columns = only fields set to true, in the order above.
• Email enrich (npm run enrich) runs only when fields.email is true; otherwise it skips.
• Maps scrape still stores a full place record in places.jsonl for resume/debugging; export and docs emphasize the toggled fields.

Example (defaults):

{
  "fields": {
    "name": true,
    "email": true,
    "phone": false,
    "address": false,
    "website": false,
    "category": true,
    "maps_url": false,
    "place_id": false,
    "city": false,
    "state": false,
    "country": false
  }
}

Configuring search categories

Same file → search.categories[]:

Defaults shipped

• Enabled: marketing-agency — queries marketing agency, advertising agency
• Disabled examples: book-publisher plus writing/audio categories (writers-association, writing-workshop, self-publishing-service, book-publishing-consultant, literary-agency, audiobook-production, podcast-production, narrator-voiceover)

Flip enabled to true or add your own category objects. Starter example:

{
  "slug": "marketing-agency",
  "enabled": true,
  "label": "marketing agency",
  "queries": ["marketing agency", "advertising agency"]
}

CLI reference

# Full US scrape for the default (first enabled) category
npm run scrape -- --states all

# Or an explicit category
npm run scrape -- --country us --category marketing-agency --states all

# Standalone scrape entry (preferred for long runs)
npx tsx src/geo_harvest.ts --country us --category marketing-agency --states all

# Enrich websites → emails (no-op if fields.email is false)
npm run enrich -- --country us --category marketing-agency

# Write leads.csv (enabled fields only)
npm run export -- --country us --category marketing-agency

# Full pipeline over enabled categories
python3 scripts/run_category_pipeline.py
# Or a subset (ignores enabled flags):
python3 scripts/run_category_pipeline.py marketing-agency book-publisher

Options

Data layout

data/<country>/<category-slug>/
  places.jsonl      # append-only checkpoint (scrape + enrich)
  progress.json     # completed tiles + CAPTCHA pause flag
  leads.csv         # export (enabled fields only)

data/ is gitignored. Deduplicate within a category by place_id. Cross-category overlap is OK.

CAPTCHA / resume

• Completed tiles are recorded in progress.json and skipped on re-run.
• Existing place_ids in places.jsonl are skipped (dedupe).
• Multiple Maps queries for a category run per tile before the tile is marked complete.
• CAPTCHA detection only treats real Google /sorry/ interstitial pathnames. Consent pages are dismissed, not treated as hard blocks.
• If Google shows a CAPTCHA, the scraper pauses, logs the tile, and sets pausedForCaptcha in progress.json.
• Resolve manually with --headed if headless retries fail.

Disclaimer

This tool is for educational and legitimate research use. Respect Google’s Terms of Service, applicable laws, and website robots/terms when enriching emails. You are responsible for how you use scraped data (including CAN-SPAM / GDPR / marketing consent). Scraping at scale may trigger rate limits or blocks; be a good citizen.