Add developer email and website fields to CSV output

- Updated CSV_FIELDS to include "developer_email" and "developer_website".
- Modified play_row function to extract developer email and website from the response.
- Updated appstore_row function to include developer website, with a note that developer email is not available from Apple's API.

RUN_PLAN.md

@@ -0,0 +1,122 @@
+# Run Plan — Collect ~2 Lakh (200k) Apps + Scrape Overnight
+
+A step-by-step runbook for one full-time laptop. Goal: discover **~200,000 apps**
+(Google Play + Apple App Store) across multiple countries, then scrape full data
+for **US + AU** (add more as needed) overnight.
+
+> Full architecture & how everything works: see **ARCHITECTURE.html**.
+
+---
+
+## The strategy in one line
+**App Store** is the volume engine (~180 apps per search term) → it carries you to 200k.
+**Google Play** is slow (~30 per query, hard cap) → it adds ~40–60k using the fast `--deep` set.
+Running Play with the full wordlist would waste ~11 hours on duplicates, so we **don't**.
+
+---
+
+## PHASE 1 — Discover the apps (run during the day, ~3–6 hrs)
+
+### Step 1. App Store — the big pull (full wordlist)
+```powershell
+python discover.py --skip-play --target 160000 --deep --terms-file english_words.txt --countries us,au,gb,in,ca,de
+```
+
+### Step 2. Google Play — bounded & fast (adds Play apps)
+```powershell
+python discover.py --skip-appstore --deep --countries us,au,gb,in,ca,de
+```
+Both steps **merge into the same `apps.json`** (Step 2 keeps Step 1's results).
+More countries = more unique apps → edit the `--countries` list to match where you operate.
+
+### Step 3. Back up the list immediately (it's a multi-hour artifact!)
+```powershell
+Copy-Item apps.json apps_backup.json
+```
+
+### Check the count anytime
+```powershell
+python -c "import json;c=json.load(open('apps.json',encoding='utf-8'));print('Play',len(c['google_play']),'+ App Store',len(c['app_store']),'=',len(c['google_play'])+len(c['app_store']))"
+```
+If under 200k, add more countries (e.g. `,fr,br,jp,mx`) and re-run **Step 1**.
+
+---
+
+## PHASE 2 — Scrape the data (overnight)
+
+### Before you start: set a faster pace (one-time)
+Open `apps.json` and set the delay in `settings` to `0.5` (currently `1.0`):
+```jsonc
+"delay_seconds": 0.5
+```
+Effective request rate ≈ `workers ÷ delay`. At `workers 8` and `delay 0.5` → ~16 req/s.
+
+### Run the scrape (US + AU)
+```powershell
+python scraper.py --countries us,au --workers 8
+```
+- Produces **one row per app per country** → ~200k apps × 2 countries ≈ 400k rows.
+- Output: **`output/app_data.csv`**
+- Auto-backups every 1,000 rows to **`output/backups/`**.
+
+### If it doesn't finish in one night
+Just **run the exact same command again** the next night — it **resumes** and skips
+everything already done. It's totally fine if this takes 1–2 nights.
+
+**Rough timing:** ~400k rows at ~16 req/s ≈ **7 hours** (one night). 3 countries ≈ 10–11 hrs.
+
+---
+
+## Safety (unattended overnight)
+- The **circuit breaker** auto-pauses all workers if a store rate-limits, then resumes.
+  Seeing occasional `THROTTLING: pausing all workers` in the log = protection working, not an error.
+- If you wake up to **constant** throttling messages → lower to `--workers 6` next run.
+- Don't push workers too high while you sleep — an IP block at 2am costs more time than running slower.
+
+---
+
+## Recovery (if something gets deleted)
+- **`output/app_data.csv` lost?** Copy the newest snapshot back, then re-run (it resumes):
+  ```powershell
+  Copy-Item output\backups\app_data.bak3.csv output\app_data.csv
+  python scraper.py --countries us,au --workers 8
+  ```
+- **`apps.json` lost?** Restore your backup:
+  ```powershell
+  Copy-Item apps_backup.json apps.json
+  ```
+
+---
+
+## Output columns (`output/app_data.csv`)
+`store, country, app_id, title, developer, developer_email, developer_website,
+category, price, currency, free, avg_rating, total_ratings, text_review_count,
+last_updated, version, url`
+
+- `developer_email` → filled for almost all **Google Play** apps; **blank for App Store** (Apple's API has none).
+- `developer_website` → filled for both stores.
+
+---
+
+## Command cheat sheet
+```powershell
+# DISCOVER (App Store volume + Play)
+python discover.py --skip-play --target 160000 --deep --terms-file english_words.txt --countries us,au,gb,in,ca,de
+python discover.py --skip-appstore --deep --countries us,au,gb,in,ca,de
+Copy-Item apps.json apps_backup.json
+
+# SCRAPE (overnight, resumable)
+python scraper.py --countries us,au --workers 8
+
+# CHECK COUNT
+python -c "import json;c=json.load(open('apps.json',encoding='utf-8'));print(len(c['google_play'])+len(c['app_store']))"
+```
+
+---
+
+## Notes / limits (honest)
+- No store has an "all apps" list — you only get what discovery finds. 200k is realistic; *every* app is not.
+- Google Play search is capped at ~30 results/query — Play volume comes from many terms, not from one big query.
+- Developer **phone numbers** are not available from either source.
+- Data collected is **public, non-personal** app metadata. Using `developer_email` for bulk
+  outreach is governed by anti-spam law (CAN-SPAM / GDPR / CASL) — get sign-off before any marketing use.

scraper.py

@@ -62,7 +62,8 @@ MAX_RATE_LIMIT_WAITS = 12
 MAX_TRANSIENT_RETRIES = 4
 
 CSV_FIELDS = [
-    "store", "country", "app_id", "title", "developer", "category",
+    "store", "country", "app_id", "title", "developer",
+    "developer_email", "developer_website", "category",
     "price", "currency", "free",
     "avg_rating", "total_ratings", "text_review_count",
     "last_updated", "version", "url",
@@ -112,6 +113,8 @@ def play_row(app_id: str, lang: str, country: str) -> dict:
         "app_id": app_id,
         "title": d.get("title"),
         "developer": d.get("developer"),
+        "developer_email": d.get("developerEmail"),
+        "developer_website": d.get("developerWebsite"),
         "category": d.get("genre"),
         "price": d.get("price"),
         "currency": d.get("currency"),
@@ -146,6 +149,8 @@ def appstore_row(app_id: int, country: str) -> dict | None:
         "app_id": meta.get("trackId"),
         "title": meta.get("trackName"),
         "developer": meta.get("sellerName"),
+        "developer_email": None,  # Apple's lookup API does not expose a dev email
+        "developer_website": meta.get("sellerUrl"),
         "category": meta.get("primaryGenreName"),
         "price": price,
         "currency": meta.get("currency"),