📱 App Store Scraper

0Big picture

There is no "all apps" endpoint in either store. So the system works in two phases: first it discovers a big list of app IDs, then it scrapes the details for every app on that list into one CSV. Everything is free — no API keys, no paid services.

1The pipeline

End-to-end, data flows left to right. The same flow runs on each PC; only the final merge is shared.

2Files in the project

3Phase 1 — Find apps (discover.py)

Because no full catalog exists, discovery queries many sources with many terms and keeps only the unique app IDs. Three sources feed one de-duplicated set per store.

The three discovery sources

How the search terms are built

The function build_terms() assembles the query list from up to four tiers:

More terms = more apps. The wordlist (--terms-file english_words.txt) is the single biggest lever toward 100k. --deeper (aaa–zzz, 17,576 terms) exists but is very slow.

Discovery flow (per store)

1. Build the term list and resolve the country list (from --countries or settings).
2. For each country, loop every term; call the source through the safety layer (retry + backoff).
3. Collect each result's app ID into a seen dictionary — duplicates are ignored automatically.
4. Print live progress: total unique = N / target.
5. Stop as soon as --target is reached (or terms are exhausted).
6. Merge into the existing apps.json (keeps your settings + anything already listed), then write the file.

4apps.json — the app list

This file is the contract between the two phases: discovery writes it, scraping reads it.

{
  "settings": {
    "country": "us",                          // default single country
    "countries": ["us","gb","in","ca","au"],  // scrape/discover across these
    "lang": "en",                             // language (Google Play)
    "review_count": 100,
    "delay_seconds": 1.0                       // polite pause; rate ≈ workers / delay
  },
  "google_play": [
    { "app_id": "com.whatsapp" }              // the id= value from the Play URL (string)
  ],
  "app_store": [
    { "app_id": 310633997, "name": "WhatsApp" } // numeric id from the App Store URL
  ]
}

5Phase 2 — Get data (scraper.py)

Scraping turns the ID list into real data. For every app × every country it makes one "detail" call and writes one CSV row. Two stores, two API calls, one unified row schema.

The two detail calls

Output columns — output/app_data.csv

CSV is UTF-8 with BOM → opens cleanly in Excel. The same columns apply to both stores.

How one job is processed

6Safety layer (netutil.py)

Error classification

The circuit breaker (a shared state machine)

• Trips on a rate-limit signal or 5 consecutive unexplained failures (catches "stealth" blocks that don't return a clean 429).
• Cooldown doubles each time it re-trips (30s → 60 → 120 … capped at 5 min) and lingers as extra spacing.
• Recovers automatically: each success shrinks the cooldown and extra delay back toward normal.

7Concurrency — why threads

These are network-wait tasks (most time is spent waiting for the server, not computing), which is the ideal case for a thread pool. With --workers N, up to N requests are in flight at once.

Effective request rate ≈ workers ÷ delay_seconds. The throttle sits above all workers, so one rate-limit signal pauses the whole pool.

8Two-PC architecture (2 CPUs, 2 IPs)

Two machines with separate IPs double throughput and the safe request budget (each store rate-limits per IP). You don't split the file by hand — both PCs read the same apps.json and each takes a different shard.

How sharding splits the work

All jobs are flattened to a list, then each PC takes jobs[shard::shards] — i.e. every Nth job. This interleaves Play/App-Store/country evenly, so neither PC gets a lopsided slice.

The 3 steps end-to-end

1. Discover once (either PC): run discover.py, then copy the resulting apps.json to both machines.
2. Scrape in parallel: PC #1 runs shard 0, PC #2 runs shard 1 (each with its own IP and its own thread pool).
3. Merge: copy both shard CSVs into one output/ folder and run merge_csv.py.

9Resume & crash safety

A 100k run can't depend on a single perfect execution. Two mechanisms make interruptions harmless:

10Configuration & flags

discover.py

scraper.py

11Command cheat sheet

① Discover ~100k apps (App Store driven)

# fastest route to 1 lakh — App Store only, using the wordlist
python discover.py --skip-play --target 100000 --terms-file english_words.txt --countries us

# include Google Play too (adds ~15–30k; can't reach 100k on its own)
python discover.py --target 100000 --terms-file english_words.txt --countries us,gb

② Scrape across 2 PCs

# --- PC #1 (IP A) ---
python scraper.py --shard 0 --shards 2 --workers 10

# --- PC #2 (IP B) ---
python scraper.py --shard 1 --shards 2 --workers 10

# ...interrupted? just run the same command again — it resumes.

③ Merge the two PCs' output

# copy both app_data_shard*of2.csv into one output/ folder, then:
python merge_csv.py

Single-PC run (no sharding)

python scraper.py --workers 10

12Limits & ground truth