## 0Big picture

There is no "all apps" endpoint in either store. So the system works in two phases: first it discovers a big list of app IDs, then it scrapes the details for every app on that list into one CSV. Everything is free — no API keys, no paid services.

2
stores covered
(Play + App Store)
~30
apps / Google Play query
(hard server cap)
~180
apps / iTunes search term
(the volume engine)
100k+
target catalog size
(App Store driven)
Two verbs, two phases
FIND apps → `discover.py` writes `apps.json`. GET data → `scraper.py` reads `apps.json` and writes `output/app_data.csv`.

## 1The pipeline

End-to-end, data flows left to right. The same flow runs on each PC; only the final merge is shared.

discover.py
FIND apps
→
apps.json
list of app IDs
→
scraper.py
GET data
→
app_data.csv
one row / app / country
→
merge_csv.py
combine 2 PCs
process (script) data file output

Step | Script | Reads | Writes | Purpose
--- | --- | --- | --- | ---
Find | `discover.py` | search APIs + charts | `apps.json` | build a big list of app IDs
Get | `scraper.py` | `apps.json` | `output/app_data.csv` | fetch metadata + ratings for each app
Merge | `merge_csv.py` | shard CSVs | `output/app_data.csv` | combine the two PCs' results, de-duped

## 2Files in the project

File | Role
--- | ---
`discover.py` | Phase 1 Finds app IDs and fills `apps.json`
`scraper.py` | Phase 2 Fetches per-app data into the CSV (threaded, shardable, resumable)
`netutil.py` | Shared safety Error classification + adaptive throttle / circuit breaker
`merge_csv.py` | Helper Merges the two PCs' shard CSVs into one
`apps.json` | Data Settings + the list of app IDs (the contract between the two phases)
`english_words.txt` | Data ~9,868 common words used as deep search terms (path to 100k)
`requirements.txt` | Dependencies: `google-play-scraper`, `requests`
`output/app_data.csv` | The final dataset

## 3Phase 1 — Find apps (discover.py)

Because no full catalog exists, discovery queries many sources with many terms and keeps only the unique app IDs. Three sources feed one de-duplicated set per store.

### The three discovery sources

Source | Store | How | Yield / call | Role
--- | --- | --- | --- | ---
Keyword search | Google Play | `google_play_scraper.search(term)` | ~30 (hard cap) | breadth via many terms
iTunes Search API | App Store | `itunes.apple.com/search?term=` | ~180 | main volume engine
RSS top charts | App Store | `.../rss/{feed}/genre={id}` (free/grossing/paid × 23 genres) | up to 200 | popular apps, fast

### How the search terms are built

The function `build_terms()` assembles the query list from up to four tiers:

~155 curated
topic keywords
+
a–z
26 letters
+
aa–zz
`--deep` (676)
+
wordlist
`--terms-file` (~9,868)
=
~9,943 terms
de-duped, ordered

More terms = more apps. The wordlist (`--terms-file english_words.txt`) is the single biggest lever toward 100k. `--deeper` (aaa–zzz, 17,576 terms) exists but is very slow.

### Discovery flow (per store)

1. Build the term list and resolve the country list (from `--countries` or settings).
2. For each country, loop every term; call the source through the safety layer (retry + backoff).
3. Collect each result's app ID into a `seen` dictionary — duplicates are ignored automatically.
4. Print live progress: `total unique = N / target`.
5. Stop as soon as `--target` is reached (or terms are exhausted).
6. Merge into the existing `apps.json` (keeps your settings + anything already listed), then write the file.

Why Google Play can't reach 100k
Play search returns only ~30 apps per query no matter what — that's Google's cap, not the code's. So Play contributes ~15–30k; the App Store's ~180/term search is what carries you to 100k.
De-duplication is automatic and idempotent
Discovery keys on app ID, and `merge()` drops duplicates against what's already in `apps.json`. Re-running only adds new apps — it never loses or doubles existing ones. If `apps.json` is missing, a fresh one is created automatically.

## 4apps.json — the app list

This file is the contract between the two phases: discovery writes it, scraping reads it.

    {
      "settings": {
        "country": "us",                          // default single country
        "countries": ["us","gb","in","ca","au"],  // scrape/discover across these
        "lang": "en",                             // language (Google Play)
        "review_count": 100,
        "delay_seconds": 1.0                       // polite pause; rate ≈ workers / delay
      },
      "google_play": [
        { "app_id": "com.whatsapp" }              // the id= value from the Play URL (string)
      ],
      "app_store": [
        { "app_id": 310633997, "name": "WhatsApp" } // numeric id from the App Store URL
      ]
    }

Store | ID type | Where it comes from
--- | --- | ---
Google Play | string | `play.google.com/store/apps/details?id=com.whatsapp`
App Store | number | `apps.apple.com/us/app/.../id310633997`

## 5Phase 2 — Get data (scraper.py)

Scraping turns the ID list into real data. For every app × every country it makes one "detail" call and writes one CSV row. Two stores, two API calls, one unified row schema.

### The two detail calls

### Google Play

`google_play_scraper.app(app_id, lang, country)`

Scrapes the public listing page → title, developer, genre, price, score, ratings count, review count, last-updated, version, URL.

### Apple App Store

`itunes.apple.com/lookup?id=&country=`

Official lookup API → trackName, seller, genre, price, average rating, rating count, release date, version, URL.

### Output columns — `output/app_data.csv`

Column | Meaning
--- | ---
`store` | `google_play` or `app_store`
`country` | which region this row was fetched for
`app_id`, `title`, `developer`, `category` | identity
`price`, `currency`, `free` | pricing
`avg_rating` | final average star rating
`total_ratings` | total number of ratings
`text_review_count` | number of written reviews (Play only)
`last_updated` | date the app was last updated
`version` | latest version
`url` | store listing link

CSV is UTF-8 with BOM → opens cleanly in Excel. The same columns apply to both stores.

### How one job is processed

job
(store, id, country)
→
throttle.wait()
honor cooldown
→
fetch
Play / App Store
→
row
flushed to CSV

## 6Safety layer (netutil.py)

The #1 rule: slow is fine — getting the IP blocked is not.
Every request (in both scripts) passes through a shared throttle. The moment a store pushes back, all workers slow down together instead of hammering until blocked.

### Error classification

Signal | Meaning | Reaction
--- | --- | ---
429 / 403 "too many requests", "quota" | rate-limited | Trip circuit breaker, wait it out, retry generously
408 / 5xx, timeout, connection | transient hiccup | short exponential backoff, a few retries
404 / not-found | healthy answer | skip immediately — no wasted retries

### The circuit breaker (a shared state machine)

RUNNING
requests flow freely
→
TRIPPED
all workers pause
→
COOLDOWN
30→60→120…→300s
→
RECOVER
speed back up on success

- Trips on a rate-limit signal or 5 consecutive unexplained failures (catches "stealth" blocks that don't return a clean 429).
- Cooldown doubles each time it re-trips (30s → 60 → 120 … capped at 5 min) and lingers as extra spacing.
- Recovers automatically: each success shrinks the cooldown and extra delay back toward normal.

Reading the logs
A line like `THROTTLING: pausing all workers ~60s` is the protection working, not an error. If it fires constantly, that IP is at its limit → lower `--workers` on that machine.

## 7Concurrency — why threads

These are network-wait tasks (most time is spent waiting for the server, not computing), which is the ideal case for a thread pool. With `--workers N`, up to N requests are in flight at once.

job queue
all apps × countries
→
worker 1
worker 2
… worker N
→
CSV sink
thread-safe, flush per row

Effective request rate ≈ `workers ÷ delay_seconds`. The throttle sits above all workers, so one rate-limit signal pauses the whole pool.

## 8Two-PC architecture (2 CPUs, 2 IPs)

Two machines with separate IPs double throughput and the safe request budget (each store rate-limits per IP). You don't split the file by hand — both PCs read the same `apps.json` and each takes a different shard.

### How sharding splits the work

All jobs are flattened to a list, then each PC takes `jobs[shard::shards]` — i.e. every Nth job. This interleaves Play/App-Store/country evenly, so neither PC gets a lopsided slice.

apps.json
jobs: 0,1,2,3,4,5,6,7…

### PC #1 — IP A

`--shard 0 --shards 2`

does jobs 0, 2, 4, 6 …

writes `app_data_shard0of2.csv`

### PC #2 — IP B

`--shard 1 --shards 2`

does jobs 1, 3, 5, 7 …

writes `app_data_shard1of2.csv`

shard0 CSV
↘
merge_csv.py
de-duped by store+country+id
↙
shard1 CSV
↓
app_data.csv
complete combined dataset

### The 3 steps end-to-end

1. Discover once (either PC): run `discover.py`, then copy the resulting `apps.json` to both machines.
2. Scrape in parallel: PC #1 runs shard 0, PC #2 runs shard 1 (each with its own IP and its own thread pool).
3. Merge: copy both shard CSVs into one `output/` folder and run `merge_csv.py`.

÷2
work per PC
(sharding)
×~10
per PC
(thread pool)
×2 IPs
safe rate budget
(per-IP limits)
28h → ~1h
rough wall-clock
for 100k records

## 9Resume & crash safety

A 100k run can't depend on a single perfect execution. Two mechanisms make interruptions harmless:

### 💾 Flush-as-you-go

Each row is written and flushed to the CSV the instant it's fetched. A crash, block, or `Ctrl-C` never loses completed work.

### ⏯️ Resume on restart

On startup the scraper reads the existing CSV and skips apps already done (keyed by store + country + id). Just re-run the same command to continue.

Practical meaning
If a PC is interrupted at app 40,000 of 50,000, re-running the exact same command picks up at 40,001. Use `--no-resume` only if you deliberately want to re-scrape everything.

## 10Configuration & flags

### discover.py

Flag | Default | Purpose
--- | --- | ---
`--target N` | 0 (all terms) | stop each store after N unique apps
`--terms-file PATH` | — | extra search terms (best lever to 100k)
`--deep` | off | add 2-letter terms aa–zz (676)
`--deeper` | off | add 3-letter terms aaa–zzz (17,576; slow)
`--countries a,b,c` | settings | regions to discover across
`--skip-play` / `--skip-appstore` | off | run only one store
`--search-limit` / `--chart-limit` | 200 / 200 | results per iTunes term / per chart
`--hits` | 30 | Play results per term (server caps ~30)
`--delay` | 0.5 | polite pause between discovery requests

### scraper.py

Flag | Default | Purpose
--- | --- | ---
`--workers N` | 10 | concurrent requests (thread pool)
`--shard i` | 0 | this machine's slice index
`--shards N` | 1 | total machines splitting the work
`--countries a,b,c` | settings | regions to scrape (one row each)
`--no-resume` | off | ignore existing CSV, scrape all again
`--out DIR` / `--config PATH` | output / apps.json | output folder / input file

## 11Command cheat sheet

### ① Discover ~100k apps (App Store driven)

    # fastest route to 1 lakh — App Store only, using the wordlist
    python discover.py --skip-play --target 100000 --terms-file english_words.txt --countries us

    # include Google Play too (adds ~15–30k; can't reach 100k on its own)
    python discover.py --target 100000 --terms-file english_words.txt --countries us,gb

### ② Scrape across 2 PCs

    # --- PC #1 (IP A) ---
    python scraper.py --shard 0 --shards 2 --workers 10

    # --- PC #2 (IP B) ---
    python scraper.py --shard 1 --shards 2 --workers 10

    # ...interrupted? just run the same command again — it resumes.

### ③ Merge the two PCs' output

    # copy both app_data_shard*of2.csv into one output/ folder, then:
    python merge_csv.py

### Single-PC run (no sharding)

    python scraper.py --workers 10

## 12Limits & ground truth

Reality | Detail
--- | ---
No full catalog | Neither store lists "all apps"; you only get what discovery finds.
Google Play search cap | ~30 results per query — Play tops out ~15–30k regardless of terms.
App Store is the volume | iTunes search ~180/term → this is what reaches 100k.
Per-IP rate limits | Both stores throttle per IP; 2 PCs/IPs ≈ 2× safe budget. Proxies extend further.
Per-country data | Ratings & pricing differ by country → one row per app per country.
Terms of Service | Both stores restrict automated scraping. Fine for internal research at modest scale; get sign-off before commercial redistribution.

Tuning rule of thumb
Start at `--workers 10`. If you see constant `THROTTLING` lines, lower workers (e.g. 6) — fewer cooldowns beats pushing too hard. Back up `apps.json` after a big discovery run; it's the expensive artifact.