dhruv.godvani
03bf0fc923
- discover.py: find app IDs (Play search, iTunes Search API, RSS charts) -> apps.json - scraper.py: fetch metadata + ratings per app/country, threaded, shardable, resumable - netutil.py: shared retry/backoff + adaptive circuit breaker (rate-limit safe) - merge_csv.py: combine sharded outputs from multiple machines - ARCHITECTURE.html: full architecture & flow documentation - english_words.txt: deep search-term wordlist Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
74 lines
2.3 KiB
Markdown
74 lines
2.3 KiB
Markdown
# App Store Scraper (Google Play + Apple App Store)
|
|
|
|
For a **specific list of apps** (defined in `apps.json`) this produces **one combined CSV**
|
|
with core metadata, the **last-updated date**, and the **final rating** (average + total
|
|
number of ratings). Python, free libraries only.
|
|
|
|
## What it uses
|
|
|
|
| Store | Source |
|
|
|-------|--------|
|
|
| Google Play | `google-play-scraper` (scrapes public pages) |
|
|
| Apple App Store | iTunes Lookup API (**official**, free) |
|
|
|
|
No paid services, no API keys.
|
|
|
|
## Setup
|
|
|
|
```powershell
|
|
python -m pip install -r requirements.txt
|
|
```
|
|
|
|
## Configure — `apps.json`
|
|
|
|
```jsonc
|
|
{
|
|
"settings": {
|
|
"country": "us", // store country code
|
|
"lang": "en", // language (Play only)
|
|
"delay_seconds": 1.0 // pause between requests (avoid blocks)
|
|
},
|
|
"google_play": [
|
|
{ "app_id": "com.whatsapp" } // the "id=" value in the Play URL
|
|
],
|
|
"app_store": [
|
|
{ "app_id": 310633997, "name": "WhatsApp" } // the numeric "id" in the App Store URL ("name" is just a label)
|
|
]
|
|
}
|
|
```
|
|
|
|
**Finding IDs**
|
|
- Play Store: `play.google.com/store/apps/details?id=`**`com.whatsapp`** → use `com.whatsapp`
|
|
- App Store: `apps.apple.com/us/app/whatsapp-messenger/id`**`310633997`** → use `310633997`
|
|
|
|
## Run
|
|
|
|
```powershell
|
|
python scraper.py
|
|
```
|
|
|
|
## Output — `output/app_data.csv` (one row per app, both stores together)
|
|
|
|
| Column | Meaning |
|
|
|--------|---------|
|
|
| `store` | `google_play` or `app_store` |
|
|
| `app_id`, `title`, `developer`, `category` | identity |
|
|
| `price`, `currency`, `free` | pricing |
|
|
| `avg_rating` | final average star rating |
|
|
| `total_ratings` | total number of ratings |
|
|
| `text_review_count` | number of text reviews (Play only) |
|
|
| `last_updated` | date the app was last updated |
|
|
| `version` | latest version |
|
|
| `url` | store listing URL |
|
|
|
|
UTF-8 with BOM — opens cleanly in Excel.
|
|
|
|
## Limitations (important)
|
|
|
|
- **No "all apps" list exists.** Neither store exposes a full catalog; you scrape only
|
|
the apps you list here. Use `discover.py` to auto-fill `apps.json` from top charts.
|
|
- **Rate limits / blocking** — keep `delay_seconds > 0`; heavy use may need proxies.
|
|
- **Terms of Service** — both stores restrict automated scraping. Fine for internal research
|
|
at small scale; get legal sign-off before commercial redistribution.
|
|
- **Per-country data** — results differ by `country`.
|