Files

T

dhruv.godvani 03bf0fc923 Initial commit: App Store + Google Play scraper

- discover.py: find app IDs (Play search, iTunes Search API, RSS charts) -> apps.json
- scraper.py: fetch metadata + ratings per app/country, threaded, shardable, resumable
- netutil.py: shared retry/backoff + adaptive circuit breaker (rate-limit safe)
- merge_csv.py: combine sharded outputs from multiple machines
- ARCHITECTURE.html: full architecture & flow documentation
- english_words.txt: deep search-term wordlist

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-06-17 17:54:18 +05:30

2.3 KiB

Raw Permalink Blame History

App Store Scraper (Google Play + Apple App Store)

For a specific list of apps (defined in apps.json) this produces one combined CSV with core metadata, the last-updated date, and the final rating (average + total number of ratings). Python, free libraries only.

What it uses

Store	Source
Google Play	`google-play-scraper` (scrapes public pages)
Apple App Store	iTunes Lookup API (official, free)

No paid services, no API keys.

Setup

python -m pip install -r requirements.txt

Configure — `apps.json`

{
  "settings": {
    "country": "us",        // store country code
    "lang": "en",           // language (Play only)
    "delay_seconds": 1.0    // pause between requests (avoid blocks)
  },
  "google_play": [
    { "app_id": "com.whatsapp" }        // the "id=" value in the Play URL
  ],
  "app_store": [
    { "app_id": 310633997, "name": "WhatsApp" }   // the numeric "id" in the App Store URL ("name" is just a label)
  ]
}

Finding IDs

Play Store: play.google.com/store/apps/details?id=com.whatsapp → use com.whatsapp
App Store: apps.apple.com/us/app/whatsapp-messenger/id310633997 → use 310633997

Run

python scraper.py

Output — `output/app_data.csv` (one row per app, both stores together)

Column	Meaning
`store`	`google_play` or `app_store`
`app_id`, `title`, `developer`, `category`	identity
`price`, `currency`, `free`	pricing
`avg_rating`	final average star rating
`total_ratings`	total number of ratings
`text_review_count`	number of text reviews (Play only)
`last_updated`	date the app was last updated
`version`	latest version
`url`	store listing URL

UTF-8 with BOM — opens cleanly in Excel.

Limitations (important)

No "all apps" list exists. Neither store exposes a full catalog; you scrape only the apps you list here. Use discover.py to auto-fill apps.json from top charts.
Rate limits / blocking — keep delay_seconds > 0; heavy use may need proxies.
Terms of Service — both stores restrict automated scraping. Fine for internal research at small scale; get legal sign-off before commercial redistribution.
Per-country data — results differ by country.

2.3 KiB Raw Permalink Blame History