dhruv.godvani
03bf0fc923
- discover.py: find app IDs (Play search, iTunes Search API, RSS charts) -> apps.json - scraper.py: fetch metadata + ratings per app/country, threaded, shardable, resumable - netutil.py: shared retry/backoff + adaptive circuit breaker (rate-limit safe) - merge_csv.py: combine sharded outputs from multiple machines - ARCHITECTURE.html: full architecture & flow documentation - english_words.txt: deep search-term wordlist Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
542 lines
30 KiB
HTML
542 lines
30 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en">
|
||
<head>
|
||
<meta charset="UTF-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<title>App Store Scraper — Architecture & Flow</title>
|
||
<style>
|
||
:root{
|
||
--bg:#0d1117; --panel:#161b22; --panel2:#1c2330; --border:#30363d;
|
||
--text:#e6edf3; --muted:#9aa7b4; --accent:#58a6ff; --green:#3fb950;
|
||
--amber:#d29922; --red:#f85149; --purple:#bc8cff; --cyan:#39c5cf;
|
||
--code:#0b1020;
|
||
}
|
||
*{box-sizing:border-box}
|
||
html{scroll-behavior:smooth}
|
||
body{
|
||
margin:0; background:var(--bg); color:var(--text);
|
||
font-family:"Segoe UI",system-ui,Roboto,Helvetica,Arial,sans-serif;
|
||
line-height:1.6; font-size:16px;
|
||
}
|
||
.wrap{max-width:1080px; margin:0 auto; padding:0 24px 120px}
|
||
header.hero{
|
||
background:linear-gradient(135deg,#11203b 0%,#0d1117 60%);
|
||
border-bottom:1px solid var(--border); padding:54px 24px 40px; text-align:center;
|
||
}
|
||
header.hero h1{margin:0 0 8px; font-size:2.3rem; letter-spacing:.3px}
|
||
header.hero p{margin:4px 0; color:var(--muted)}
|
||
.pill{display:inline-block; background:var(--panel2); border:1px solid var(--border);
|
||
color:var(--cyan); border-radius:999px; padding:4px 14px; font-size:.8rem; margin-top:14px}
|
||
h2{font-size:1.55rem; margin:54px 0 14px; padding-bottom:8px; border-bottom:2px solid var(--border)}
|
||
h2 .num{color:var(--accent); font-variant-numeric:tabular-nums; margin-right:10px}
|
||
h3{font-size:1.15rem; margin:30px 0 10px; color:var(--cyan)}
|
||
p{margin:10px 0}
|
||
a{color:var(--accent); text-decoration:none}
|
||
a:hover{text-decoration:underline}
|
||
code{background:var(--code); border:1px solid var(--border); padding:1px 6px;
|
||
border-radius:5px; font-family:"Cascadia Code",Consolas,monospace; font-size:.86em; color:#ffd9a0}
|
||
pre{background:var(--code); border:1px solid var(--border); border-radius:10px;
|
||
padding:16px 18px; overflow:auto; font-family:"Cascadia Code",Consolas,monospace;
|
||
font-size:.85rem; line-height:1.5}
|
||
pre code{background:none; border:none; padding:0; color:#c9d6e3}
|
||
.muted{color:var(--muted)}
|
||
/* TOC */
|
||
nav.toc{background:var(--panel); border:1px solid var(--border); border-radius:12px;
|
||
padding:18px 22px; margin:34px 0}
|
||
nav.toc h4{margin:0 0 10px; color:var(--muted); font-weight:600; text-transform:uppercase; font-size:.78rem; letter-spacing:1px}
|
||
nav.toc ol{margin:0; padding-left:20px; columns:2; column-gap:40px}
|
||
nav.toc li{margin:5px 0}
|
||
/* cards / panels */
|
||
.panel{background:var(--panel); border:1px solid var(--border); border-radius:12px; padding:18px 22px; margin:18px 0}
|
||
.grid{display:grid; gap:14px}
|
||
.grid.c2{grid-template-columns:1fr 1fr}
|
||
.grid.c3{grid-template-columns:repeat(3,1fr)}
|
||
@media(max-width:760px){.grid.c2,.grid.c3{grid-template-columns:1fr} nav.toc ol{columns:1}}
|
||
/* tables */
|
||
table{width:100%; border-collapse:collapse; margin:16px 0; font-size:.92rem}
|
||
th,td{border:1px solid var(--border); padding:9px 12px; text-align:left; vertical-align:top}
|
||
th{background:var(--panel2); color:var(--text); font-weight:600}
|
||
tr:nth-child(even) td{background:rgba(255,255,255,.02)}
|
||
/* flow diagram boxes */
|
||
.flow{display:flex; align-items:center; flex-wrap:wrap; gap:10px; justify-content:center; margin:24px 0}
|
||
.node{background:var(--panel2); border:1px solid var(--border); border-radius:10px;
|
||
padding:12px 16px; text-align:center; min-width:130px}
|
||
.node .t{font-weight:700} .node .s{font-size:.78rem; color:var(--muted)}
|
||
.node.file{border-color:var(--green)} .node.file .t{color:var(--green)}
|
||
.node.proc{border-color:var(--accent)} .node.proc .t{color:var(--accent)}
|
||
.node.out{border-color:var(--purple)} .node.out .t{color:var(--purple)}
|
||
.arrow{color:var(--muted); font-size:1.5rem; font-weight:bold}
|
||
.vthread{display:flex; flex-direction:column; align-items:center; gap:6px}
|
||
/* badges */
|
||
.badge{display:inline-block; border-radius:6px; padding:2px 8px; font-size:.75rem; font-weight:600; border:1px solid}
|
||
.b-green{color:var(--green); border-color:var(--green); background:rgba(63,185,80,.08)}
|
||
.b-amber{color:var(--amber); border-color:var(--amber); background:rgba(210,153,34,.08)}
|
||
.b-red{color:var(--red); border-color:var(--red); background:rgba(248,81,73,.08)}
|
||
.b-blue{color:var(--accent); border-color:var(--accent); background:rgba(88,166,255,.08)}
|
||
/* callouts */
|
||
.call{border-left:4px solid var(--accent); background:var(--panel); border-radius:0 10px 10px 0; padding:12px 18px; margin:16px 0}
|
||
.call.warn{border-left-color:var(--amber)}
|
||
.call.danger{border-left-color:var(--red)}
|
||
.call.ok{border-left-color:var(--green)}
|
||
.call .h{font-weight:700; margin-bottom:4px}
|
||
/* step list */
|
||
.steps{counter-reset:step; list-style:none; padding:0}
|
||
.steps li{position:relative; padding:14px 16px 14px 58px; margin:12px 0; background:var(--panel); border:1px solid var(--border); border-radius:10px}
|
||
.steps li::before{counter-increment:step; content:counter(step);
|
||
position:absolute; left:16px; top:14px; width:28px; height:28px; border-radius:50%;
|
||
background:var(--accent); color:#06121f; font-weight:800; display:flex; align-items:center; justify-content:center}
|
||
/* kpi */
|
||
.kpi{display:flex; gap:14px; flex-wrap:wrap; margin:18px 0}
|
||
.kpi .k{flex:1; min-width:150px; background:var(--panel); border:1px solid var(--border); border-radius:12px; padding:16px; text-align:center}
|
||
.kpi .k .v{font-size:1.7rem; font-weight:800; color:var(--cyan)}
|
||
.kpi .k .l{font-size:.8rem; color:var(--muted)}
|
||
footer{color:var(--muted); text-align:center; padding:30px; border-top:1px solid var(--border); margin-top:60px; font-size:.85rem}
|
||
.legend{display:flex; gap:18px; flex-wrap:wrap; font-size:.82rem; color:var(--muted); justify-content:center; margin-top:10px}
|
||
.legend span{display:flex; align-items:center; gap:6px}
|
||
.dot{width:12px; height:12px; border-radius:3px; display:inline-block}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
|
||
<header class="hero">
|
||
<h1>📱 App Store Scraper</h1>
|
||
<p>Architecture & Data-Flow Reference</p>
|
||
<p class="muted">Google Play + Apple App Store · discovery → scraping → combined CSV · scaled across 2 PCs</p>
|
||
<span class="pill">Self-contained document — open in any browser</span>
|
||
</header>
|
||
|
||
<div class="wrap">
|
||
|
||
<nav class="toc">
|
||
<h4>Contents</h4>
|
||
<ol>
|
||
<li><a href="#overview">Big picture</a></li>
|
||
<li><a href="#pipeline">The pipeline</a></li>
|
||
<li><a href="#files">Files in the project</a></li>
|
||
<li><a href="#discover">Phase 1 — Find apps (discover.py)</a></li>
|
||
<li><a href="#appsjson">apps.json (the app list)</a></li>
|
||
<li><a href="#scrape">Phase 2 — Get data (scraper.py)</a></li>
|
||
<li><a href="#safety">Safety layer (netutil.py)</a></li>
|
||
<li><a href="#concurrency">Concurrency (threads)</a></li>
|
||
<li><a href="#twopc">Two-PC architecture</a></li>
|
||
<li><a href="#resume">Resume & crash safety</a></li>
|
||
<li><a href="#config">Configuration & flags</a></li>
|
||
<li><a href="#cheatsheet">Command cheat sheet</a></li>
|
||
<li><a href="#limits">Limits & ground truth</a></li>
|
||
</ol>
|
||
</nav>
|
||
|
||
<!-- ============================================================= -->
|
||
<h2 id="overview"><span class="num">0</span>Big picture</h2>
|
||
<p>
|
||
There is <strong>no "all apps" endpoint</strong> in either store. So the system works in
|
||
<strong>two phases</strong>: first it <em>discovers</em> a big list of app IDs, then it
|
||
<em>scrapes</em> the details for every app on that list into one CSV. Everything is free —
|
||
no API keys, no paid services.
|
||
</p>
|
||
|
||
<div class="kpi">
|
||
<div class="k"><div class="v">2</div><div class="l">stores covered<br>(Play + App Store)</div></div>
|
||
<div class="k"><div class="v">~30</div><div class="l">apps / Google Play query<br>(hard server cap)</div></div>
|
||
<div class="k"><div class="v">~180</div><div class="l">apps / iTunes search term<br>(the volume engine)</div></div>
|
||
<div class="k"><div class="v">100k+</div><div class="l">target catalog size<br>(App Store driven)</div></div>
|
||
</div>
|
||
|
||
<div class="call ok">
|
||
<div class="h">Two verbs, two phases</div>
|
||
<strong>FIND apps</strong> → <code>discover.py</code> writes <code>apps.json</code>.
|
||
<strong>GET data</strong> → <code>scraper.py</code> reads <code>apps.json</code> and writes <code>output/app_data.csv</code>.
|
||
</div>
|
||
|
||
<!-- ============================================================= -->
|
||
<h2 id="pipeline"><span class="num">1</span>The pipeline</h2>
|
||
<p>End-to-end, data flows left to right. The same flow runs on each PC; only the final merge is shared.</p>
|
||
|
||
<div class="flow">
|
||
<div class="node proc"><div class="t">discover.py</div><div class="s">FIND apps</div></div>
|
||
<span class="arrow">→</span>
|
||
<div class="node file"><div class="t">apps.json</div><div class="s">list of app IDs</div></div>
|
||
<span class="arrow">→</span>
|
||
<div class="node proc"><div class="t">scraper.py</div><div class="s">GET data</div></div>
|
||
<span class="arrow">→</span>
|
||
<div class="node out"><div class="t">app_data.csv</div><div class="s">one row / app / country</div></div>
|
||
<span class="arrow">→</span>
|
||
<div class="node proc"><div class="t">merge_csv.py</div><div class="s">combine 2 PCs</div></div>
|
||
</div>
|
||
<div class="legend">
|
||
<span><span class="dot" style="background:var(--accent)"></span>process (script)</span>
|
||
<span><span class="dot" style="background:var(--green)"></span>data file</span>
|
||
<span><span class="dot" style="background:var(--purple)"></span>output</span>
|
||
</div>
|
||
|
||
<table>
|
||
<tr><th>Step</th><th>Script</th><th>Reads</th><th>Writes</th><th>Purpose</th></tr>
|
||
<tr><td>Find</td><td><code>discover.py</code></td><td>search APIs + charts</td><td><code>apps.json</code></td><td>build a big list of app IDs</td></tr>
|
||
<tr><td>Get</td><td><code>scraper.py</code></td><td><code>apps.json</code></td><td><code>output/app_data.csv</code></td><td>fetch metadata + ratings for each app</td></tr>
|
||
<tr><td>Merge</td><td><code>merge_csv.py</code></td><td>shard CSVs</td><td><code>output/app_data.csv</code></td><td>combine the two PCs' results, de-duped</td></tr>
|
||
</table>
|
||
|
||
<!-- ============================================================= -->
|
||
<h2 id="files"><span class="num">2</span>Files in the project</h2>
|
||
<table>
|
||
<tr><th>File</th><th>Role</th></tr>
|
||
<tr><td><code>discover.py</code></td><td><span class="badge b-blue">Phase 1</span> Finds app IDs and fills <code>apps.json</code></td></tr>
|
||
<tr><td><code>scraper.py</code></td><td><span class="badge b-blue">Phase 2</span> Fetches per-app data into the CSV (threaded, shardable, resumable)</td></tr>
|
||
<tr><td><code>netutil.py</code></td><td><span class="badge b-green">Shared safety</span> Error classification + adaptive throttle / circuit breaker</td></tr>
|
||
<tr><td><code>merge_csv.py</code></td><td><span class="badge b-green">Helper</span> Merges the two PCs' shard CSVs into one</td></tr>
|
||
<tr><td><code>apps.json</code></td><td><span class="badge b-amber">Data</span> Settings + the list of app IDs (the contract between the two phases)</td></tr>
|
||
<tr><td><code>english_words.txt</code></td><td><span class="badge b-amber">Data</span> ~9,868 common words used as deep search terms (path to 100k)</td></tr>
|
||
<tr><td><code>requirements.txt</code></td><td>Dependencies: <code>google-play-scraper</code>, <code>requests</code></td></tr>
|
||
<tr><td><code>output/app_data.csv</code></td><td>The final dataset</td></tr>
|
||
</table>
|
||
|
||
<!-- ============================================================= -->
|
||
<h2 id="discover"><span class="num">3</span>Phase 1 — Find apps <span class="muted" style="font-size:1rem">(discover.py)</span></h2>
|
||
<p>
|
||
Because no full catalog exists, discovery <strong>queries many sources with many terms</strong>
|
||
and keeps only the unique app IDs. Three sources feed one de-duplicated set per store.
|
||
</p>
|
||
|
||
<h3>The three discovery sources</h3>
|
||
<table>
|
||
<tr><th>Source</th><th>Store</th><th>How</th><th>Yield / call</th><th>Role</th></tr>
|
||
<tr><td>Keyword search</td><td>Google Play</td><td><code>google_play_scraper.search(term)</code></td><td><span class="badge b-amber">~30</span> (hard cap)</td><td>breadth via many terms</td></tr>
|
||
<tr><td>iTunes Search API</td><td>App Store</td><td><code>itunes.apple.com/search?term=</code></td><td><span class="badge b-green">~180</span></td><td><strong>main volume engine</strong></td></tr>
|
||
<tr><td>RSS top charts</td><td>App Store</td><td><code>.../rss/{feed}/genre={id}</code> (free/grossing/paid × 23 genres)</td><td>up to 200</td><td>popular apps, fast</td></tr>
|
||
</table>
|
||
|
||
<h3>How the search terms are built</h3>
|
||
<p>The function <code>build_terms()</code> assembles the query list from up to four tiers:</p>
|
||
<div class="flow">
|
||
<div class="node"><div class="t">~155 curated</div><div class="s">topic keywords</div></div>
|
||
<span class="arrow">+</span>
|
||
<div class="node"><div class="t">a–z</div><div class="s">26 letters</div></div>
|
||
<span class="arrow">+</span>
|
||
<div class="node"><div class="t">aa–zz</div><div class="s"><code>--deep</code> (676)</div></div>
|
||
<span class="arrow">+</span>
|
||
<div class="node"><div class="t">wordlist</div><div class="s"><code>--terms-file</code> (~9,868)</div></div>
|
||
<span class="arrow">=</span>
|
||
<div class="node out"><div class="t">~9,943 terms</div><div class="s">de-duped, ordered</div></div>
|
||
</div>
|
||
<p class="muted">
|
||
More terms = more apps. The wordlist (<code>--terms-file english_words.txt</code>) is the single
|
||
biggest lever toward 100k. <code>--deeper</code> (aaa–zzz, 17,576 terms) exists but is very slow.
|
||
</p>
|
||
|
||
<h3>Discovery flow (per store)</h3>
|
||
<ol class="steps">
|
||
<li>Build the term list and resolve the country list (from <code>--countries</code> or settings).</li>
|
||
<li>For each country, loop every term; call the source through the safety layer (retry + backoff).</li>
|
||
<li>Collect each result's <strong>app ID</strong> into a <code>seen</code> dictionary — duplicates are ignored automatically.</li>
|
||
<li>Print live progress: <code>total unique = N / target</code>.</li>
|
||
<li>Stop as soon as <code>--target</code> is reached (or terms are exhausted).</li>
|
||
<li><strong>Merge</strong> into the existing <code>apps.json</code> (keeps your settings + anything already listed), then write the file.</li>
|
||
</ol>
|
||
|
||
<div class="call">
|
||
<div class="h">Why Google Play can't reach 100k</div>
|
||
Play search returns only <strong>~30 apps per query no matter what</strong> — that's Google's cap, not the
|
||
code's. So Play contributes ~15–30k; the App Store's ~180/term search is what carries you to 100k.
|
||
</div>
|
||
|
||
<div class="call ok">
|
||
<div class="h">De-duplication is automatic and idempotent</div>
|
||
Discovery keys on app ID, and <code>merge()</code> drops duplicates against what's already in
|
||
<code>apps.json</code>. Re-running only <em>adds</em> new apps — it never loses or doubles existing ones.
|
||
If <code>apps.json</code> is missing, a fresh one is created automatically.
|
||
</div>
|
||
|
||
<!-- ============================================================= -->
|
||
<h2 id="appsjson"><span class="num">4</span>apps.json — the app list</h2>
|
||
<p>This file is the <strong>contract</strong> between the two phases: discovery writes it, scraping reads it.</p>
|
||
<pre><code>{
|
||
"settings": {
|
||
"country": "us", // default single country
|
||
"countries": ["us","gb","in","ca","au"], // scrape/discover across these
|
||
"lang": "en", // language (Google Play)
|
||
"review_count": 100,
|
||
"delay_seconds": 1.0 // polite pause; rate ≈ workers / delay
|
||
},
|
||
"google_play": [
|
||
{ "app_id": "com.whatsapp" } // the id= value from the Play URL (string)
|
||
],
|
||
"app_store": [
|
||
{ "app_id": 310633997, "name": "WhatsApp" } // numeric id from the App Store URL
|
||
]
|
||
}</code></pre>
|
||
<table>
|
||
<tr><th>Store</th><th>ID type</th><th>Where it comes from</th></tr>
|
||
<tr><td>Google Play</td><td>string</td><td><code>play.google.com/store/apps/details?id=<strong>com.whatsapp</strong></code></td></tr>
|
||
<tr><td>App Store</td><td>number</td><td><code>apps.apple.com/us/app/.../id<strong>310633997</strong></code></td></tr>
|
||
</table>
|
||
|
||
<!-- ============================================================= -->
|
||
<h2 id="scrape"><span class="num">5</span>Phase 2 — Get data <span class="muted" style="font-size:1rem">(scraper.py)</span></h2>
|
||
<p>
|
||
Scraping turns the ID list into real data. For <strong>every app × every country</strong> it makes one
|
||
"detail" call and writes one CSV row. Two stores, two API calls, one unified row schema.
|
||
</p>
|
||
|
||
<h3>The two detail calls</h3>
|
||
<div class="grid c2">
|
||
<div class="panel">
|
||
<h3 style="margin-top:0">Google Play</h3>
|
||
<p><code>google_play_scraper.app(app_id, lang, country)</code></p>
|
||
<p class="muted">Scrapes the public listing page → title, developer, genre, price, score, ratings count, review count, last-updated, version, URL.</p>
|
||
</div>
|
||
<div class="panel">
|
||
<h3 style="margin-top:0">Apple App Store</h3>
|
||
<p><code>itunes.apple.com/lookup?id=&country=</code></p>
|
||
<p class="muted">Official lookup API → trackName, seller, genre, price, average rating, rating count, release date, version, URL.</p>
|
||
</div>
|
||
</div>
|
||
|
||
<h3>Output columns — <code>output/app_data.csv</code></h3>
|
||
<table>
|
||
<tr><th>Column</th><th>Meaning</th></tr>
|
||
<tr><td><code>store</code></td><td><code>google_play</code> or <code>app_store</code></td></tr>
|
||
<tr><td><code>country</code></td><td>which region this row was fetched for</td></tr>
|
||
<tr><td><code>app_id</code>, <code>title</code>, <code>developer</code>, <code>category</code></td><td>identity</td></tr>
|
||
<tr><td><code>price</code>, <code>currency</code>, <code>free</code></td><td>pricing</td></tr>
|
||
<tr><td><code>avg_rating</code></td><td>final average star rating</td></tr>
|
||
<tr><td><code>total_ratings</code></td><td>total number of ratings</td></tr>
|
||
<tr><td><code>text_review_count</code></td><td>number of written reviews (Play only)</td></tr>
|
||
<tr><td><code>last_updated</code></td><td>date the app was last updated</td></tr>
|
||
<tr><td><code>version</code></td><td>latest version</td></tr>
|
||
<tr><td><code>url</code></td><td>store listing link</td></tr>
|
||
</table>
|
||
<p class="muted">CSV is UTF-8 with BOM → opens cleanly in Excel. The same columns apply to both stores.</p>
|
||
|
||
<h3>How one job is processed</h3>
|
||
<div class="flow">
|
||
<div class="node">job<br><span class="s">(store, id, country)</span></div>
|
||
<span class="arrow">→</span>
|
||
<div class="node proc">throttle.wait()<br><span class="s">honor cooldown</span></div>
|
||
<span class="arrow">→</span>
|
||
<div class="node proc">fetch<br><span class="s">Play / App Store</span></div>
|
||
<span class="arrow">→</span>
|
||
<div class="node out">row<br><span class="s">flushed to CSV</span></div>
|
||
</div>
|
||
|
||
<!-- ============================================================= -->
|
||
<h2 id="safety"><span class="num">6</span>Safety layer <span class="muted" style="font-size:1rem">(netutil.py)</span></h2>
|
||
<div class="call danger">
|
||
<div class="h">The #1 rule: slow is fine — getting the IP blocked is not.</div>
|
||
Every request (in both scripts) passes through a shared throttle. The moment a store pushes back,
|
||
<strong>all workers slow down together</strong> instead of hammering until blocked.
|
||
</div>
|
||
|
||
<h3>Error classification</h3>
|
||
<table>
|
||
<tr><th>Signal</th><th>Meaning</th><th>Reaction</th></tr>
|
||
<tr><td><span class="badge b-red">429 / 403</span> "too many requests", "quota"</td><td>rate-limited</td><td><strong>Trip circuit breaker</strong>, wait it out, retry generously</td></tr>
|
||
<tr><td><span class="badge b-amber">408 / 5xx</span>, timeout, connection</td><td>transient hiccup</td><td>short exponential backoff, a few retries</td></tr>
|
||
<tr><td><span class="badge b-green">404</span> / not-found</td><td>healthy answer</td><td>skip immediately — <em>no</em> wasted retries</td></tr>
|
||
</table>
|
||
|
||
<h3>The circuit breaker (a shared state machine)</h3>
|
||
<div class="flow">
|
||
<div class="node" style="border-color:var(--green)"><div class="t" style="color:var(--green)">RUNNING</div><div class="s">requests flow freely</div></div>
|
||
<span class="arrow">→</span>
|
||
<div class="node" style="border-color:var(--red)"><div class="t" style="color:var(--red)">TRIPPED</div><div class="s">all workers pause</div></div>
|
||
<span class="arrow">→</span>
|
||
<div class="node" style="border-color:var(--amber)"><div class="t" style="color:var(--amber)">COOLDOWN</div><div class="s">30→60→120…→300s</div></div>
|
||
<span class="arrow">→</span>
|
||
<div class="node" style="border-color:var(--green)"><div class="t" style="color:var(--green)">RECOVER</div><div class="s">speed back up on success</div></div>
|
||
</div>
|
||
<ul>
|
||
<li><strong>Trips on</strong> a rate-limit signal <em>or</em> 5 consecutive unexplained failures (catches "stealth" blocks that don't return a clean 429).</li>
|
||
<li><strong>Cooldown doubles</strong> each time it re-trips (30s → 60 → 120 … capped at 5 min) and lingers as extra spacing.</li>
|
||
<li><strong>Recovers automatically</strong>: each success shrinks the cooldown and extra delay back toward normal.</li>
|
||
</ul>
|
||
|
||
<div class="call warn">
|
||
<div class="h">Reading the logs</div>
|
||
A line like <code>THROTTLING: pausing all workers ~60s</code> is the protection <em>working</em>, not an error.
|
||
If it fires constantly, that IP is at its limit → lower <code>--workers</code> on that machine.
|
||
</div>
|
||
|
||
<!-- ============================================================= -->
|
||
<h2 id="concurrency"><span class="num">7</span>Concurrency — why threads</h2>
|
||
<p>
|
||
These are <strong>network-wait</strong> tasks (most time is spent waiting for the server, not computing),
|
||
which is the ideal case for a <strong>thread pool</strong>. With <code>--workers N</code>, up to N requests are
|
||
in flight at once.
|
||
</p>
|
||
<div class="flow">
|
||
<div class="node file"><div class="t">job queue</div><div class="s">all apps × countries</div></div>
|
||
<span class="arrow">→</span>
|
||
<div class="vthread">
|
||
<div class="node proc" style="min-width:200px">worker 1</div>
|
||
<div class="node proc" style="min-width:200px">worker 2</div>
|
||
<div class="node proc" style="min-width:200px">… worker N</div>
|
||
</div>
|
||
<span class="arrow">→</span>
|
||
<div class="node out"><div class="t">CSV sink</div><div class="s">thread-safe, flush per row</div></div>
|
||
</div>
|
||
<p>
|
||
Effective request rate ≈ <code>workers ÷ delay_seconds</code>. The throttle sits above all workers,
|
||
so one rate-limit signal pauses the whole pool.
|
||
</p>
|
||
|
||
<!-- ============================================================= -->
|
||
<h2 id="twopc"><span class="num">8</span>Two-PC architecture <span class="muted" style="font-size:1rem">(2 CPUs, 2 IPs)</span></h2>
|
||
<p>
|
||
Two machines with <strong>separate IPs</strong> double throughput <em>and</em> the safe request budget
|
||
(each store rate-limits per IP). You don't split the file by hand — both PCs read the <em>same</em>
|
||
<code>apps.json</code> and each takes a different <strong>shard</strong>.
|
||
</p>
|
||
|
||
<h3>How sharding splits the work</h3>
|
||
<p>
|
||
All jobs are flattened to a list, then each PC takes <code>jobs[shard::shards]</code> — i.e. every Nth job.
|
||
This interleaves Play/App-Store/country evenly, so neither PC gets a lopsided slice.
|
||
</p>
|
||
<div class="flow">
|
||
<div class="node file" style="min-width:200px"><div class="t">apps.json</div><div class="s">jobs: 0,1,2,3,4,5,6,7…</div></div>
|
||
</div>
|
||
<div class="grid c2">
|
||
<div class="panel">
|
||
<h3 style="margin-top:0">PC #1 — IP A</h3>
|
||
<p><code>--shard 0 --shards 2</code></p>
|
||
<p>does jobs <strong>0, 2, 4, 6 …</strong></p>
|
||
<p class="muted">writes <code>app_data_shard0of2.csv</code></p>
|
||
</div>
|
||
<div class="panel">
|
||
<h3 style="margin-top:0">PC #2 — IP B</h3>
|
||
<p><code>--shard 1 --shards 2</code></p>
|
||
<p>does jobs <strong>1, 3, 5, 7 …</strong></p>
|
||
<p class="muted">writes <code>app_data_shard1of2.csv</code></p>
|
||
</div>
|
||
</div>
|
||
<div class="flow">
|
||
<div class="node out">shard0 CSV</div>
|
||
<span class="arrow">↘</span>
|
||
<div class="node proc"><div class="t">merge_csv.py</div><div class="s">de-duped by store+country+id</div></div>
|
||
<span class="arrow">↙</span>
|
||
<div class="node out">shard1 CSV</div>
|
||
</div>
|
||
<div class="flow">
|
||
<span class="arrow">↓</span>
|
||
</div>
|
||
<div class="flow">
|
||
<div class="node out"><div class="t">app_data.csv</div><div class="s">complete combined dataset</div></div>
|
||
</div>
|
||
|
||
<h3>The 3 steps end-to-end</h3>
|
||
<ol class="steps">
|
||
<li><strong>Discover once</strong> (either PC): run <code>discover.py</code>, then copy the resulting <code>apps.json</code> to both machines.</li>
|
||
<li><strong>Scrape in parallel</strong>: PC #1 runs shard 0, PC #2 runs shard 1 (each with its own IP and its own thread pool).</li>
|
||
<li><strong>Merge</strong>: copy both shard CSVs into one <code>output/</code> folder and run <code>merge_csv.py</code>.</li>
|
||
</ol>
|
||
|
||
<div class="kpi">
|
||
<div class="k"><div class="v">÷2</div><div class="l">work per PC<br>(sharding)</div></div>
|
||
<div class="k"><div class="v">×~10</div><div class="l">per PC<br>(thread pool)</div></div>
|
||
<div class="k"><div class="v">×2 IPs</div><div class="l">safe rate budget<br>(per-IP limits)</div></div>
|
||
<div class="k"><div class="v">28h → ~1h</div><div class="l">rough wall-clock<br>for 100k records</div></div>
|
||
</div>
|
||
|
||
<!-- ============================================================= -->
|
||
<h2 id="resume"><span class="num">9</span>Resume & crash safety</h2>
|
||
<p>A 100k run can't depend on a single perfect execution. Two mechanisms make interruptions harmless:</p>
|
||
<div class="grid c2">
|
||
<div class="panel">
|
||
<h3 style="margin-top:0">💾 Flush-as-you-go</h3>
|
||
<p>Each row is written and flushed to the CSV <strong>the instant it's fetched</strong>. A crash, block,
|
||
or <kbd>Ctrl-C</kbd> never loses completed work.</p>
|
||
</div>
|
||
<div class="panel">
|
||
<h3 style="margin-top:0">⏯️ Resume on restart</h3>
|
||
<p>On startup the scraper reads the existing CSV and <strong>skips apps already done</strong>
|
||
(keyed by store + country + id). Just re-run the same command to continue.</p>
|
||
</div>
|
||
</div>
|
||
<div class="call ok">
|
||
<div class="h">Practical meaning</div>
|
||
If a PC is interrupted at app 40,000 of 50,000, re-running the exact same command picks up at 40,001.
|
||
Use <code>--no-resume</code> only if you deliberately want to re-scrape everything.
|
||
</div>
|
||
|
||
<!-- ============================================================= -->
|
||
<h2 id="config"><span class="num">10</span>Configuration & flags</h2>
|
||
|
||
<h3>discover.py</h3>
|
||
<table>
|
||
<tr><th>Flag</th><th>Default</th><th>Purpose</th></tr>
|
||
<tr><td><code>--target N</code></td><td>0 (all terms)</td><td>stop each store after N unique apps</td></tr>
|
||
<tr><td><code>--terms-file PATH</code></td><td>—</td><td>extra search terms (best lever to 100k)</td></tr>
|
||
<tr><td><code>--deep</code></td><td>off</td><td>add 2-letter terms aa–zz (676)</td></tr>
|
||
<tr><td><code>--deeper</code></td><td>off</td><td>add 3-letter terms aaa–zzz (17,576; slow)</td></tr>
|
||
<tr><td><code>--countries a,b,c</code></td><td>settings</td><td>regions to discover across</td></tr>
|
||
<tr><td><code>--skip-play</code> / <code>--skip-appstore</code></td><td>off</td><td>run only one store</td></tr>
|
||
<tr><td><code>--search-limit</code> / <code>--chart-limit</code></td><td>200 / 200</td><td>results per iTunes term / per chart</td></tr>
|
||
<tr><td><code>--hits</code></td><td>30</td><td>Play results per term (server caps ~30)</td></tr>
|
||
<tr><td><code>--delay</code></td><td>0.5</td><td>polite pause between discovery requests</td></tr>
|
||
</table>
|
||
|
||
<h3>scraper.py</h3>
|
||
<table>
|
||
<tr><th>Flag</th><th>Default</th><th>Purpose</th></tr>
|
||
<tr><td><code>--workers N</code></td><td>10</td><td>concurrent requests (thread pool)</td></tr>
|
||
<tr><td><code>--shard i</code></td><td>0</td><td>this machine's slice index</td></tr>
|
||
<tr><td><code>--shards N</code></td><td>1</td><td>total machines splitting the work</td></tr>
|
||
<tr><td><code>--countries a,b,c</code></td><td>settings</td><td>regions to scrape (one row each)</td></tr>
|
||
<tr><td><code>--no-resume</code></td><td>off</td><td>ignore existing CSV, scrape all again</td></tr>
|
||
<tr><td><code>--out DIR</code> / <code>--config PATH</code></td><td>output / apps.json</td><td>output folder / input file</td></tr>
|
||
</table>
|
||
|
||
<!-- ============================================================= -->
|
||
<h2 id="cheatsheet"><span class="num">11</span>Command cheat sheet</h2>
|
||
|
||
<h3>① Discover ~100k apps (App Store driven)</h3>
|
||
<pre><code># fastest route to 1 lakh — App Store only, using the wordlist
|
||
python discover.py --skip-play --target 100000 --terms-file english_words.txt --countries us
|
||
|
||
# include Google Play too (adds ~15–30k; can't reach 100k on its own)
|
||
python discover.py --target 100000 --terms-file english_words.txt --countries us,gb</code></pre>
|
||
|
||
<h3>② Scrape across 2 PCs</h3>
|
||
<pre><code># --- PC #1 (IP A) ---
|
||
python scraper.py --shard 0 --shards 2 --workers 10
|
||
|
||
# --- PC #2 (IP B) ---
|
||
python scraper.py --shard 1 --shards 2 --workers 10
|
||
|
||
# ...interrupted? just run the same command again — it resumes.</code></pre>
|
||
|
||
<h3>③ Merge the two PCs' output</h3>
|
||
<pre><code># copy both app_data_shard*of2.csv into one output/ folder, then:
|
||
python merge_csv.py</code></pre>
|
||
|
||
<h3>Single-PC run (no sharding)</h3>
|
||
<pre><code>python scraper.py --workers 10</code></pre>
|
||
|
||
<!-- ============================================================= -->
|
||
<h2 id="limits"><span class="num">12</span>Limits & ground truth</h2>
|
||
<table>
|
||
<tr><th>Reality</th><th>Detail</th></tr>
|
||
<tr><td>No full catalog</td><td>Neither store lists "all apps"; you only get what discovery finds.</td></tr>
|
||
<tr><td>Google Play search cap</td><td><strong>~30 results per query</strong> — Play tops out ~15–30k regardless of terms.</td></tr>
|
||
<tr><td>App Store is the volume</td><td>iTunes search ~180/term → this is what reaches 100k.</td></tr>
|
||
<tr><td>Per-IP rate limits</td><td>Both stores throttle per IP; 2 PCs/IPs ≈ 2× safe budget. Proxies extend further.</td></tr>
|
||
<tr><td>Per-country data</td><td>Ratings & pricing differ by country → one row per app per country.</td></tr>
|
||
<tr><td>Terms of Service</td><td>Both stores restrict automated scraping. Fine for internal research at modest scale; get sign-off before commercial redistribution.</td></tr>
|
||
</table>
|
||
|
||
<div class="call warn">
|
||
<div class="h">Tuning rule of thumb</div>
|
||
Start at <code>--workers 10</code>. If you see constant <code>THROTTLING</code> lines, <strong>lower</strong> workers
|
||
(e.g. 6) — fewer cooldowns beats pushing too hard. Back up <code>apps.json</code> after a big discovery run;
|
||
it's the expensive artifact.
|
||
</div>
|
||
|
||
</div>
|
||
|
||
<footer>
|
||
App Store Scraper — Architecture & Flow · Google Play + Apple App Store · self-contained reference (no internet needed to view)
|
||
</footer>
|
||
|
||
</body>
|
||
</html>
|