dhruv.godvani/App_Scraper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
5 Commits 1 Branch 0 Tags
main
Commit Graph

3 Commits

Author SHA1 Message Date
dhruv.godvani 22582bdcd7 Add developer email and website fields to CSV output 
- Updated CSV_FIELDS to include "developer_email" and "developer_website".
- Modified play_row function to extract developer email and website from the response.
- Updated appstore_row function to include developer website, with a note that developer email is not available from Apple's API.
 2026-06-18 12:36:17 +05:30
dhruv.godvani 159c00762d feat: enhance CSV handling with backup snapshots and last updated date parsing 
- Added support for parsing multiple date formats for the "lastUpdatedOn" field from Google Play.
- Implemented a backup mechanism in CsvSink to create snapshots of the CSV file every N rows, with a configurable number of backups to keep.
- Updated the CsvSink class to handle file flushing and backup creation safely.
- Modified the run function to allow configuration of backup settings via command-line arguments.
 2026-06-17 19:25:06 +05:30
dhruv.godvani 03bf0fc923 Initial commit: App Store + Google Play scraper 
- discover.py: find app IDs (Play search, iTunes Search API, RSS charts) -> apps.json
- scraper.py: fetch metadata + ratings per app/country, threaded, shardable, resumable
- netutil.py: shared retry/backoff + adaptive circuit breaker (rate-limit safe)
- merge_csv.py: combine sharded outputs from multiple machines
- ARCHITECTURE.html: full architecture & flow documentation
- english_words.txt: deep search-term wordlist

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-17 17:54:18 +05:30