dhruv.godvani/App_Scraper
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
4 Commits 1 Branch 0 Tags
159c00762dcbb3e95aa07075df3ceae7fbe64ca1
Commit Graph

4 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
dhruv.godvani 159c00762d feat: enhance CSV handling with backup snapshots and last updated date parsing 
- Added support for parsing multiple date formats for the "lastUpdatedOn" field from Google Play.
- Implemented a backup mechanism in CsvSink to create snapshots of the CSV file every N rows, with a configurable number of backups to keep.
- Updated the CsvSink class to handle file flushing and backup creation safely.
- Modified the run function to allow configuration of backup settings via command-line arguments.
 2026-06-17 19:25:06 +05:30
dhruv.godvani e958ede9ea Merge Gitea auto-init; keep project files 2026-06-17 17:55:51 +05:30
dhruv.godvani 03bf0fc923 Initial commit: App Store + Google Play scraper 
- discover.py: find app IDs (Play search, iTunes Search API, RSS charts) -> apps.json
- scraper.py: fetch metadata + ratings per app/country, threaded, shardable, resumable
- netutil.py: shared retry/backoff + adaptive circuit breaker (rate-limit safe)
- merge_csv.py: combine sharded outputs from multiple machines
- ARCHITECTURE.html: full architecture & flow documentation
- english_words.txt: deep search-term wordlist

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-17 17:54:18 +05:30
dhruv.godvani dcac573bcd Initial commit 2026-06-17 12:20:20 +00:00