Reddit Scraper (LLM Data)
Python scraper with rotating proxy support for collecting Reddit data for LLM training
A Python scraper leveraging the requests library with private rotating proxy support, designed to efficiently collect Reddit data for training or fine-tuning large language models.
Features
- Rotating proxy support to avoid rate limiting
- Configurable subreddit targeting
- Structured JSON output for LLM training pipelines
- Handles pagination, deleted posts, and edge cases
Use Case
Collecting high-quality conversational and domain-specific text data from Reddit for LLM fine-tuning and pre-training datasets.
Tech Stack
Python · requests · Rotating Proxies · JSON · Reddit API