Python Web Scraping Engineer - Large Scale Data Extraction

Job Title: Python Web Scraping Engineer - Large Scale Data Extraction

Job Description:
We are seeking a Python Web Scraper for a large-scale data extraction project. This role involves developing, executing, and managing web scraping processes for the scraping of 10,000,000 webpages, using multiple cloud instances. The ideal candidate will possess the ability to efficiently utilize server resources, navigate the challenges of large data transfer, and maintain system integrity throughout the project.

Responsibilities:
*Develop and execute a Python-based web scraping process to extract data from 10,000,000 webpages.
*Efficiently manage cloud instances (e.g., AWS EC2) for executing the scraping process.
*Monitor and manage the allocated server resources to ensure optimum performance.
*Handle data storage and transfer effectively, minimizing costs while maintaining data integrity.
*Implement and manage proxies or VPN services as required to avoid IP blocking.
*Regularly monitor and maintain the system, troubleshooting and resolving issues promptly.
*Collaborate with the team to ensure project completion within specified time frames.

Qualifications:
*Proven experience with Python, specifically in web scraping tasks.
*Experience with cloud platforms (e.g., AWS, Google Cloud, Azure).
*Familiarity with databases and data storage solutions.
*Understanding of web protocols and techniques to avoid IP blocking (e.g., proxies, VPNs).
*Strong problem-solving skills and the ability to work in a fast-paced environment.
*Excellent communication and team collaboration skills.
*Preferred Qualifications:
*Prior experience with large-scale web scraping projects.
*Knowledge of cost optimization techniques for cloud platforms.

Please apply to this link:
----------