{"loaderData":{"0":null,"0-0":null,"0-0-32":{"tag":"blog-author","pageNumber":2,"author":{"id":"6abcd3180ffe3e314374c36b","slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","longBio":"Igor is Head of Development at Proxy-Seller, leading the engineering team that builds and scales the platform. A DevOps specialist focused on CI/CD, automation, and reliability engineering that keeps proxy infrastructure fast, stable, and available under heavy load.","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","authorsFocus":"Web scraping & benchmarks","specialization":"CI/CD & automation","experience":"12+ yrs in DevOps","titleInCase":"Igor Efimenko’s articles","socialLinks":{"linkedin":"https://www.linkedin.com/in/igor-efimenko-99708612a/"},"photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"},"articlesCount":66,"totalViews":725,"avgReadTimeMinutes":9,"groups":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","name":"Web Scraping","count":66}],"topCategories":[{"category":"Web Scraping","posts":66}],"localeCode":"en"},"articles":{"totalElements":66,"totalPages":4,"content":[{"id":"6a049cdb35baad34714afd33","tag":"how-to-scrape-netflix-data-using-python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":4,"title":"How to scrape Netflix data using Python","body":"

Harvesting data from Netflix can provide in-depth details about movies and TV series including items like titles, dates of release, categories of content, and overview contents. This article demonstrates how to extract data from multiple Netflix movie pages using Python, requests, and lxml. Since Netflix doesn't provide an open API for movie data, scraping allows us to gather valuable content data that can support recommendations, content analysis, and other applications.

Requirements and installation

To start, ensure that requests and lxml libraries are installed. Use the following commands to set up your environment:

\npip install requests\npip install lxml\n\n

These libraries are essential for sending HTTP requests to the Netflix pages and parsing HTML content to extract required data.

Setting up the scraper

To access Netflix pages, we need a list of URLs that we’ll iterate through to retrieve movie details. This tutorial will scrape Netflix’s movie title, year, duration, description, genre, and more from each specified page URL.

Headers and proxy setup

Netflix employs strict anti-bot measures, so using correct headers and proxies (if necessary) can prevent detection. In this script, we mimic a real browser by setting up custom headers with a User-Agent, language preferences, and other parameters, making our requests look more legitimate.

\n\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'priority': 'u=0, i',\n    'sec-ch-ua': '\"Chromium\";v=\"130\", \"Google Chrome\";v=\"130\", \"Not?A_Brand\";v=\"99\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36',\n}\n\n

If needed, proxies can be added to make requests from different IP addresses, further reducing the likelihood of being flagged. Example of integrating a proxy with IP address authentication:

\n\nproxies = {\n    'http': 'http://your_proxy_address:port',\n    'https': 'http://your_proxy_address:port',\n}\nresponse = requests.get(url, headers=headers, proxies=proxies)\n\n

Step-by-step code breakdown

Step 1. Setting up URL list

We specify a list of Netflix movie URLs, which our script will iterate through to extract data.

\n\nurls_list = [\n    'Https link', \n    'Https link'\n]\n\n

Step 2. Sending the HTTP request

Each URL is accessed with the requests.get() method, passing the headers to avoid detection.

\n\nresponse = requests.get(url, headers=headers)\n\n

Step 3. Parsing the HTML content

Using lxml, we parse the HTML response to navigate and extract data using XPath expressions.

\nfrom lxml.html import fromstring\nparser = fromstring(response.text)\n\n

Step 4. Extracting data elements

Using XPath, we capture essential movie details like title, year, duration, description, genre, subtitles, and more. Below is how each field is extracted:

\ntitle = parser.xpath('//h1[@class=\"title-title\"]/text()')[0]\nyear = parser.xpath('//span[@data-uia=\"item-year\"]/text()')[0]\nduration = parser.xpath('//span[@class=\"duration\"]/text()')[0]\ndescription = parser.xpath('//div[@class=\"title-info-synopsis\"]/text()')[0]\nmaturity_number = parser.xpath('//span[@class=\"maturity-number\"]/text()')[0]\nstarring = parser.xpath('//span[@data-uia=\"info-starring\"]/text()')[0]\ngenre = parser.xpath('//a[@data-uia=\"item-genre\"]/text()')[0]\ngenres = parser.xpath('//span[@data-uia=\"more-details-item-genres\"]/a/text()')\nsubtitles = ''.join(parser.xpath('//span[@data-uia=\"more-details-item-subtitle\"]/text()'))\naudio = ''.join(parser.xpath('//span[@data-uia=\"more-details-item-audio\"]/text()'))\n\n

Step 5. Saving data in a list

We store each movie’s data in a dictionary and append it to a list. This approach keeps data organized and ready to be saved in CSV.

\n\ndata = {\n    'title': title,\n    'year': year,\n    'duration': duration,\n    'description': description,\n    'maturity_number': maturity_number,\n    'starring': starring,\n    'genre': genre,\n    'genres': genres,\n    'subtitles': subtitles,\n    'audio': audio\n}\nextracted_data.append(data)\n\n

Step 6. Writing data to CSV

Finally, after iterating through all URLs, we write the accumulated data to a CSV file.

\n\nimport csv\n\nwith open('netflix_data.csv', 'w', newline='', encoding='utf-8') as f:\n    writer = csv.DictWriter(f, fieldnames=extracted_data[0].keys())\n    writer.writeheader()\n    writer.writerows(extracted_data)\n\n

Complete code

The complete code combines all steps with headers and proxy setup.

\nimport requests\nfrom lxml.html import fromstring\nimport csv\n\nurls_list = [\n    'Https link', \n    'Https link'\n]\n\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'priority': 'u=0, i',\n    'sec-ch-ua': '\"Chromium\";v=\"130\", \"Google Chrome\";v=\"130\", \"Not?A_Brand\";v=\"99\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36',\n}\n\nproxies = {\n    'http': 'http://your_proxy_address:port',\n    'https': 'http://your_proxy_address:port',\n}\n\nextracted_data = []\nfor url in urls_list:\n    response = requests.get(url, headers=headers, proxies=proxies)\n    parser = fromstring(response.text)\n    \n    title = parser.xpath('//h1[@class=\"title-title\"]/text()')[0]\n    year = parser.xpath('//span[@data-uia=\"item-year\"]/text()')[0]\n    duration = parser.xpath('//span[@class=\"duration\"]/text()')[0]\n    description = parser.xpath('//div[@class=\"title-info-synopsis\"]/text()')[0]\n    maturity_number = parser.xpath('//span[@class=\"maturity-number\"]/text()')[0]\n    starring = parser.xpath('//span[@data-uia=\"info-starring\"]/text()')[0]\n    genre = parser.xpath('//a[@data-uia=\"item-genre\"]/text()')[0]\n    genres = parser.xpath('//span[@data-uia=\"more-details-item-genres\"]/a/text()')\n    subtitles = ''.join(parser.xpath('//span[@data-uia=\"more-details-item-subtitle\"]/text()'))\n    audio = ''.join(parser.xpath('//span[@data-uia=\"more-details-item-audio\"]/text()'))\n\n    data = {\n        'title': title,\n        'year': year,\n        'duration': duration,\n        'description': description,\n        'maturity_number': maturity_number,\n        'starring': starring,\n        'genre': genre,\n        'genres': genres,\n        'subtitles': subtitles,\n        'audio': audio\n    }\n    extracted_data.append(data)\n\nwith open('netflix_data.csv', 'w', newline='', encoding='utf-8') as f:\n    writer = csv.DictWriter(f, fieldnames=extracted_data[0].keys())\n    writer.writeheader()\n    writer.writerows(extracted_data)\nprint('saved into netflix_data.csv')\n\n

Scraping Netflix data with Python provides a practical way to access content details without an official API. By using headers, proxies, and parsing techniques, we gather and save valuable data effectively. This script can be customized for various streaming analytics, recommendations, or content monitoring, helping you utilize Netflix data for broader applications.

","createdDate":"2025-04-21T06:41:35.000+00:00","description":"Harvesting data from Netflix can provide in-depth details about movies and TV series including items like titles, dates of release, categories of content, and overview contents.","countOfViews":7,"attachments":[],"mainImg":{"id":"6a2497ac1079162e2e8b2bd8","url":"https://proxy-seller.com/files/fs/image/6a2497ac1079162e2e8b2bd8_how-to-scrape-netflix-data-using-python_mainImg.png","name":"how-to-scrape-netflix-data-using-python_mainImg.png","size":41388,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"how-to-scrape-netflix-data-using-python_img","metaTitle":"How to Scrape Data from Netflix Using Python","metaDescription":"Explore a detailed guide on extracting data from the Netflix platform. Step-by-step script creation using proxies."},{"id":"6a049cdd35baad34714afd49","tag":"how-to-scrape-tripadvisor-data-using-python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":4,"title":"How to scrape TripAdvisor data using Python","body":"

TripAdvisor is a widely used travel portal where users write reviews for accommodation, dining, sightseeing, etc. Enriching the Data with TripAdvisor reviews could be helpful for Travel Analysis, Competitor Studies, etc. In this guide, we shall cover how TripAdvisor data can be extracted using python and the data stored in a csv format.

Installation of required libraries

To build this scraper, we’ll use the following Python libraries:

requests: For making HTTP requests to get page content.
lxml: For parsing HTML content and extracting elements with XPath.

Install the required libraries using pip:

\npip install requests lxml\n\n

The role of headers and proxies in effective data scraping

When scraping data from websites like TripAdvisor, it's crucial to properly configure the request headers, especially the User-Agent. By setting this header, you can disguise your requests as those coming from legitimate users, which minimizes the risk of your scraping activities triggering blocks due to unusual traffic patterns. Additionally, employing proxy servers is essential to circumvent restrictions related to the number of requests permissible from a single IP address, thus facilitating more extensive data collection efforts.

Step-by-step guide to scraping TripAdvisor

We’ll go through the process of scraping a list of hotel pages, extracting details, and saving them into a CSV file. Let’s break down each part.

Step 1. Import libraries and set up URLs

To begin, import the necessary libraries:

\nimport requests\nfrom lxml.html import fromstring\nimport csv\n\n

Then, define a list of URLs for the hotel pages you plan to scrape data from:

\nurls_list = [\n    'Https link',\n    'Https link'\n]\n\n

Step 2. Setting headers for requests

To ensure that your requests mimic those from a real browser, it's crucial to configure headers properly. This step helps to bypass anti-bot protection systems on websites and minimizes the risk of being blocked.

\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'sec-ch-ua': '\"Chromium\";v=\"130\", \"Google Chrome\";v=\"130\", \"Not?A_Brand\";v=\"99\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36',\n}\n\n

Step 3. Using proxies

Proxies can help bypass IP-based restrictions. In the example below, we use a proxy with IP address authentication. Here’s how to add a proxy to requests.

\nproxies = {\n    'http': 'http://your_proxy_address:port',\n    'https': 'http://your_proxy_address:port',\n}\nresponse = requests.get(url, headers=headers, proxies=proxies)\n\n

Step 4. Sending requests and parsing the HTML

For each URL, send a request and parse the response HTML:

\nextracted_data = []\nfor url in urls_list:\n    response = requests.get(url, headers=headers)  # Add proxies=proxies if needed\n    parser = fromstring(response.text)\n\n

Step 5. Extracting data using XPath

Using XPath, we can target specific elements on the page:

\n title = parser.xpath('//h1[@data-automation=\"mainH1\"]/text()')[0]\n    about = parser.xpath('//div[@class=\"_T FKffI bmUTE\"]/div/div/text()')[0].strip()\n    images_url = parser.xpath('//div[@data-testid=\"media_window_test\"]/div/div/button/picture/source/@srcset')\n    price = parser.xpath('//div[@data-automation=\"commerce_module_visible_price\"]/text()')[0]\n    ratings = parser.xpath('//div[@class=\"jVDab W f u w GOdjs\"]/@aria-label')[0].split(' ')[0]\n    features = parser.xpath('//div[@class=\"f Q2 _Y tyUdl\"]/div[2]/span/span/span/text()')\n    reviews = parser.xpath('//span[@class=\"JguWG\"]/span//text()')\n    listing_by = parser.xpath('//div[@class=\"biGQs _P pZUbB KxBGd\"]/text()')[0]\n    similar_experiences = parser.xpath('//div[@data-automation=\"shelfCard\"]/a/@href')\n\n

Step 6. Store extracted data

Store the extracted information in a dictionary and append it to a list:

\ndata = {\n        'title': title,\n        'about': about,\n        'price': price,\n        'listing by': listing_by,\n        'ratings': ratings,\n        'image_urls': images_url,\n        'features': features,\n        'reviews': reviews,\n        'similar_experiences': similar_experiences\n    }\n    extracted_data.append(data)\n\n

Step 7. Saving data to CSV

After scraping, save the data into a CSV file:

\ncsv_columns = ['title', 'about', 'price', 'listing by', 'ratings', 'image_urls', 'features', 'reviews', 'similar_experiences']\n\nwith open(\"tripadvisor_data.csv\", 'w', newline='', encoding='utf-8') as csvfile:\n    writer = csv.DictWriter(csvfile, fieldnames=csv_columns)\n    writer.writeheader()\n    for data in extracted_data:\n        writer.writerow(data)\n\n

Complete code

\nimport requests\nfrom lxml.html import fromstring\nimport csv\n\nurls_list = [\n    'Https link',\n    'Https link'\n]\n\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'sec-ch-ua': '\"Chromium\";v=\"130\", \"Google Chrome\";v=\"130\", \"Not?A_Brand\";v=\"99\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36',\n}\n\nproxies = {\n    'http': 'http://your_proxy_address:port',\n    'https': 'http://your_proxy_address:port',\n}\n\nextracted_data = []\nfor url in urls_list:\n    response = requests.get(url, headers=headers, proxies=proxies)  \n    parser = fromstring(response.text)\n    \n    title = parser.xpath('//h1[@data-automation=\"mainH1\"]/text()')[0]\n    about = parser.xpath('//div[@class=\"_T FKffI bmUTE\"]/div/div/text()')[0].strip()\n    images_url = parser.xpath('//div[@data-testid=\"media_window_test\"]/div/div/button/picture/source/@srcset')\n    price = parser.xpath('//div[@data-automation=\"commerce_module_visible_price\"]/text()')[0]\n    ratings = parser.xpath('//div[@class=\"jVDab W f u w GOdjs\"]/@aria-label')[0].split(' ')[0]\n    features = parser.xpath('//div[@class=\"f Q2 _Y tyUdl\"]/div[2]/span/span/span/text()')\n    reviews = parser.xpath('//span[@class=\"JguWG\"]/span//text()')\n    listing_by = parser.xpath('//div[@class=\"biGQs _P pZUbB KxBGd\"]/text()')[0]\n    similar_experiences = parser.xpath('//div[@data-automation=\"shelfCard\"]/a/@href')\n    \n    data = {\n        'title': title,\n        'about': about,\n        'price': price,\n        'listing by': listing_by,\n        'ratings': ratings,\n        'image_urls': images_url,\n        'features': features,\n        'reviews': reviews,\n        'similar_experiences': similar_experiences\n    }\n    extracted_data.append(data)\n\ncsv_columns = ['title', 'about', 'price', 'listing by', 'ratings', 'image_urls', 'features', 'reviews', 'similar_experiences']\n\nwith open(\"tripadvisor_data.csv\", 'w', newline='', encoding='utf-8') as csvfile:\n    writer = csv.DictWriter(csvfile, fieldnames=csv_columns)\n    writer.writeheader()\n    for data in extracted_data:\n        writer.writerow(data)\n\nprint('saved into tripadvisor_data.csv')\n\n

This guide not only lays a technical foundation for data scraping but also opens up avenues for comprehensive analysis in the tourism sector. The methods and techniques detailed here empower users to delve deeper into market trends and consumer behaviors. Such insights are crucial for developing robust reputation management strategies and conducting competitive analyses. By leveraging this guide, users can enhance their understanding of the dynamics within the tourism industry, making it possible to devise informed, strategic decisions that drive success.

","createdDate":"2025-04-18T09:55:59.000+00:00","description":"TripAdvisor is a widely used travel portal where users write reviews for accommodation, dining, sightseeing, etc. Enriching the Data with TripAdvisor reviews could be helpful for Travel Analysis, Competitor Studies, etc.","countOfViews":7,"attachments":[],"mainImg":{"id":"6a2497d41079162e2e8b329b","url":"https://proxy-seller.com/files/fs/image/6a2497d41079162e2e8b329b_how-to-scrape-tripadvisor-data-using-python_mainImg.png","name":"how-to-scrape-tripadvisor-data-using-python_mainImg.png","size":44496,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"how-to-scrape-tripadvisor-data-using-python_img","metaTitle":"How to Scrape Data from TripAdvisor Using Python","metaDescription":"A step-by-step guide to building a Python script for collecting data from the TripAdvisor platform and saving it in CSV format."},{"id":"6a049cde35baad34714afd61","tag":"overview-of-the-scrapy-web-scraping-framework","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":13,"title":"Overview of the Scrapy Web Scraping Framework","body":"

Scrapy is a robust, high-level framework designed for web scraping and data extraction, making it ideal for tasks like data parsing, price monitoring, user behavior analysis, social media insights, and SEO analysis. This framework is equipped to handle large volumes of data efficiently. It includes built-in mechanisms for managing HTTP requests, error handling, and ensuring compliance with robots.txt, which are essential for navigating complex and large-scale web data collection projects. This review will delve into what Scrapy is, how it functions, and the features it offers to users, providing a comprehensive understanding of its capabilities and applications.

$\"1.png\"$

How the Scrapy Framework Works

The Scrapy framework is a powerful open-source web scraping tool written in Python, designed for high-efficiency crawling and extracting structured data from websites. Scrapy can:

organize extracted data into JSON and CSV formats;
directly store it in databases such as SQLite, MySQL, PostgreSQL, and MongoDB;
support various parsing methods, including CSS selectors and XPath;
handle JSON and XML responses, which is crucial when dealing with APIs.

Scrapy operates through “spiders” – specialized crawlers that follow defined instructions to navigate web pages and harvest data. These spiders are essentially scripts that identify and capture specific types of objects like text, images, or links. An interactive crawling shell provided by Scrapy allows for real-time testing and debugging of these spiders, greatly enhancing the setup and optimization process of the crawler.

Scrapy Components

Key components of the Scrapy architecture include:

Items: data captured from web pages is structured into items, which are Python objects formatted as key-value pairs. These items, customizable according to user needs, can encapsulate various attributes such as text and images and are easily exportable to formats like JSON or CSV, or directly to databases.
Item pipeline: after data is captured by spiders, it enters the item pipeline. This is a series of processing steps where data is validated, cleaned, and possibly stored. Each step in the pipeline is defined by a Python class capable of tasks like error correction and data cleaning.
Requests and responses: Scrapy manages its tasks asynchronously, enhancing efficiency by allowing prioritization of requests, automatic retries of failed requests, and handling of common web operations like redirects and cookies.
Export feed: Scrapy can generate feeds from extracted data in various formats such as JSON, CSV, or XML, and it supports output to databases or cloud storage, providing flexibility in how data is handled post-extraction.
AutoThrottle: this feature automatically manages the rate of requests sent by the spiders based on server load and response time, optimizing efficiency and minimizing the burden on the websites being scraped.
Selectors: these are tools within Scrapy that use CSS or XPath to identify and extract elements from web pages. Based on the powerful Parcel library, selectors help to precisely target data needed from complex web pages.
Services: Scrapy includes built-in services that enhance functionality, such as event logging, statistics collection, email notifications, and Telnet console access, which aid in monitoring and controlling the scraping process.

Overall, Scrapy stands out as one of the most robust and flexible web scraping tools available, suitable for everything from simple data extraction tasks to complex large-scale web mining projects.

Features of the Scrapy Tool

This section highlights the key features of the Scrapy framework:

speed in data collection and processing;
the ability to extend functionality;
portability.

These attributes differentiate Scrapy from its competitors and establish it as a popular choice in the web scraping arena.

Speed of Operation

Scrapy is powered by Twisted, an asynchronous open-source network engine. Unlike synchronous operations, where one task must be completed before another begins, Twisted allows for tasks to be executed in parallel. This means Scrapy spiders can send multiple requests and process responses simultaneously, enhancing speed and efficiency in data collection, particularly for large-scale projects or when scanning multiple sites simultaneously.

The speed of Scrapy is further boosted by several factors:

Request management: Scrapy supports request prioritization, allowing for strategic execution orders during the scraping process. This prioritization can be manually set at request creation and adjusted dynamically as the scraper operates.
Caching mechanisms: Scrapy employs caching to reuse previously gathered data, accelerating the scraping process, reducing delays, and lessening the load on servers. The framework supports various caching strategies, such as in-memory or disk-based storage, and allows customization of cache lifespan and other settings.
Optimized selectors: the framework utilizes selectors to pinpoint specific parts of an HTML document using XPath or CSS. These selectors can be directly applied through response objects or through methods like response.xpath() and response.css(), streamlining the data extraction process.
Parallelism and scalability: Scrapy enables users to set the level of parallelism, optimizing resource use according to the scale of the project, whether it’s gathering data from smaller sites or extensive web portals.
Interactive shell: Scrapy includes an interactive shell for testing hypotheses and observing crawler behavior in real time, which is invaluable for speeding up development and debugging.

$\"2en.png\"$

Together, these features establish Scrapy as one of the fastest tools (read more about other various tools) available for efficiently scraping and collecting data from a multitude of websites, making it an invaluable resource for tasks such as product price monitoring, job listings, news gathering, social media analysis, and academic research.

Extensibility of Functionality

Scrapy’s modular architecture enhances its adaptability and extensibility, making it well-suited for a variety of complex data collection tasks. Its support for integration with various data stores such as MongoDB, PostgreSQL, and Elasticsearch, as well as queue management systems like Redis and RabbitMQ, allows for the effective handling of large data volumes. Additionally, Scrapy can integrate with monitoring or logging platforms like Prometheus or Logstash, enabling scalable and customizable scraper configurations for projects ranging from machine learning data collection to search engine development.

Advanced Scrapy customization tools

Scrapy’s architecture consists of:

Signals: Scrapy utilizes a signal system that allows developers to customize responses to specific events, such as the start or end of data collection, errors, or spider terminations. For instance, a signal can trigger an email report upon completion of a crawl.
Extensions: users can develop custom extensions to introduce unique functionalities to Scrapy. Examples include extensions for uploading data directly to Amazon S3 or Google Cloud.
Middleware: Scrapy’s middleware layer offers the ability to alter requests and responses during processing. This feature is particularly useful for adding functionalities like CAPTCHA handling or session management without modifying the core engine.
Integration with other libraries: Scrapy seamlessly integrates with popular Python libraries. For example, it can work alongside Pandas for data analysis and filtering or Elasticsearch for efficient data storage and retrieval.
Custom selectors: for websites where standard CSS or XPath selectors prove inadequate due to poor structure, Scrapy allows the creation of custom selectors to more precisely extract data.
Custom pipelines: users can construct custom data pipelines to perform tasks such as data cleaning, normalization, or validation. These pipelines facilitate processes like auto-saving data in JSON format or direct loading into databases like MongoDB or PostgreSQL.

Furthermore, Scrapy’s ability to support custom modules for API interactions provides a robust framework for scaling and tailoring solutions to meet the demands of large-scale data processing and complex project requirements.

Portability

Another significant advantage of Scrapy is its portability. The framework supports multiple operating systems, including Windows, macOS, and Linux, making it versatile for use across various development environments. Installation is straightforward using the Python package manager (pip), and thanks to Scrapy's modular structure and flexible configuration, projects can easily be transferred between machines without significant changes.

Furthermore, Scrapy supports virtual environments, which isolate project dependencies and avoid conflicts with other installed packages. This feature is particularly valuable when working on multiple projects simultaneously or when deploying applications to a server, ensuring a clean and stable development environment.

Scrapy Client Interface

For more effective work with Scrapy, it is recommended to use a code editor like Visual Studio Code (VS Code) or its analogs, since interaction with the framework is carried out via the command line (CLI). This allows for managing projects, scanning sites, and configuring spiders more efficiently. Additionally, using virtual environments to manage dependencies can help avoid conflicts between libraries and package versions, ensuring a smoother workflow.

Creating and running a project in Scrapy involves a series of straightforward steps:

First, ensure that Scrapy is installed by running the following command in your terminal:\n
```
pip install scrapy
```
\n
Once Scrapy is installed, create a new directory for your project and navigate into it. Initialize your new project with the command below. For instance, if you're naming your project “myproject”.\n
```
scrapy startproject myproject
```
\n

This command sets up a standard directory structure for your project as follows.\n

myproject/\n       scrapy.cfg            # Project settings\n       myproject/\n               __init__.py\n               items.py          # Data model definitions\n               middlewares.py    # Middlewares\n               pipelines.py      # Data processing\n               settings.py       # Scrapy settings\n               spiders/          # Spiders folder\n                    __init__.py\n

Next, to create a spider, navigate to the “spiders” folder within your project directory and create a new file, for example, “quotes_spider.py”. Populate it with the following basic spider code to scrape quotes:\n

import scrapy\n\nclass QuotesSpider(scrapy.Spider):\n    name = \"quotes\"\n    start_urls = ['http://quotes.toscrape.com/']\n\n    def parse(self, response):\n        for quote in response.css('div.quote'):\n            yield {\n                'text': quote.css('span.text::text').get(),\n                'author': quote.css('span small::text').get(),\n            }\n

To start the spider and begin crawling, use the command:\n
```
scrapy crawl quotes
```
\n
Here, “quotes” is the name of the spider defined in the QuotesSpider class. Scrapy will execute the spider to crawl the specified URL and extract data according to your defined settings.
\n
If you wish to save the scraped data, for instance in a JSON file, you can do so by adding the -o parameter to specify the output file like this:\n
```
scrapy crawl quotes -o quotes.json
```
\n

Scrapy is a robust, free web scraping framework designed to give developers comprehensive tools for automated data extraction and processing from web pages. Its asynchronous architecture and modular structure ensure high speed and excellent scalability, facilitating the expansion of functionality as needed. Additionally, Scrapy's seamless integration with various libraries and data storage solutions, along with support for custom protocols, simplifies the customization of the crawler to meet specific project requirements. This makes the web scraping process not only more efficient but also more adaptable and user-friendly.

Pricing and Licensing of Scrapy

Scrapy is 100% open source and free to use. It operates under the BSD license, allowing you to use, modify, and distribute it without any cost. There are no hidden fees or subscriptions attached to Scrapy itself.

However, Scrapy does not list any pricing for advanced features or enterprise support within its core offerings. If you need professional support, additional tools, or cloud services, you should consider contacting Scrapy or Zyte directly.

Zyte, the company behind Scrapy, offers paid plans and subscription services through Scrapy Cloud (formerly Scrapinghub). The platform provides benefits like:

cloud deployment;
automated scheduling;
a scalable scraping infrastructure designed to handle large projects efficiently.

Here’s what to keep in mind:

Scrapy is free to start, perfect for hands-on developers who want complete control.
For cloud-based scraping, easier management, and enterprise-grade support, explore Zyte’s paid offerings.
Contacting Zyte will help you understand pricing and plans tailored to your project scale and needs.

Knowing this helps you decide if you want to build your Scrapy projects solely in your local environment or leverage Zyte’s cloud tools for easier scaling and management.

What Are the Main Competitors, and How Does Scrapy Compare?

When choosing a web scraping tool, it's essential to understand how Scrapy compares to alternatives. The Best Web Scrapers directory highlights various scraping frameworks, but you need to assess their strengths based on your task.

Popular Alternatives and When to Choose Them

Here’s a quick breakdown of popular competitors and when to pick them:

Beautiful Soup: Best for smaller, simple scraping jobs. It’s easy to learn and ideal for quick HTML parsing in Python, but it’s slower and less scalable than Scrapy. Use it for simple data extraction or prototyping.
Selenium: This is for scraping JavaScript-heavy websites where you need full browser rendering. It’s slower because it controls the actual browser, but it’s necessary when pages rely on dynamic content.
Puppeteer: Built on Node.js, Puppeteer uses headless Chrome for automation. It’s great for complex interactions and scraping Single Page Applications (SPAs). If you prefer JavaScript over Python, this might be the tool for you.
Portia: Created by Zyte, Portia is a visual scraper that requires minimal coding. It’s perfect if you want to create spiders by annotating web pages rather than writing Python code. It’s more user-friendly but less flexible than Scrapy.
Commercial platforms from Zyte: These provide enterprise-level scaling, proxy management, and scheduling. There are services that offer managed services that make large-scale scraping easier but come with added costs.

Scrapy's Strengths and Limitations

Scrapy shines in several areas:

It is fast and scalable thanks to asynchronous processing.
It gives you full control over scraping logic and workflows.
Being open source means it has a transparent development process and an active community ready to help.

But Scrapy also has limitations:

It requires programming knowledge, especially Python and web scraping concepts like XPath and CSS selectors.
It does not natively manage CAPTCHAs, demanding integration with third-party tools.
Proxy handling is external; you must set up proxy services yourself for IP rotation and anonymity.

Understanding these differences helps you pick the scraper that fits your project scope and skill level. You can also explore additional information about web scraping vs web crawling to get a clearer picture of how each approach works.

What Are the Pros and Cons of Using Scrapy?

While this article does not cover explicit pros and cons, here are typical advantages and drawbacks that matter when working with Scrapy Python projects.

Pros:

Open source and free to use, backed by a strong community. This means regular updates, security patches, and plenty of tutorials and guides.
Fast and scalable architecture using asynchronous programming, making it suitable for medium- to large-sized data scraping tasks.
Highly extensible: you can customize middlewares, pipelines, and extensions to tailor data processing and scraping workflows exactly how you want.

Cons:

Requires Python skills and a grasp of asynchronous programming, CSS selectors, and XPath. Beginners may find this challenging.
Scrapy doesn’t handle CAPTCHA challenges out of the box. You’ll need to integrate third-party solution services or browser automation to bypass them.
Proxy management is not built-in. You must rely on external proxy providers for IP rotation, which adds complexity, cost, and maintenance.

Proxy-Seller: A Recommended External Solution

One excellent proxy service to complement the Scrapy proxy setup is Proxy-Seller. It offers a broad range of proxy options, including residential, ISP, data center, and mobile proxies. These proxies support SOCKS5 and HTTP(S) with authentication via username/password or IP whitelisting.

Key practical features of Proxy-Seller for Scrapy users include:

Performance: Fast proxy speeds up to 1 Gbps with 99% uptime reliability.
Coverage: Extensive global IP coverage for GEO-targeting in over 220 countries.
Management: User-friendly dashboard with auto-renewal, easy management, and API access for automation.
Support: 24/7 customer support with proxy replacement or refund within 24 hours if issues occur.

Integrating Proxy-Seller with your Scrapy projects helps you avoid IP bans, bypass geo-restrictions, and overcome rate limits, improving your scraping’s reliability and scale.

Proxy-Seller sources IPs ethically, following official agreements to provide clean, non-blacklisted proxies compliant with GDPR and CCPA regulations. Their competitive pricing includes bulk and long-term rental discounts, plus flexible packages designed specifically for web scraping needs such as Scrapy proxy integration.

Using Proxy-Seller alongside Scrapy streamlines proxy management, reduces operational overhead, and enhances overall scraping performance. This makes it a practical solution for beginners and experienced developers aiming to build robust, large-scale Scrapy projects.

","createdDate":"2025-04-16T16:22:48.000+00:00","description":"Scrapy is a robust, high-level framework designed for web scraping and data extraction, making it ideal for tasks like data parsing, price monitoring, user behavior analysis, social media insights, and SEO analysis.","countOfViews":10,"attachments":[{"id":"6a2497e31079162e2e8b350b","url":"https://proxy-seller.com/files/fs/image/6a2497e31079162e2e8b350b_2a6fibiinprfpmsuofizlue09bvvg6pp.png","name":"2a6fibiinprfpmsuofizlue09bvvg6pp.png","size":56831,"type":"image/png"},{"id":"6a2497e31079162e2e8b350c","url":"https://proxy-seller.com/files/fs/image/6a2497e31079162e2e8b350c_j70totefe1ed7rutl8qji1tleqm09yvu.png","name":"j70totefe1ed7rutl8qji1tleqm09yvu.png","size":186837,"type":"image/png"}],"mainImg":{"id":"6a2497e31079162e2e8b3509","url":"https://proxy-seller.com/files/fs/image/6a2497e31079162e2e8b3509_overview-of-the-scrapy-web-scraping-framework_mainImg.png","name":"overview-of-the-scrapy-web-scraping-framework_mainImg.png","size":39471,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"overview-of-the-scrapy-web-scraping-framework_img","metaTitle":"Scrapy: A Complete Overview of the Web Scraping Framework","metaDescription":"What Scrapy is and how it works — a framework for web data collection. Functionality, advantages, and features of Scrapy. How to run a “spider” in Scrapy for price analysis, review collection, or creating a proxy aggregator."},{"id":"6a049ced35baad34714afe42","tag":"how-to-scrape-quora-questions-and-answers-using-python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":6,"title":"How to Scrape Quora Questions and Answers Using Python","body":"

This article focuses on how to extract Quora data and why it could be useful, and how Python can be used to automate this task. We'll also look into the main tools and libraries needed to scrape from this site.

Why Scrape Quora Data?

Quora is one of the most popular question-answering websites that people from around the world use. It is especially prominent in the USA, UK, Canada, Australia and other English speaking countries. There are numerous benefits when analyzing the Quora data, such as:

Providing up-to-date information on relevant topics, expert analysis and trending questions across various subjects.
Interacting successfully with target audiences by addressing their inquiries and leveraging the trust for a business or personal expert reputation.
Gaining quality answers, sharing of experiences and information without performing holistic extensive searches.

The platform can be used in research, content development or building AI solutions.

Understanding Quora Web Scraping

Web scraping refers to the act of extracting information and data from a website or a web page. It is a modern way of gathering a lot of information and converting it to an organized structure like a CSV file.

What is web scraping Quora platform give us:

the question itself.
links corresponding to some quora pages.
the number of answers and questions so far.
the people who authored the answers.
the dates they were published.

Thus, Quora data scraping gives insight on user participation, significance, and the popularity of different questions as well as the answers they receive.

Essential Tools for Scraping Quora

In order to scrape Quora data Python libraries are used. Below are the most relevant resources that will enable you to achieve that goal:

BeautifulSoup — for parsing the HTML pages and collecting the sought information.
Requests — for formulating Hypertext Transfer Protocol requests and getting the web pages.
Selenium — for browser automation, as well as for the capturing of dynamically-generated content.

These libraries make it possible to interact with web pages and collect information from Quora effortlessly.

Building a Quora Scraping Python Script

This subsection addresses the construction of a scraper, which will be a stepwise attention. Prepare the workspace, install requisite libraries, create procedures for sending requests, for parsing the html code, and for working with the retrieved information.

Step 1: Setting Up Your Python Project Environment

In advance to scraping one must prepare the working environment:

Download and install Python from the official page.

Set up a virtual environment to compartmentalize your libraries in use.\n

\npython -m venv quora_scraper\nsource quora_scraper/bin/activate  # For MacOS and Linux\nquora_scraper\\Scripts\\activate     # For Windows\n\n

Step 2: Installing Required Python Libraries

Install the necessary libraries that you will need for web scraping in this step:

\npip install requests beautifulsoup4\n\n

These interfaces are essential for transmitting HTTP requests and subsequently analyzing the received HTML data.

Step 3: Sending Requests to Quora

To call a certain API and get the HTML page back, we will make use of the Requests library. It is intuitive and gets the job done very fast.

\nurl = \"https://www.quora.com/How-do-you-open-your-own-company\"\nresponse = requests.get(url)\n\n

Step 4: Parsing the HTML Response

In order to use Python, to scrape Quora answers we will need another library called BeautifulSoup. This is how you install it:

\nsoup = BeautifulSoup(response.text, \"lxml\")\nquestion = soup.find(class_='q-box qu-userSelect--text')\n\n

Step 5: Extracting Quora Questions and Answers

To extract the questions and answers, you only need to find the required HTML tag which has these details. Let us take a case:

\ntext_of_answers = []\nfor answer_id in range(6):\n   answers = soup.find_all(\n       class_=f'q-box dom_annotate_question_answer_item_{answer_id} qu-borderAll qu-borderColor--raised qu-boxShadow--small qu-mb--small qu-bg--raised')\n\n   for answer in answers:\n       text = answer.find('p', class_='q-text qu-display--block qu-wordBreak--break-word qu-textAlign--start')\n       text_of_answers.append(text)\n\n

Step 6: Handling Pagination for Multiple Pages

Quora tends to have a lot of answers and often you need to scroll the page a lot in order to see everything. For this we will be using Selenium to automate the website.

Below is how you set up Selenium and use proxies to hide your identity better:

\nfrom selenium.webdriver.chrome.options import Options\nfrom seleniumwire import webdriver as wiredriver\n\nchrome_options = Options()\nchrome_options.add_argument(f'--proxy-server={proxy_address}')\nchrome_options.add_argument(f'--proxy-auth={proxy_username}:{proxy_password}')\n\nchrome_options.add_argument(\"--headless\")\n\ndriver = wiredriver.Chrome(options=chrome_options)\n\n

After you have set everything up in Selenium, you just need to make an HTTP request, and then keep scrolling down and capturing these data pieces:

\nurl = \"https://www.quora.com/How-do-you-open-your-own-company\"\ndriver.execute_script('window.scrollBy(0, 1000)')\n\ndriver.get(url)\n\n

Step 7: Storing the Scraped Data Efficiently

When you are done collecting the data, you want to keep them in a way that makes it easy to analyze later on. The best way to do this is to keep them in JSON format, as it is the most widely used.

\nwith open(f'quora_data.json', 'w', encoding='utf-8') as json_file:\n   json.dump(text_of_answers, json_file, ensure_ascii=False, indent=4)\n\n

Step 8: Exporting Data to CSV or JSON Format

We saved data to JSON, but in some cases it might be desired to do it in more than one format at once. So here you have a function that does exactly that:

\ndef export_data(data, csv_filename=\"quora_data.csv\", json_filename=\"quora_data.json\"):\n    save_to_csv(data, csv_filename)\n    save_to_json(data, json_filename)\n\nexport_data(text_of_answers)\n\n

Step 9: Handling Rate Limits and Blocking Issues

Quora has some defenses against automated data scraping, as such trying to send a lot of requests to Quora will result in cutting off your IP address. There are ways around this though.

Adding some time before sending the next request.\n
This is basic mimicry of human behavior that adds some form of lag when a request is sent.
\n
```
\nimport time\nimport random\n\ndef random_delay():\n    time.sleep(random.uniform(2, 5))\n\n
```
\n
Add this function after every query to increase the chances of not being blocked.
\n

Use proxy servers.\n

One can easily avoid blocking the IP address by using proxy servers. Let’s see how it can be used with Requests and Selenium.

requests:\n

\nurl = \"https://www.quora.com/How-do-you-open-your-own-company\" \n\n\n\nproxy = 'LOGIN:PASSWORD@ADDRESS:PORT'\nproxies = {\n   \"http\": f\"http://{proxy}\",\n   \"https\": f\"https://{proxy}\",\n}\n\nresponse = requests.get(url, proxies=proxies)\n\n

Selenium:\n

\nproxy_address = \"\"\nproxy_username = \"\"\nproxy_password = \"\"\n\nchrome_options = Options()\nchrome_options.add_argument(f'--proxy-server={proxy_address}')\nchrome_options.add_argument(f'--proxy-auth={proxy_username}:{proxy_password}')\n\nchrome_options.add_argument(\"--headless\")\n\ndriver = wiredriver.Chrome(options=chrome_options)\n\n

Step 10: Putting Everything Together

So after you have covered all steps, it is time to combine them and put them into one script.

\nimport json\nfrom selenium.webdriver.chrome.options import Options\nfrom seleniumwire import webdriver as wiredriver\nfrom bs4 import BeautifulSoup\n\nproxy_address = \"\"\nproxy_username = \"\"\nproxy_password = \"\"\n\nchrome_options = Options()\nchrome_options.add_argument(f'--proxy-server={proxy_address}')\nchrome_options.add_argument(f'--proxy-auth={proxy_username}:{proxy_password}')\n\nchrome_options.add_argument(\"--headless\")\n\ndriver = wiredriver.Chrome(options=chrome_options)\n\nurl = \"https://www.quora.com/How-do-you-open-your-own-company\"\ndriver.execute_script('window.scrollBy(0, 1000)')\n\ndriver.get(url)\n\nhtml = driver.page_source\n\nsoup = BeautifulSoup(html, \"lxml\")\n\nquestion = soup.find(class_='q-box qu-userSelect--text')\n\n# Answers\ntext_of_answers = [{'question': question}]\nfor answer_id in range(6):\n   answers = soup.find_all(\n       class_=f'q-box dom_annotate_question_answer_item_{answer_id} qu-borderAll qu-borderColor--raised qu-boxShadow--small qu-mb--small qu-bg--raised')\n\n   for answer in answers:\n       text = answer.find('p', class_='q-text qu-display--block qu-wordBreak--break-word qu-textAlign--start').text\n       text_of_answers.append({\n           'answers': text,\n       })\n\nwith open(f'Quora_data.json', 'w', encoding='utf-8') as json_file:\n   json.dump(text_of_answers, json_file, ensure_ascii=False, indent=4)\n   print(f\"Quora_data.json\")\n\n

Conclusion

In this article we have discussed the methods of Quora scraping with Python. Such scripts constructed enables users to override some restrictions along with pagination and data saving in various formats such as JSON and CSV.

For large-scale data collection, the most effective method would be using Quora’s API. On the other hand, if you strategically choose to utilize BeautifulSoup or Selenium, then using proxy servers is a must for sustained performance.

","createdDate":"2025-03-28T21:15:13.000+00:00","description":"his article focuses on how to extract Quora data and why it could be useful, and how Python can be used to automate this task.","countOfViews":7,"attachments":[],"mainImg":{"id":"6a2498d21079162e2e8b5d41","url":"https://proxy-seller.com/files/fs/image/6a2498d21079162e2e8b5d41_how-to-scrape-quora-questions-and-answers-using-python_mainImg.png","name":"how-to-scrape-quora-questions-and-answers-using-python_mainImg.png","size":42173,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"how-to-scrape-quora-questions-and-answers-using-python_img","metaTitle":"Scraping Data from Quora Using Python | Proxy Seller","metaDescription":"Learn how to scrape questions and answers from Quora. Discover methods to efficiently extract valuable data from Quora for your projects."},{"id":"6a049cee35baad34714afe50","tag":"using-a-proxy-with-python-requests","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":5,"title":"Using a Proxy with Python Requests","body":"

In a nutshell, proxies are servers that serve as an outer layer connecting to network services. They serve as the middle-men for a computer and the site from which a user wants to extract information from. The user's request, instead of being directly sent to the relevant site, is first directed to the proxy which then sends it to the end server. Hence, the proxy server's IP address is the one that the site saves instead of the user's.

So, why use a proxy?

Anonymity:

For instance, if there is a need to gather information from a site without the fear of restrictions or blocks. A proxy helps in masking the real user’s IP address changing their true digital identity.

Bypassing restrictions:

A proxy from a specific country where there are no restrictions can be used. This is because, in certain countries, access to some programs and websites is limited or blocked.

Load balancing:

If a large number of requests are sent to a particular site, chances are the system will be loaded. Consequently, there is a huge risk of being blocked. Using multiple proxies at the same time allows users to evenly distribute the requests to the particular site, hence avoiding blockage.

Setting Up and Installing Python Requests

Requests is the library for sending HTTP requests using Python. It makes it simple to do GET and POST requests. Using Python Requests with no proxy can send HTTP requests, but it does not provide anonymity or bypass restrictions.

1. Installing the Requests Library

To install requests, simply enter the command below in the terminal:

\npip install requests\n\n

2. Verifying the Installation

To verify that the required library is properly set up, you need to open the Python development environment and run the command:

\nimport requests \n\nprint(requests.__version__)\n\n

If everything was done properly, the script will return the version number.

How to Use Proxy With Requests Python

Once we install the requests library, we can start executing HTTP requests. However, to incorporate a proxy, certain configurations in Python must be done.

Let us examine the steps to follow to configure a proxy correctly for requests in Python. We'll consider Python Requests with SOCKS proxies configurations separately.

1. Configuring the Proxy Server

Using Python to set up a proxy is very easy for users using the requests library. It requires that the proxy server address be passed in a dictionary and then used when making HTTP requests.

\nproxies = {\n    \"http\": \"http://your-proxy-ip:port\",\n    \"https\": \"http://your-proxy-ip:port\",\n}\n\nresponse = requests.get(\"http://example.com\", proxies=proxies)\n\n

2. Proxy Authentication

When using Python Requests, proxy authentication is easy to set up. Let's take a closer look at the details.

\nproxies = { \n    \"http\": \"http://username:password@your-proxy-ip:port\", \n    \"https\": \"http://username:password@your-proxy-ip:port\", \n} \n\nresponse = requests.get(\"http://example.com\", proxies=proxies)\n\n

The two new fields that need to be filled in are:

username – proxy username;
password – proxy password.

3. Using SOCKS Proxies

For users who require a higher level of anonymity, or if you are working with heavily restricted sites, standard HTTP proxies might not work. In this case, SOCKS proxies may be better.

To enable SOCKS proxy support, an additional library needs to be installed using the command listed below:

\npip install requests[socks]\n\n

Once it has been installed, Python Requests allow you to set up a SOCKS proxy that can be used as shown in the example below.

\nimport requests\n\nproxies = {\n    \"http\": \"socks5h://your-proxy-ip:port\",\n    \"https\": \"socks5h://your-proxy-ip:port\",\n}\n\nresponse = requests.get(\"http://example.com\", proxies=proxies)\n\n

If the proxy server needs authentication, include them as shown below.

\nproxies = {\n    \"http\": \"socks5h://username:password@your-proxy-ip:port\",\n    \"https\": \"socks5h://username:password@your-proxy-ip:port\",\n}\n\n

Managing Sessions with Proxies in Python Requests

When making many requests through a single proxy, instead of adding it to each request, it becomes much more efficient to use Python Requests sessions with proxy.

This method of using proxies helps in retaining settings across all sessions. It also simplifies the code, improves performance, and most importantly, makes the implementation much easier.

1. Using Python Requests Sessions with Proxies

It is very easy to create a session, just type in requests.Session().

Consider the following example:

\nimport requests\n\n# Creating a session\nsession = requests.Session()\n\nsession.proxies = {\n    \"http\": \"http://username:password@your-proxy-ip:port\",\n    \"https\": \"http://username:password@your-proxy-ip:port\",\n}\n\n# Request through the session\nresponse = session.get(\"http://example.com\")\n\n

All of the Python requests within this session will use the set proxy by default without any additional configurations.

2. Handling Multiple Proxies in Python Requests

When you are actively engaging with a website either for scraping purposes or for automation, using the same proxy over and over again can get your account blocked. This is manageable by rotating proxies given you have a few different proxy servers.

The following example shows the implementation of a loop that rotates proxies for each interaction with the web page:

\nimport requests\nimport random\n\nproxies_list = [\n    \"http://username:password@your-proxy-ip[1]:port\",\n    \"http://username:password@your-proxy-ip[2]:port\",\n    \"http://username:password@your-proxy-ip[3]:port\",\n]\n\nsession = requests.Session()\n\nfor _ in range(5):\n    proxy = random.choice(proxies_list)  # Randomly choose a proxy\n    session.proxies = {\"http\": proxy, \"https\": proxy}\n\n    response = session.get(\"http://example.com\")\n    print(f\"Used proxy: {proxy}\")\n    print(response.status_code)\n\n

Conclusion

To sum up, Here are some suggestions that might help you manage proxies better:

Use sessions so you don’t have to set a proxy with every request.
Use anonymous proxies without logs for better protection.
To prevent getting blocked, use multiple proxies.

In this article, we’ve touched on what proxy servers are, how to use proxy in Python Requests properly, and manage proxies through sessions to simplify the code. The example also shows usage of both SOCKS and regular proxies, how authentication is handled, and proxy management.

","createdDate":"2025-03-24T13:39:12.000+00:00","description":"In a nutshell, proxies are servers that serve as an outer layer connecting to network services.","countOfViews":7,"attachments":[],"mainImg":{"id":"6a2498e61079162e2e8b601a","url":"https://proxy-seller.com/files/fs/image/6a2498e61079162e2e8b601a_using-a-proxy-with-python-requests_mainImg.png","name":"using-a-proxy-with-python-requests_mainImg.png","size":39647,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"using-a-proxy-with-python-requests_img","metaTitle":"Using a Proxy with Python Requests | Proxy Seller","metaDescription":"Explore how to effectively use a proxy with the Requests package in this tutorial, and gain insights on how to choose the most suitable proxy provider."},{"id":"6a049cf935baad34714afedf","tag":"how-to-scrape-booking-com-data-using-python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":6,"title":"How to scrape Booking.com data using Python","body":"

In this article, we will demonstrate how to collect data from the Booking.com website with Python. We will obtain information including, but not limited to, hotel names, ratings, prices, location addresses and their descriptions. The code provided allows you to retrieve data from hotel pages by parsing HTML content and extracting embedded JSON data.

Installing required libraries

Before running the code to scrape data from Booking.com, you'll need to install the necessary Python libraries. Here’s how you can install the required dependencies:

Requests Library: This is used to send HTTP requests to the website and fetch the HTML content of the pages.
LXML Library: This is used for parsing the HTML content and extracting data using XPath.
JSON: Built-in Python module used to handle JSON data.
CSV: Built-in Python module used for writing scraped data into a CSV file.

To install the necessary libraries, you can use pip:

\npip install requests lxml\n\n

These are the only external libraries you need, and the rest (json, csv) come pre-installed with Python.

Understanding the URL and data structure

When scraping data from Booking.com, it's important to understand the structure of the webpage and the kind of data you want to extract. Each hotel page on Booking.com contains embedded structured data in the form of JSON-LD, a format that allows easy extraction of details like name, location, and pricing. We'll be scraping this data.

Step-by-Step Scraping Process

As Booking.com is a dynamic site and implements measures to combat automated actions, we will use appropriate HTTP headers and proxies to ensure seamless scraping without the risk of blocking.

Sending HTTP Requests with Headers

Headers mimic a user session in a browser and prevent detection by Booking.com's anti-scraping systems. Without properly configured headers, the server can easily identify automated scripts, which may lead to IP blocking or captcha challenges.

To avoid being blocked by Booking.com’s anti-scraping mechanisms, we will use custom headers to simulate a legitimate user browsing the website. Here’s how you can send an HTTP request with proper headers:

\nimport requests\nfrom lxml.html import fromstring\n\nurls_list = [\"https links\"]\n\nfor url in urls_list:\n    headers = {\n        'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n        'accept-language': 'en-IN,en;q=0.9',\n        'cache-control': 'no-cache',\n        'dnt': '1',\n        'pragma': 'no-cache',\n        'priority': 'u=0, i',\n        'sec-ch-ua': '\"Chromium\";v=\"130\", \"Google Chrome\";v=\"130\", \"Not?A_Brand\";v=\"99\"',\n        'sec-ch-ua-mobile': '?0',\n        'sec-ch-ua-platform': '\"Linux\"',\n        'sec-fetch-dest': 'document',\n        'sec-fetch-mode': 'navigate',\n        'sec-fetch-site': 'none',\n        'sec-fetch-user': '?1',\n        'upgrade-insecure-requests': '1',\n        'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36',\n    }\n\n    response = requests.get(url, headers=headers)\n\n

Importance of proxies

Using proxies is necessary when scraping sites like Booking.com, which apply strict request rate limits or track IP addresses. Proxies help distribute the load of requests across different IP addresses, preventing blocks. For this purpose, both free proxies and paid proxy services with authentication by username and password or IP address can be used. In our example, we use the latter option.

\nproxies = {\n    'http': '',\n    'https': ''\n}\nresponse = requests.get(url, headers=headers, proxies=proxies)\n\n

Parsing the HTML and extracting JSON data

After sending the request, we parse the HTML content using lxml to locate the embedded JSON-LD data that contains hotel details. This step extracts the structured data from the web page that includes hotel names, prices, locations, and more.

\nparser = fromstring(response.text)\n\n# Extract embedded JSON data\nembeded_jsons = parser.xpath('//script[@type=\"application/ld+json\"]/text()')\njson_data = json.loads(embeded_jsons[0])\n\n

Extracting hotel information

Once we have the parsed JSON data, we can extract relevant fields such as hotel name, address, rating, and pricing. Below is the code to extract hotel information from the JSON:

\nname = json_data['name']\nlocation = json_data['hasMap']\npriceRange = json_data['priceRange']\ndescription = json_data['description']\nurl = json_data['url']\nratingValue = json_data['aggregateRating']['ratingValue']\nreviewCount = json_data['aggregateRating']['reviewCount']\ntype_ = json_data['@type']\npostalCode = json_data['address']['postalCode']\naddressLocality = json_data['address']['addressLocality']\naddressCountry = json_data['address']['addressCountry']\naddressRegion = json_data['address']['addressRegion']\nstreetAddress = json_data['address']['streetAddress']\nimage_url = json_data['image']\nroom_types = parser.xpath(\"//a[contains(@href, '#RD')]/span/text()\")\n\n# Append the data to the all_data list\nall_data.append({\n    \"Name\": name,\n    \"Location\": location,\n    \"Price Range\": priceRange,\n    \"Rating\": ratingValue,\n    \"Review Count\": reviewCount,\n    \"Type\": type_,\n    \"Postal Code\": postalCode,\n    \"Address Locality\": addressLocality,\n    \"Country\": addressCountry,\n    \"Region\": addressRegion,\n    \"Street Address\": streetAddress,\n    \"URL\": url,\n    \"Image URL\": image_url,\n    \"Room Types\": room_types\n})\n\n

Saving the data to CSV

Once the data has been extracted, we can save it into a CSV file for further analysis:

\n# After all URLs are processed, write the data into a CSV file\nwith open('booking_data.csv', 'w', newline='') as csvfile:\n    fieldnames = [\"Name\", \"Location\", \"Price Range\", \"Rating\", \"Review Count\", \"Type\", \"Postal Code\", \n                  \"Address Locality\", \"Country\", \"Region\", \"Street Address\", \"URL\", \"Image URL\", \"Room Types\"]\n    \n    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n    \n    # Write the header\n    writer.writeheader()\n    \n    # Write the rows of data\n    writer.writerows(all_data)\n\n

Complete code

Here’s the complete code combining all the sections:

\nimport requests\nfrom lxml.html import fromstring\nimport json\nimport csv\n\n# List of hotel URLs to scrape\nurls_list = [\n    \"Https link\", \n    \"Https link\"\n]\n\n# Initialize an empty list to hold all the scraped data\nall_data = []\n\nproxies = {\n    'http': ''\n}\n\n# Loop through each URL to scrape data\nfor url in urls_list:\n    headers = {\n        'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n        'accept-language': 'en-IN,en;q=0.9',\n        'cache-control': 'no-cache',\n        'dnt': '1',\n        'pragma': 'no-cache',\n        'priority': 'u=0, i',\n        'sec-ch-ua': '\"Chromium\";v=\"130\", \"Google Chrome\";v=\"130\", \"Not?A_Brand\";v=\"99\"',\n        'sec-ch-ua-mobile': '?0',\n        'sec-ch-ua-platform': '\"Linux\"',\n        'sec-fetch-dest': 'document',\n        'sec-fetch-mode': 'navigate',\n        'sec-fetch-site': 'none',\n        'sec-fetch-user': '?1',\n        'upgrade-insecure-requests': '1',\n        'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36',\n    }\n\n    # Sending the request to the website\n    response = requests.get(url, headers=headers, proxies=proxies)\n    \n    # Parsing the HTML content\n    parser = fromstring(response.text)\n    \n    # Extract embedded JSON data\n    embeded_jsons = parser.xpath('//script[@type=\"application/ld+json\"]/text()')\n    json_data = json.loads(embeded_jsons[0])\n\n    # Extract all hotel details from JSON\n    name = json_data['name']\n    location = json_data['hasMap']\n    priceRange = json_data['priceRange']\n    description = json_data['description']\n    url = json_data['url']\n    ratingValue = json_data['aggregateRating']['ratingValue']\n    reviewCount = json_data['aggregateRating']['reviewCount']\n    type_ = json_data['@type']\n    postalCode = json_data['address']['postalCode']\n    addressLocality = json_data['address']['addressLocality']\n    addressCountry = json_data['address']['addressCountry']\n    addressRegion = json_data['address']['addressRegion']\n    streetAddress = json_data['address']['streetAddress']\n    image_url = json_data['image']\n\n    room_types = parser.xpath(\"//a[contains(@href, '#RD')]/span/text()\")\n    \n    # Append the data to the all_data list\n    all_data.append({\n        \"Name\": name,\n        \"Location\": location,\n        \"Price Range\": priceRange,\n        \"Rating\": ratingValue,\n        \"Review Count\": reviewCount,\n        \"Type\": type_,\n        \"Postal Code\": postalCode,\n        \"Address Locality\": addressLocality,\n        \"Country\": addressCountry,\n        \"Region\": addressRegion,\n        \"Street Address\": streetAddress,\n        \"URL\": url,\n        \"Image URL\": image_url,\n        \"Room Types\": room_types\n    })\n\n# After all URLs are processed, write the data into a CSV file\nwith open('booking_data.csv', 'w', newline='') as csvfile:\n    fieldnames = [\"Name\", \"Location\", \"Price Range\", \"Rating\", \"Review Count\", \"Type\", \"Postal Code\", \n                  \"Address Locality\", \"Country\", \"Region\", \"Street Address\", \"URL\", \"Image URL\", \"Room Types\"]\n    \n    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n    \n    # Write the header\n    writer.writeheader()\n    \n    # Write the rows of data\n    writer.writerows(all_data)\n\nprint(\"Data successfully saved to booking_data.csv\")\n\n

This article showed how to scrape hotel data from Booking.com using Python. We emphasized the importance of using appropriate HTTP headers and proxies to bypass anti-scraping measures. The extracted data can be saved in a CSV file for further analysis. When scraping websites, always check the terms of service to avoid violating.

","createdDate":"2025-03-14T10:04:16.000+00:00","description":"In this article, we will demonstrate how to collect data from the Booking.com website with Python.","countOfViews":7,"attachments":[],"mainImg":{"id":"6a2499361079162e2e8b70b9","url":"https://proxy-seller.com/files/fs/image/6a2499361079162e2e8b70b9_how-to-scrape-booking-com-data-using-python_mainImg.png","name":"how-to-scrape-booking-com-data-using-python_mainImg.png","size":45946,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"how-to-scrape-booking-com-data-using-python_img","metaTitle":"Guide to Scraping Data from Booking.com Using Python","metaDescription":"Learn how to scrape Booking.com to extract hotel data. Use a Python script that retrieves data by circumventing blocks and saves it to a CSV file."},{"id":"6a049d0235baad34714aff8c","tag":"how-to-scrape-instagram-data-using-python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":11,"title":"How to Scrape Instagram Data Using Python","body":"

Gaining access to Instagram data can be tricky due to various anti-bot mechanisms, login requirements, and rate limits. However, you can extract useful information from public profiles with the right tools and techniques. This article will guide you through how to scrape Instagram user data using Python by making API requests to Instagram’s backend, extracting information from the returned JSON data, and saving it into a JSON file.

Setting Up the Required Libraries

Before we get into the code, make sure you have installed the required Python libraries.

pip install requests python-box

requests: To make HTTP requests.
python-box: Simplifies data access by turning dictionaries into objects that allow dot notation access.

We'll break the code into different sections for better understanding, including sending the request, obtaining and parsing the data, using proxies to avoid detection, and simplifying JSON parsing with the Box library.

Step 1. Making the API request

The frontend of Instagram is heavily secured, but the backend offers API endpoints that can be used without authentication. We will use one of these points going forward.

This API provides detailed information about a user's profile, including their description, follower count, and posts. Let's explore how to request data using the requests library in Python (read more about the best Python libraries for web scraping).

Explanation:

Headers: Instagram blocks most bot requests by analyzing request headers. The x-ig-app-id is essential because it mimics a request coming from the Instagram app itself.
The User-Agent string represents the browser making the request, tricking Instagram into believing it's a real user.
Backend API Request: The URL \n
```
https://i.instagram.com/api/v1/users/web_profile_info/?username={username}
```
\nis part of Instagram’s backend API. It provides detailed information about a public profile.
Handling the JSON Response: We use response.json() to convert the API response into a JSON object that we can easily parse and extract information from.

\nimport requests\n\n# Define headers to mimic a real browser request\nheaders = {\n    \"x-ig-app-id\": \"936619743392459\",  # Instagram app ID to authenticate the request\n    \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36\",\n    \"Accept-Language\": \"en-US,en;q=0.9,ru;q=0.8\",\n    \"Accept-Encoding\": \"gzip, deflate, br\",\n    \"Accept\": \"*/*\",\n}\n\n# Replace this with the username you want to scrape\nusername = 'testtest'\n\n# Send an API request to get profile data\nresponse = requests.get(f'https://i.instagram.com/api/v1/users/web_profile_info/?username={username}', headers=headers)\nresponse_json = response.json()  # Parse the response into a JSON object\n\n

Step 2. Handling Proxies to Bypass Rate-Limiting

Since Instagram restricts repeated requests from the same IP address (click here to learn how to check your IP address), using proxies is a must for large-scale scraping. If you plan to scale requests safely, buy private proxies so each worker/thread can use its own stable credentials and you can reduce collision-based blocks. A proxy routes your requests through different IP addresses, helping you avoid detection. The same approach can be applied to other social platforms - for example, when working with Snapchat data, setting up a proxy for Snapchat helps distribute requests more evenly and keep sessions stable during scraping.

To set up a proxy server, you will need the IP address, port number, and, if required, a username and password for authentication. If your target environment supports IPv6, you can also buy IPv6 address access and plug it into the same proxy string format (address + port + auth).

\nproxies = {\n    'http': 'http://<proxy_username>:<proxy_password>@<proxy_ip>:<proxy_port>',\n    'https': 'https://<proxy_username>:<proxy_password>@<proxy_ip>:<proxy_port>',\n}\n\nresponse = requests.get(f'https://i.instagram.com/api/v1/users/web_profile_info/?username={username}', headers=headers, proxies=proxies)\n

To improve stability and distribute traffic across authenticated endpoints for production use, consider using a dedicated Instagram proxy.

Step 3. Simplifying JSON Parsing with Box

Instagram's API returns a complex nested JSON structure, which can be difficult to navigate using traditional dictionary-based access. To make parsing easier, we can use the Box library, which allows accessing JSON data using dot notation instead of dictionary keys.

Explanation:

Box: This library converts a JSON dictionary into an object, allowing us to access deeply nested fields using dot notation. For instance, instead of writing response_json['data']['user']['full_name'], we can simply write response_json.data.user.full_name.
Extracting Data: We extract useful profile information like the user's full name, ID, biography, whether it’s a business or professional account, verification status, and follower count.

\nfrom box import Box\n\nresponse_json = Box(response.json())\n\n# Extract user profile data\nuser_data = {\n    'full name': response_json.data.user.full_name,\n    'id': response_json.data.user.id,\n    'biography': response_json.data.user.biography,\n    'business account': response_json.data.user.is_business_account,\n    'professional account': response_json.data.user.is_professional_account,\n    'category name': response_json.data.user.category_name,\n    'is verified': response_json.data.user.is_verified,\n    'profile pic url': response_json.data.user.profile_pic_url_hd,\n    'followers': response_json.data.user.edge_followed_by.count,\n    'following': response_json.data.user.edge_follow.count,\n}\n\n

Step 4. Extracting Video and Timeline Data

Once the profile data is extracted, we can also scrape data from the user’s video timeline and regular posts.

Explanation:

Video Data: This section extracts data about the user’s Instagram videos, including the video URL, view count, comment count, and the video's duration.
Timeline Media: Similarly, this section extracts data from the user's timeline, capturing the post's media URL, likes, and comments.

\n# Extract video data\nprofile_video_data = []\nfor element in response_json.data.user.edge_felix_video_timeline.edges:\n    video_data = {\n        'id': element.node.id,\n        'short code': element.node.shortcode,\n        'video url': element.node.video_url,\n        'view count': element.node.video_view_count,\n        'comment count': element.node.edge_media_to_comment.count,\n        'like count': element.node.edge_liked_by.count,\n        'duration': element.node.video_duration,\n    }\n    profile_video_data.append(video_data)\n\n# Extract timeline media data (photos and videos)\nprofile_timeline_media_data = []\nfor element in response_json.data.user.edge_owner_to_timeline_media.edges:\n    media_data = {\n        'id': element.node.id,\n        'short code': element.node.shortcode,\n        'media url': element.node.display_url,\n        'comment count': element.node.edge_media_to_comment.count,\n        'like count': element.node.edge_liked_by.count,\n    }\n    profile_timeline_media_data.append(media_data)\n\n

Step 5. Saving the Data to JSON Files

Once you’ve extracted all the data, the next step is to save it to a JSON file for further analysis or storage. We use Python's json module to write the extracted data to JSON files. Each file will be neatly formatted, thanks to the indent=4 parameter, which makes it easy to read and process the data.

\nimport json\n\n# Save user data to a JSON file\nwith open(f'{username}_profile_data.json', 'w') as file:\n    json.dump(user_data, file, indent=4)\n\n# Save video data to a JSON file\nwith open(f'{username}_video_data.json', 'w') as file:\n    json.dump(profile_video_data, file, indent=4)\n\n# Save timeline media data to a JSON file\nwith open(f'{username}_timeline_media_data.json', 'w') as file:\n    json.dump(profile_timeline_media_data, file, indent=4)\n\n

Complete Code

Here’s the complete Python script that combines all the previously discussed sections. This code scrapes user profile data, video data, and timeline media data from Instagram, handles the necessary headers and proxies, and saves the extracted information to JSON files.

\nimport requests\nfrom box import Box\nimport json\n\n# Headers to mimic a real browser request to Instagram's backend API\nheaders = {\n    \"x-ig-app-id\": \"936619743392459\", \n    \"User-Agent\": \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/62.0.3202.94 Safari/537.36\",\n    \"Accept-Language\": \"en-US,en;q=0.9,ru;q=0.8\",\n    \"Accept-Encoding\": \"gzip, deflate, br\",\n    \"Accept\": \"*/*\",\n}\n\n# Set a proxy to avoid rate-limiting and detection (optional)\nproxies = {\n    'http': 'http://<proxy_username>:<proxy_password>@<proxy_ip>:<proxy_port>',\n    'https': 'https://<proxy_username>:<proxy_password>@<proxy_ip>:<proxy_port>',\n}\n\n# The Instagram username to scrape\nusername = 'testtest'\n\n# Send a request to Instagram's backend API to get profile data\nresponse = requests.get(f'https://i.instagram.com/api/v1/users/web_profile_info/?username={username}', \n                        headers=headers, proxies=proxies)\nresponse_json = Box(response.json())  # Convert the response to Box object for easy navigation\n\n# Extract user profile data\nuser_data = {\n    'full name': response_json.data.user.full_name,\n    'id': response_json.data.user.id,\n    'biography': response_json.data.user.biography,\n    'business account': response_json.data.user.is_business_account,\n    'professional account': response_json.data.user.is_professional_account,\n    'category name': response_json.data.user.category_name,\n    'is verified': response_json.data.user.is_verified,\n    'profile pic url': response_json.data.user.profile_pic_url_hd,\n    'followers': response_json.data.user.edge_followed_by.count,\n    'following': response_json.data.user.edge_follow.count,\n}\n\n# Extract video data from the user's video timeline\nprofile_video_data = []\nfor element in response_json.data.user.edge_felix_video_timeline.edges:\n    video_data = {\n        'id': element.node.id,\n        'short code': element.node.shortcode,\n        'video url': element.node.video_url,\n        'view count': element.node.video_view_count,\n        'comment count': element.node.edge_media_to_comment.count,\n        'like count': element.node.edge_liked_by.count,\n        'duration': element.node.video_duration,\n    }\n    profile_video_data.append(video_data)\n\n# Extract timeline media data (photos and videos)\nprofile_timeline_media_data = []\nfor element in response_json.data.user.edge_owner_to_timeline_media.edges:\n    media_data = {\n        'id': element.node.id,\n        'short code': element.node.shortcode,\n        'media url': element.node.display_url,\n        'comment count': element.node.edge_media_to_comment.count,\n        'like count': element.node.edge_liked_by.count,\n    }\n    profile_timeline_media_data.append(media_data)\n\n# Save user profile data to a JSON file\nwith open(f'{username}_profile_data.json', 'w') as file:\n    json.dump(user_data, file, indent=4)\nprint(f'saved json: {username}_profile_data.json')\n\n# Save video data to a JSON file\nwith open(f'{username}_video_data.json', 'w') as file:\n    json.dump(profile_video_data, file, indent=4)\nprint(f'saved json: {username}_video_data.json')\n\n# Save timeline media data to a JSON file\nwith open(f'{username}_timeline_media_data.json', 'w') as file:\n    json.dump(profile_timeline_media_data, file, indent=4)\nprint(f'saved json: {username}_timeline_media_data.json')\n\n

Why Scraping Instagram is Tricky with the Instagram Scraper Python

Let's outline the reasons Instagram scraper Python may be difficult to use.

Strong Anti-Bot Login Wall

Instagram has put up a strong anti-bot login wall. This wall makes it hard to access data, even from public pages. When you visit Instagram through a scraper, you often hit login prompts that block your requests. You might think a VPN or datacenter IP can fix this, but they usually can't. These IPs are quickly detected and blocked. When Instagram is especially aggressive, a mobile proxy 5g can help because the traffic pattern and IP reputation look closer to real mobile users.

Need for Residential Proxies

Residential proxies offer a better way around these login walls. They use real IPs from regular users, so Instagram sees them as normal visitors. This reduces the chances of being blocked. You need proxies that rotate IPs and locations for the best results. In practice, a residential proxy pool makes rotation smoother, letting you spread requests across many real-user IPs while keeping geos consistent.

Intermittent Login Modals

Another problem is Instagram’s intermittent login modals when you try to access posts directly. These pop-ups appear randomly, interrupting your scraper's flow. This causes your Python Instagram scraper to fail, forcing retries.

Complex API and Maintenance Burden

You could try intercepting Instagram’s hidden API calls to get data. But this is complex and needs constant maintenance. Instagram changes its API frequently and aggressively blocks scrapers. You must stay up to date with the exact API request headers to avoid errors and bans.

Proxy-Seller: The Essential Proxy Engine for Bypassing Instagram Blocks

Using residential proxies along with updated headers is key. Proxy-Seller is a top provider of such proxies. They offer over 20 million rotating residential IP addresses worldwide. You can target proxies by country, city, or ISP precisely. You also get options like sticky sessions or rotation by time or request count.

Here are Proxy-Seller’s features you’ll find useful for Instagram scraper Python projects:

High trust factor and excellent reputation among users
Full compliance with GDPR, CCPA, and other privacy laws
Fast proxy speeds up to 1 Gbps, suitable for heavy scraping
24/7 reliable customer support
Flexible pricing plans for scrapers of all scales

This makes Proxy-Seller ideal for scraping Instagram data without interruptions. These proxies help your Python Instagram scraper stay effectively under Instagram’s radar.

Cloud Platform for Sustainable Instagram Scraping

For a reliable and sustainable scraping workflow, use the Apify platform. Apify hosts ready-made web scrapers, called Actors, including many designed for Instagram. You don’t have to build everything from scratch, which saves time and hassle.

There are over 230 Instagram Scraping Actors available on the Apify Store. They cover tasks like profile data extraction, post scraping, comment collection, and more. Using these Actors means you get tried-and-tested solutions that handle Instagram’s challenges.

Cloud-based API scrapers like those on Apify offer key benefits:

They run on Apify’s servers, so your local machine isn’t involved.
They use rotating proxies built into the platform, reducing blocks.
Updates to the scrapers happen seamlessly, keeping up with Instagram API changes.
You get easy-to-use APIs to trigger scraping and download results.

Using Apify’s Instagram Scraper Python Actors makes your scraping setup sustainable. You save yourself from bot detection headaches by relying on Apify’s anti-blocking infrastructure.

Next Steps and Advanced Usage with Instagram Scraper Python

Once you master basic scraping, extend your scripts with pagination or batched scraping to collect more data beyond default limits. Apify Actors support cursor-based pagination, letting you automatically scrape multiple pages of Instagram posts.

Exporting scraped data is simple. You can save results locally as JSON or CSV using Python’s built-in json and csv modules. Alternatively, upload data directly to cloud databases like MongoDB Atlas or PostgreSQL. Use pymongo for MongoDB or psycopg2 for PostgreSQL to integrate smoothly.

After scraping, analyze post metrics using Python libraries like pandas for data handling and matplotlib or seaborn for visualization. This lets you track trends over time, helping you understand Instagram engagement patterns.

To automate scraping, use Apify’s ScheduleClient. It lets you set up cron-like jobs that run your scraper at regular intervals. This setup eliminates manual runs, so you collect fresh data continuously.

Integrating Premium Proxies for Custom Scripts

If you need custom scrapers beyond Apify’s ready Actors, integrate residential or rotating proxies in your configurations. Proxy-Seller simplifies proxy management with:

Support for SOCKS5 and HTTP(S) protocols
Two authentication types: username/password and IP whitelisting
Easy-to-use API and dashboard for proxy control
Variety of proxy types, including ISP proxies, datacenter IPv4/IPv6, and mobile proxies
24/7 customer support helping with configurations and troubleshooting

A private ISP proxy can be a practical option when you want more stable, ISP-backed IPs for repeated profile requests without constantly rotating endpoints.

These features make proxy integration in Python scripts and frameworks straightforward. You can fine-tune the proxy rotation and scale according to your scraping needs.

Monitoring and Troubleshooting

Finally, monitor your API usage and implement rate limit checks in your code to avoid blocks. Set up logging using Python’s logging module or third-party tools like Sentry. This helps catch scraper failures early and maintain stable operations.

To Sum Up

Scraping Instagram data with Python can be done by leveraging the backend API provided by Instagram, which helps bypass some of the frontend restrictions. Using the right headers to mimic browser behavior and employing proxies to avoid rate-limiting are critical steps. The Box library further simplifies the process by making JSON parsing more intuitive with dot notation. Before you start scraping Instagram at scale, remember to comply with Instagram's terms of service, and make sure your scraping efforts do not violate their policies.

And remember, by combining Apify’s Instagram scraper Python Actors with Proxy-Seller’s premium proxies and earlier-mentioned advanced strategies, you’ll build a robust, scalable Instagram email scraper Python GitHub or a general scraper that runs smoothly and sustainably.

","createdDate":"2025-02-26T09:31:19.000+00:00","description":"Gaining access to Instagram data can be tricky due to various anti-bot mechanisms, login requirements, and rate limits.","countOfViews":9,"attachments":[],"mainImg":{"id":"6a2499f11079162e2e8b8fe6","url":"https://proxy-seller.com/files/fs/image/6a2499f11079162e2e8b8fe6_how-to-scrape-instagram-data-using-python_mainImg.png","name":"how-to-scrape-instagram-data-using-python_mainImg.png","size":44007,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"how-to-scrape-instagram-data-using-python_img","metaTitle":"Scraping Instagram Data Using Python","metaDescription":"Explore how to scrape user data from Instagram, including profile names and follower counts using Python, through the social network’s official API endpoint."},{"id":"6a049d0535baad34714affdb","tag":"how-to-scrape-ebay-data-using-python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":4,"title":"How to Scrape eBay Data Using Python","body":"

eBay is a well-known online platform that offers trading opportunities in a wide range of products among its registered users. In this guide, we’ll explain how to scrape data from eBay listing using Python. As such, we will be interested in details that are available from the listing itself as well as from going to each of the products in turn for more fine details.

Requirements

To get started, make sure you have the following Python libraries installed:

Requests: For making HTTP requests.
lxml: For parsing HTML content.
Pandas: For saving data to a CSV file.

Install these libraries using:

\npip install requests lxml pandas\n\n

Understanding the eBay URL Structure for Pagination

When searching for products on eBay, each page URL can be modified to navigate through paginated results. For example:

Page 1: https://www.ebay.com/sch/i.html?_nkw=laptop
Page 2: https://www.ebay.com/sch/i.html?_nkw=laptop&_pgn=2

The _pgn parameter is used to navigate through multiple pages of listings, enabling the retrieval of extensive data. Let's begin the scraping process.

Step 1: Sending requests to eBay

To begin, we’ll set up headers to mimic a real browser request, which helps avoid detection and potential blocking by eBay’s anti-bot measures. Then we’ll send a request to the listing page to gather the links for each product.

\nimport requests\nfrom lxml.html import fromstring\n\n# Define headers to simulate a real browser\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'priority': 'u=0, i',\n    'sec-ch-ua': '\"Google Chrome\";v=\"129\", \"Not=A?Brand\";v=\"8\", \"Chromium\";v=\"129\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36',\n}\n\n# Request parameters for the search query\nparams = {\n    '_nkw': 'laptop',\n}\n\n# Send a request to the eBay listing page\nlisting_page_response = requests.get('https link', params=params, headers=headers)\nlisting_parser = fromstring(listing_page_response.text)\n\n

Step 2: Parsing the listing page

On the listing page, we’ll extract the URLs for individual products. This allows us to visit each product page to gather specific details, such as the product title, price, and more.

\n# Parse the listing page to extract product links\nlinks = listing_parser.xpath('//div[@class=\"s-item__info clearfix\"]/a[@_sp=\"p2351460.m1686.l7400\"]/@href')\n\n# Output a sample of the links found\nprint(\"Product Links:\", links[:5])  # Display the first five product links\n\n

Step 3: Scraping product data

With the product URLs in hand, we’ll visit each product page and extract the following details:

Product title;
Price;
Shipping cost;
Product condition;
Available quantity;
Sold quantity;
Payment options;
Return policy.

Next, we’ll loop through each link and use XPath expressions to locate the required information on the product page.

\nproduct_data = []\n\nfor url in links:\n    # Send a request to the product page\n    product_page_response = requests.get(url, headers=headers)\n    product_parser = fromstring(product_page_response.text)\n    \n    # Extract data using XPath\n    try:\n        product_title = product_parser.xpath('//h1[@class=\"x-item-title__mainTitle\"]/span/text()')[0]\n        price = product_parser.xpath('//div[@data-testid=\"x-price-primary\"]/span/text()')[0]\n        shipping_cost = product_parser.xpath('//div[@class=\"ux-labels-values col-12 ux-labels-values--shipping\"]//div[@class=\"ux-labels-values__values-content\"]/div/span/text()')[0]\n        product_condition = product_parser.xpath('//div[@class=\"x-item-condition-text\"]/div/span/span[2]/text()')[0]\n        available_quantity = product_parser.xpath('//div[@class=\"x-quantity__availability\"]/span/text()')[0]\n        sold_quantity = product_parser.xpath('//div[@class=\"x-quantity__availability\"]/span/text()')[1]\n        payment_options = ', '.join(product_parser.xpath('//div[@class=\"ux-labels-values col-12 ux-labels-values__column-last-row ux-labels-values--payments\"]/div[2]/div/div//span/@aria-label'))\n        return_policy = product_parser.xpath('//div[@class=\"ux-layout-section ux-layout-section--returns\"]//div[@class=\"ux-labels-values__values-content\"]/div/span/text()')[0]\n        \n        # Store data in a dictionary\n        product_info = {\n            'Title': product_title,\n            'Price': price,\n            'Shipping Cost': shipping_cost,\n            'Condition': product_condition,\n            'Available Quantity': available_quantity,\n            'Sold Quantity': sold_quantity,\n            'Payment Options': payment_options,\n            'Return Policy': return_policy,\n        }\n        product_data.append(product_info)\n    \n    except IndexError as e:\n        print(f\"An error occurred: {e}\")\n\n

Step 4: Saving data to a CSV file

After collecting the data, we can save it into a CSV file using Pandas.

\nimport pandas as pd\n\n# Convert data to DataFrame\ndf = pd.DataFrame(product_data)\n\n# Save to CSV\ndf.to_csv('ebay_product_data.csv', index=False)\nprint(\"Data saved to ebay_product_data.csv\")\n\n

Handling rate-limiting and bypassing detection on eBay

eBay employs rate-limiting to prevent excessive requests. Here are a few methods to avoid detection:

Use Proxies: Rotate between different IP addresses.
Adjust Request Intervals: Implement delays between requests.
Randomize User Agents: Vary the user-agent string to avoid detection.

By following these best practices, you can minimize the risk of getting blocked and continue scraping data efficiently.

Complete code

Here’s the full code for scraping eBay data and saving it to a CSV file:

\nimport requests\nimport random\nfrom lxml.html import fromstring\nimport pandas as pd\n\nuseragents = ['Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36',\n             'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36',\n             'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36']\n\n# Define headers for request\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'priority': 'u=0, i',\n    'sec-ch-ua': '\"Google Chrome\";v=\"129\", \"Not=A?Brand\";v=\"8\", \"Chromium\";v=\"129\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': random.choice(useragents),\n}\n\n# Search query parameters\nparams = {'_nkw': 'laptop'}\nproxies = {\n    'http': 'IP:PORT',\n    'https': 'IP:PORT'\n}\n\n# Fetch the listing page\nlisting_page_response = requests.get('https://www.ebay.com/sch/i.html', params=params, headers=headers, proxies=proxies)\nlisting_parser = fromstring(listing_page_response.text)\nlinks = listing_parser.xpath('//div[@class=\"s-item__info clearfix\"]/a[@_sp=\"p2351460.m1686.l7400\"]/@href')\n\n# Extract product data\nproduct_data = []\nfor url in links:\n    product_page_response = requests.get(url, headers=headers, proxies=proxies)\n    product_parser = fromstring(product_page_response.text)\n    try:\n        product_info = {\n            'Title': product_parser.xpath('//h1[@class=\"x-item-title__mainTitle\"]/span/text()')[0],\n            'Price': product_parser.xpath('//div[@data-testid=\"x-price-primary\"]/span/text()')[0],\n            'Shipping Cost': product_parser.xpath('//div[@class=\"ux-labels-values col-12 ux-labels-values--shipping\"]//div[@class=\"ux-labels-values__values-content\"]/div/span/text()')[0],\n            'Condition': product_parser.xpath('//div[@class=\"x-item-condition-text\"]/div/span/span[2]/text()')[0],\n            'Available Quantity': product_parser.xpath('//div[@class=\"x-quantity__availability\"]/span/text()')[0],\n            'Sold Quantity': product_parser.xpath('//div[@class=\"x-quantity__availability\"]/span/text()')[1],\n            'Payment Options': ', '.join(product_parser.xpath('//div[@class=\"ux-labels-values col-12 ux-labels-values__column-last-row ux-labels-values--payments\"]/div[2]/div/div//span/@aria-label')),\n            'Return Policy': product_parser.xpath('//div[@class=\"ux-layout-section ux-layout-section--returns\"]//div[@class=\"ux-labels-values__values-content\"]/div/span/text()')[0]\n        }\n        product_data.append(product_info)\n    except IndexError:\n        continue\n\n# Save to CSV\ndf = pd.DataFrame(product_data)\ndf.to_csv('ebay_product_data.csv', index=False)\nprint(\"Data saved to ebay_product_data.csv\")\n\n

Scraping eBay with Python allows for efficient data collection on products, pricing, and trends. In this guide, we covered scraping listings, handling pagination, setting headers, and using proxies to avoid detection. Remember to respect eBay’s terms of service by using responsible request intervals and proxy rotation. With these tools, you can now easily gather and analyze eBay data for market insights. Happy scraping!

","createdDate":"2025-02-12T14:52:47.000+00:00","description":"eBay is a well-known online platform that offers trading opportunities in a wide range of products among its registered users.","countOfViews":12,"attachments":[],"mainImg":{"id":"6a249a301079162e2e8b9aa8","url":"https://proxy-seller.com/files/fs/image/6a249a301079162e2e8b9aa8_how-to-scrape-ebay-data-using-python_mainImg.png","name":"how-to-scrape-ebay-data-using-python_mainImg.png","size":41440,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"how-to-scrape-ebay-data-using-python_img","metaTitle":"Guide to Scraping Data from eBay Using Python","metaDescription":"This guide covers scraping listings on eBay with Python, managing pagination, setting up headers, and using proxies to avoid detection."},{"id":"6a049d0935baad34714b000f","tag":"how-to-scrape-glassdoor-data-using-python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":4,"title":"How to Scrape Glassdoor Data Using Python","body":"

Glassdoor is one of the best platforms that provides a wide range of information for people both seeking jobs and wanting to hire them, including salary information, employer reviews and job applications. In this guide, we’ll walk through the process of scraping job listings from Glassdoor using Python and Playwright. Playwright is essential here because Glassdoor employs strong anti-bot measures, which can flag and block traditional scraping libraries. With Playwright, we can simulate a real browser and include proxies, helping us bypass these detection systems.

Due to Glassdoor's robust anti-scraping mechanisms, direct requests with libraries like requests can lead to IP blocking or CAPTCHA challenges. Playwright allows us to automate a browser, making our interactions more human-like. By adding proxies and browser headers, we can further avoid detection.

Requirements

To get started, you’ll need to install Playwright and the lxml library for HTML parsing. You can install them as follows:

\npip install playwright lxml\nplaywright install\n\n

Scraping Glassdoor job listings

We’ll walk through each step, from loading the page with Playwright to extracting job details and saving the data into a CSV file.

Step 1. Setting up the browser and making requests

First, set up Playwright with a proxy to connect to Glassdoor. This helps prevent getting blocked and allows the browser to load the page as if a real user were visiting the site.

\nfrom playwright.async_api import async_playwright\nfrom lxml.html import fromstring\n\nasync def scrape_job_listings():\n    async with async_playwright() as p:\n        browser = await p.chromium.launch(\n            headless=False,\n            proxy={\"server\": '', 'username': '', 'password': ''}\n        )\n        page = await browser.new_page()\n        await page.goto('https link', timeout=60000)\n        content = await page.content()\n        await browser.close()\n        return content\n\n# Call the function to retrieve page content\nhtml_content = await scrape_job_listings()\n\n

Step 2. Parsing the HTML and extracting data

After loading the page, use lxml to parse the HTML content and extract relevant job information. Here’s how to parse the job title, location, salary, and other details for each job listing:

\nparser = fromstring(html_content)\njob_posting_elements = parser.xpath('//li[@data-test=\"jobListing\"]')\n\njobs_data = []\nfor element in job_posting_elements:\n    job_title = element.xpath('.//a[@data-test=\"job-title\"]/text()')[0]\n    job_location = element.xpath('.//div[@data-test=\"emp-location\"]/text()')[0]\n    salary = ' '.join(element.xpath('.//div[@data-test=\"detailSalary\"]/text()')).strip()\n    job_link = element.xpath('.//a[@data-test=\"job-title\"]/@href')[0]\n    easy_apply = bool(element.xpath('.//div[@data-role-variant=\"featured\"]'))\n    company = element.xpath('.//span[@class=\"EmployerProfile_compactEmployerName__LE242\"]/text()')[0]\n    \n    job_data = {\n        'company': company,\n        'job_title': job_title,\n        'job_location': job_location,\n        'job_link': job_link,\n        'salary': salary,\n        'easy_apply': easy_apply\n    }\n    jobs_data.append(job_data)\n\n

Step 3. Saving data to a CSV file

Once we’ve extracted the job details, we can save them into a CSV file for easy data analysis.

\nimport csv\n\nwith open('glassdoor_job_listings.csv', 'w', newline='', encoding='utf-8') as file:\n    writer = csv.DictWriter(file, fieldnames=['company', 'job_title', 'job_location', 'job_link', 'salary', 'easy_apply'])\n    writer.writeheader()\n    writer.writerows(jobs_data)\n\n

Complete code

\nimport csv\nfrom playwright.async_api import async_playwright\nfrom lxml.html import fromstring\n\nasync def scrape_job_listings():\n    # Setup the Playwright browser with proxy to avoid detection\n    async with async_playwright() as p:\n        browser = await p.chromium.launch(\n            headless=False,\n            proxy={\"server\": '', 'username': '', 'password': ''}\n        )\n        page = await browser.new_page()\n        await page.goto('https://www.glassdoor.com/Job/united-states-software-engineer-jobs-SRCH_IL.0,13_IN1_KO14,31.htm', timeout=60000)\n        \n        # Retrieve the page content and close the browser\n        content = await page.content()\n        await browser.close()\n        \n        # Parse the content with lxml\n        parser = fromstring(content)\n        job_posting_elements = parser.xpath('//li[@data-test=\"jobListing\"]')\n        \n        # Extract data for each job listing\n        jobs_data = []\n        for element in job_posting_elements:\n            job_title = element.xpath('.//a[@data-test=\"job-title\"]/text()')[0]\n            job_location = element.xpath('.//div[@data-test=\"emp-location\"]/text()')[0]\n            salary = ' '.join(element.xpath('.//div[@data-test=\"detailSalary\"]/text()')).strip()\n            job_link = \"https://www.glassdoor.com\" + element.xpath('.//a[@data-test=\"job-title\"]/@href')[0]\n            easy_apply = bool(element.xpath('.//div[@data-role-variant=\"featured\"]'))\n            company = element.xpath('.//span[@class=\"EmployerProfile_compactEmployerName__LE242\"]/text()')[0]\n            \n            job_data = {\n                'company': company,\n                'job_title': job_title,\n                'job_location': job_location,\n                'job_link': job_link,\n                'salary': salary,\n                'easy_apply': easy_apply\n            }\n            jobs_data.append(job_data)\n    \n        # Save the data to a CSV file\n        with open('glassdoor_job_listings.csv', 'w', newline='', encoding='utf-8') as file:\n            writer = csv.DictWriter(file, fieldnames=['company', 'job_title', 'job_location', 'job_link', 'salary', 'easy_apply'])\n            writer.writeheader()\n            writer.writerows(jobs_data)\n\n# Run the scraping function\nimport asyncio\nasyncio.run(scrape_job_listings())\n\n

Explanation of the Complete Code:

Browser Setup with Proxy: The code initiates a browser session with Playwright, incorporating a proxy to mimic human browsing behavior. The headless=False setting enables the browser window to remain open, which can further help bypass bot detection.
Navigating to the Job Listings Page: The script visits the Glassdoor job listings URL for software engineering jobs in the United States.
Parsing the Content: The job data is extracted using lxml for HTML parsing. We capture the job title, location, salary, job link, company name, and if it’s an easy apply job.
Saving to CSV: After extracting all the data, the script saves it to a CSV file, glassdoor_job_listings.csv, with columns for each attribute.

Respecting Glassdoor's terms of service

When scraping Glassdoor or any other website, it’s essential to follow responsible scraping practices:

Respect Rate Limits: Avoid overwhelming the server by implementing delays between requests.
Use Rotating Proxies: Minimize the risk of getting banned by rotating proxies and IPs.
Comply with Terms of Service: Regularly review the website’s terms of service and avoid actions that violate them.

When you know how to scrape Glassdoor’s data using Python and Playwright, you will easily enhance your ability to collect job listings. This technique, when combined with the use of proxies and appropriate headers, is effective in eliminating the risk of being blocked by Glassdoor. You need to also take note of ethical scraping policies to prevent crashing the servers of Glassdoor. By keeping to these measures, you can now harvest as well as process useful employment information from Glassdoor for your own use or that of your company.

","createdDate":"2025-02-04T13:47:21.000+00:00","description":"Glassdoor is one of the best platforms that provides a wide range of information for people both seeking jobs and wanting to hire them.","countOfViews":9,"attachments":[],"mainImg":{"id":"6a249a781079162e2e8ba60a","url":"https://proxy-seller.com/files/fs/image/6a249a781079162e2e8ba60a_how-to-scrape-glassdoor-data-using-python_mainImg.png","name":"how-to-scrape-glassdoor-data-using-python_mainImg.png","size":43029,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"how-to-scrape-glassdoor-data-using-python_img","metaTitle":"Step-by-Step Creation of a Python Script for Scraping Glassdoor","metaDescription":"A guide on how to scrape data from Glassdoor using Python. Scraping information about job titles, locations, and salaries."},{"id":"6a049d0b35baad34714b002f","tag":"how-to-scrape-medium-articles-using-python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":4,"title":"How to scrape Medium articles using Python","body":"

Extracting articles from Medium can be of utmost importance for purposes such as content evaluation, data collecting, or monitoring authors and their works. In this tutorial, we will learn how to scrape medium – an article website for writers, using python programming language. We will discuss how data such as article title, the name of the author, the name of the publication, and the text body itself can be extracted from a given web URL of a Medium article.

Requirements

For this tutorial, we’ll be scraping this article on Medium: “9 Python Built-in Decorators That Optimize Your Code Significantly”.

Before you start, install the following libraries:

Requests: To send HTTP requests to Medium.
lxml: For parsing HTML content.
Pandas: To save the data to a CSV file.

Install them with the following commands:

\npip install requests\npip install lxml \npip install pandas\n\n

Understanding the importance of headers and proxies

Medium uses bot detection techniques to prevent unauthorized scraping. Proper headers and proxies are crucial for avoiding bot detection and for responsible scraping.

Headers: These simulate a request as if it’s coming from a real browser. They include information like the browser type, accepted content types, and caching behavior.

\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'priority': 'u=0, i',\n    'sec-ch-ua': '\"Google Chrome\";v=\"129\", \"Not=A?Brand\";v=\"8\", \"Chromium\";v=\"129\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36',\n}\n\n

Proxies: Proxies can mask your IP address, rotating it periodically to make it less likely for the requests to get blocked by Medium. Here is an example of using it with IP address authentication:

\nproxies = {\n    'http': 'IP:PORT',\n    'https': 'IP:PORT'\n}\n\nresponse = requests.get(\n 'https://medium.com/techtofreedom/9-python-built-in-decorators-that-optimize-your-code-significantly-bc3f661e9017',\n    headers=headers,\n    proxies=proxies\n)\n\n

Sending a request to Medium

Here's how to set up the headers and send a request to the article URL:

\nimport requests\n\n# Headers to simulate a real browser request\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'priority': 'u=0, i',\n    'sec-ch-ua': '\"Google Chrome\";v=\"129\", \"Not=A?Brand\";v=\"8\", \"Chromium\";v=\"129\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36',\n}\n\nurl = 'https://medium.com/techtofreedom/9-python-built-in-decorators-that-optimize-your-code-significantly-bc3f661e9017'\nresponse = requests.get(url, headers=headers)\n\n

Extracting data

Once we have the page content, we can parse it and extract relevant information.

Parsing HTML Content

We'll use lxml to parse the HTML response and extract specific elements. Here’s how to do it:

\nfrom lxml.html import fromstring\n\nparser = fromstring(response.text)\n\n# Extract data\ntitle = parser.xpath('//h1[@data-testid=\"storyTitle\"]/text()')[0]\nauthor = parser.xpath('//a[@data-testid=\"authorName\"]/text()')[0]\npublication_name = parser.xpath('//a[@data-testid=\"publicationName\"]/p/text()')[0]\npublication_date = parser.xpath('//span[@data-testid=\"storyPublishDate\"]/text()')[0]\ncontent = '\\n '.join(parser.xpath('//div[@class=\"ci bh ga gb gc gd\"]/p/text()'))\nauth_followers = parser.xpath('//span[@class=\"pw-follower-count bf b bg z bk\"]/a/text()')[0]\nsub_title = parser.xpath('//h2[@id=\"1de6\"]/text()')[0]\n

Now, we'll create a dictionary to hold all extracted data. This makes it easier to save to a CSV file.

\n# Save data in a dictionary\narticle_data = {\n    'Title': title,\n    'Author': author,\n    'Publication': publication_name,\n    'Date': publication_date,\n    'Followers': auth_followers,\n    'Subtitle': sub_title,\n    'Content': content,\n}\n\nprint(article_data)\n\n

Saving data to a CSV file

Finally, let’s save the data to a CSV file for further analysis or record-keeping.

\nimport pandas as pd\n\n# Convert dictionary to DataFrame and save as CSV\ndf = pd.DataFrame([article_data])\ndf.to_csv('medium_article_data.csv', index=False)\nprint(\"Data saved to medium_article_data.csv\")\n\n

Full code

Here's the complete code for scraping the Medium article data:

\nimport requests\nfrom lxml.html import fromstring\nimport pandas as pd\n\n# Headers to mimic a browser\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'priority': 'u=0, i',\n    'sec-ch-ua': '\"Google Chrome\";v=\"129\", \"Not=A?Brand\";v=\"8\", \"Chromium\";v=\"129\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36',\n}\n\n\nproxies = {\n    'http': 'IP:PORT',\n    'https': 'IP:PORT'\n}\n\n# Requesting the page\nurl = 'https://medium.com/techtofreedom/9-python-built-in-decorators-that-optimize-your-code-significantly-bc3f661e9017'\nresponse = requests.get(url, headers=headers, proxies=proxies)\n\n# Parsing the page\nparser = fromstring(response.text)\n\n# Extract data\ntitle = parser.xpath('//h1[@data-testid=\"storyTitle\"]/text()')[0]\nauthor = parser.xpath('//a[@data-testid=\"authorName\"]/text()')[0]\npublication_name = parser.xpath('//a[@data-testid=\"publicationName\"]/p/text()')[0]\npublication_date = parser.xpath('//span[@data-testid=\"storyPublishDate\"]/text()')[0]\ncontent = '\\n '.join(parser.xpath('//div[@class=\"ci bh ga gb gc gd\"]/p/text()'))\nauth_followers = parser.xpath('//span[@class=\"pw-follower-count bf b bg z bk\"]/a/text()')[0]\nsub_title = parser.xpath('//h2[@id=\"1de6\"]/text()')[0]\n\n# Saving data\narticle_data = {\n    'Title': title,\n    'Author': author,\n    'Publication': publication_name,\n    'Date': publication_date,\n    'Followers': auth_followers,\n    'Subtitle': sub_title,\n    'Content': content,\n}\n\n# Save to CSV\ndf = pd.DataFrame([article_data])\ndf.to_csv('medium_article_data.csv', index=False)\nprint(\"Data saved to medium_article_data.csv\")\n\n

Scraping content from Medium should be conducted responsibly. Excessive request load on servers can affect the service's performance, and scraping data without permission may violate the website's terms of use. Always check the robots.txt file and terms before scraping any website.

","createdDate":"2025-01-29T09:41:51.000+00:00","description":"Extracting articles from Medium can be of utmost importance for purposes such as content evaluation, data collecting.","countOfViews":8,"attachments":[],"mainImg":{"id":"6a249ab71079162e2e8bb23a","url":"https://proxy-seller.com/files/fs/image/6a249ab71079162e2e8bb23a_how-to-scrape-medium-articles-using-python_mainImg.png","name":"how-to-scrape-medium-articles-using-python_mainImg.png","size":39790,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"how-to-scrape-medium-articles-using-python_img","metaTitle":"Scraping Medium Articles Using Python","metaDescription":"This guide provides a script for scraping Medium articles, including details such as the article title, author's name, publication title, and saving them in a CSV file."},{"id":"6a049d0e35baad34714b0056","tag":"how-to-scrape-aliexpress-data-using-python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":5,"title":"How to scrape AliExpress data using Python","body":"

Uncovering information from e-retailers such as AliExpress can be highly beneficial for product information gathering, monitoring price fluctuations, collecting reviews, and so on. In this article, we will explore the process of acquiring information about products (such as name, price, rating, etc.) and also review scraping product reviews. We will also demonstrate how to make the scraper dynamic by passing the product URL, automatically retrieving the product ID, and saving the data into a CSV file.

This tutorial will use Playwright to render dynamic content and requests to fetch review data. We’ll also ensure the scraper is ethical and complies with best practices. We’ll also ensure the scraper is ethical and complies with best practices. For stable connections and region-specific data collection during testing, consider using Aliexpress proxy.

Requirements

Before we begin, ensure you have the following Python libraries installed:

Playwright: used to interact with the browser and render dynamic content.
Requests: used to fetch reviews via the AliExpress API.
lxml: for parsing the HTML content.
Pandas: this is used to save the data to a CSV file.

You can install these packages by running the following commands:

\n# Install Playwright\npip install playwright\n\n\n# Install Requests\npip install requests\n\n\n# Install lxml for parsing HTML\npip install lxml\n\n\n# Install Pandas for data manipulation and saving\npip install pandas\n\n

After installing Playwright, you will also need to install the required browser binaries:

\nplaywright install\n\n

This will download and set up the necessary browser for Playwright to function properly.

Step 1. Sending requests with Playwright

AliExpress product pages are dynamic, meaning they load content via JavaScript. To handle this, we’ll use Playwright, a Python library that allows you to control a headless browser and interact with dynamic content.

Here's how you can send a request and navigate to the product page:

\nfrom playwright.async_api import async_playwright\n\nasync def get_page_content(url):\n    async with async_playwright() as p:\n        # Launch the browser with a proxy if needed (can be removed if not using proxy)\n        browser = await p.firefox.launch(\n            headless=False,\n            proxy={\"server\": '', 'username': '', 'password': ''}\n        )\n        page = await browser.new_page()\n        await page.goto(url, timeout=60000)\n\n        # Extract page content\n        content = await page.content()\n        await browser.close()\n        \n        return content\n\n# Example URL\nurl = 'https://www.aliexpress.com/item/3256805354456256.html'\n\n

Step 2. Extracting product data

Once we have the page content, we can extract the product data using lxml and XPath queries. We will gather details like the product title, price, rating, number of reviews, and the number of items sold.

\nfrom lxml.html import fromstring\n\ndef extract_product_data(content):\n    parser = fromstring(content)\n    \n    # Extract product details using XPath\n    title = parser.xpath('//h1[@data-pl=\"product-title\"]/text()')[0].strip()\n    price = parser.xpath('//div[@class=\"price--current--I3Zeidd product-price-current\"]/span/text()')[0].strip()\n    rating = ' '.join(parser.xpath('//a[@class=\"reviewer--rating--xrWWFzx\"]/strong/text()')).strip()\n    total_reviews = parser.xpath('//a[@class=\"reviewer--reviews--cx7Zs_V\"]/text()')[0].strip()\n    sold_count = parser.xpath('//span[@class=\"reviewer--sold--ytPeoEy\"]/text()')[0].strip()\n\n    product_data = {\n        'title': title,\n        'price': price,\n        'rating': rating,\n        'total_reviews': total_reviews,\n        'sold_count': sold_count\n    }\n\n    return product_data\n\n\n

This code uses XPath to extract relevant product details from the HTML content of the page.

Step 3. Scraping product reviews

AliExpress has a separate API endpoint for fetching product reviews. You can extract the product ID from the URL dynamically and use it to fetch reviews via requests. In this function:

The product ID is extracted from the product URL dynamically.
We fetch the reviews using the AliExpress review API.
The review texts are extracted and returned as a list.

\nimport requests\n\ndef extract_product_id(url):\n    # Extract product ID from the URL\n    product_id = url.split('/')[-1].split('.')[0]\n    return product_id\n\ndef scrape_reviews(product_id, page_num=1, page_size=10):\n    headers = {\n        'accept': 'application/json, text/plain, */*',\n        'accept-language': 'en-IN,en;q=0.9',\n        'referer': f'https://www.aliexpress.com/item/{product_id}.html',\n        'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36',\n    }\n\n    params = {\n        'productId': product_id,\n        'lang': 'en_US',\n        'country': 'US',\n        'page': str(page_num),\n        'pageSize': str(page_size),\n        'filter': 'all',\n        'sort': 'complex_default',\n    }\n\n    response = requests.get('https://feedback.aliexpress.com/pc/searchEvaluation.do', params=params, headers=headers)\n    reviews = response.json()['data']['evaViewList']\n\n    # Extract review text only\n    review_texts = [review['buyerFeedback'] for review in reviews]\n    \n    return review_texts\n\n

Step 4. Saving data into a CSV file

After scraping the product details and reviews, we save this data into a CSV file using the pandas library.

\nimport pandas as pd\n\ndef save_to_csv(product_data, reviews, product_id):\n    # Save product details to CSV\n    df_product = pd.DataFrame([product_data])\n    df_product.to_csv(f'product_{product_id}_data.csv', index=False)\n\n    # Save reviews to CSV\n    df_reviews = pd.DataFrame({'reviews': reviews})\n    df_reviews.to_csv(f'product_{product_id}_reviews.csv', index=False)\n    \n    print(f\"Data saved for product {product_id}.\")\n\n

The product details and reviews are saved into separate CSV files with the product ID included in the filename for easy identification.

Step 5. Dynamic product ID retrieval

Here’s how the complete dynamic workflow works:

Pass any AliExpress product URL.
The product ID is extracted from the URL.
The scraper fetches product data and reviews.
Data is saved into CSV files with the product ID included.

\n# Extract product ID from the URL\ndef extract_product_id(url):\n    return url.split('/')[-1].split('.')[0]\n\n

Final complete code

\nfrom playwright.async_api import async_playwright\nfrom lxml.html import fromstring\nimport requests\nimport pandas as pd\n\n# Get page content using Playwright\nasync def get_page_content(url):\n    async with async_playwright() as p:\n        browser = await p.firefox.launch(\n            headless=False,\n            proxy={\"server\": '', 'username': '', 'password': ''}\n        )\n        page = await browser.new_page()\n        await page.goto(url, timeout=60000)\n        content = await page.content()\n        await browser.close()\n        return content\n\n# Extract product data\ndef extract_product_data(content):\n    parser = fromstring(content)\n    title = parser.xpath('//h1[@data-pl=\"product-title\"]/text()')[0].strip()\n    price = parser.xpath('//div[@class=\"price--current--I3Zeidd product-price-current\"]/span/text()')[0].strip()\n    rating = ' '.join(parser.xpath('//a[@class=\"reviewer--rating--xrWWFzx\"]/strong/text()')).strip()\n    total_reviews = parser.xpath('//a[@class=\"reviewer--reviews--cx7Zs_V\"]/text()')[0].strip()\n    sold_count = parser.xpath('//span[@class=\"reviewer--sold--ytPeoEy\"]/text()')[0].strip()\n\n    return {\n        'title': title,\n        'price': price,\n        'rating': rating,\n        'total_reviews': total_reviews,\n        'sold_count': sold_count\n    }\n\n# Extract product ID from the URL\ndef extract_product_id(url):\n    return url.split('/')[-1].split('.')[0]\n\n# Scrape reviews\ndef scrape_reviews(product_id, page_num=1, page_size=10):\n    headers = {\n        'accept': 'application/json, text/plain, */*',\n        'referer': f'https://www.aliexpress.com/item/{product_id}.html',\n        'user-agent': 'Mozilla/5.0'\n    }\n    params = {\n        'productId': product_id,\n        'lang': 'en_US',\n        'page': str(page_num),\n        'pageSize': str(page_size),\n    }\n    response = requests.get('https://feedback.aliexpress.com/pc/searchEvaluation.do', params=params, headers=headers)\n    reviews = response.json()['data']['evaViewList']\n    return [review['buyerFeedback'] for review in reviews]\n\n# Save product data and reviews to CSV\ndef save_to_csv(product_data, reviews, product_id):\n    pd.DataFrame([product_data]).to_csv(f'product_{product_id}_data.csv', index=False)\n    pd.DataFrame({'reviews': reviews}).to_csv(f'product_{product_id}_reviews.csv', index=False)\n    print(f'Saved into: product_{product_id}_data.csv')\n    print(f'Saved into: product_{product_id}_reviews.csv')\n\n# Main function\nasync def main(url):\n    content = await get_page_content(url)\n    product_data = extract_product_data(content)\n    product_id = extract_product_id(url)\n    reviews = scrape_reviews(product_id)\n    save_to_csv(product_data, reviews, product_id)\n\n# Run the scraper\nimport asyncio\nurl = 'https://www.aliexpress.com/item/3256805354456256.html'\nasyncio.run(main(url))\n\n

Ethical considerations

When scraping data, it's important to follow ethical guidelines:

Respect AliExpress’s terms of service: Always check the terms of service before scraping a website. Avoid violating their rules to prevent getting banned.
Throttle your requests: Sending too many requests in a short time can overload their servers. Consider adding delays between requests.
Avoid personal data: Do not collect or scrape personal information without proper consent.

Following these guidelines will help you scrape ethically and responsibly, minimizing risks for both users and the AliExpress system.

","createdDate":"2025-01-24T09:55:46.000+00:00","description":"Uncovering information from e-retailers such as AliExpress can be highly beneficial for product information gathering, monitoring price fluctuations, collecting reviews, and so on.","countOfViews":7,"attachments":[],"mainImg":{"id":"6a249ae71079162e2e8bbb0a","url":"https://proxy-seller.com/files/fs/image/6a249ae71079162e2e8bbb0a_how-to-scrape-aliexpress-data-using-python_mainImg.png","name":"how-to-scrape-aliexpress-data-using-python_mainImg.png","size":42016,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"how-to-scrape-aliexpress-data-using-python_img","metaTitle":"Scraping Data from AliExpress Using Python","metaDescription":"This guide demonstrates how to dynamically extract data from AliExpress, retrieving URLs, product IDs, their names, and reviews, and saving the results in CSV format."},{"id":"6a049d1035baad34714b008b","tag":"how-to-scrape-imdb-data-using-python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":5,"title":"How to scrape IMDB data using Python","body":"

In today’s world, extracting data from online social platforms such as IMDB can be an effective road towards collecting much-needed movie-related information for research or enjoyment purposes. In this tutorial, we will walk through scraping the Top 250 movies on IMDB using Python and extract details such as movie titles, movie summaries, ratings, genres, and more.

When scraping websites like IMDB, it's crucial to simulate the behavior of a real user to minimize the risk of detection and ensure successful data retrieval. Here are some strategies that can be employed:

Avoid IP Blocking: Websites often limit the number of requests that can be made from a single IP address to prevent scraping. By using proxies, you can spread your requests over multiple IP addresses, reducing the risk of being blocked.
Ensure Anonymity: Proxies mask your real IP address, which not only helps to protect your privacy but also makes it harder for websites to track scraping activities back to you.
Comply with Speed Limits: Distributing requests through multiple proxies can help manage the frequency of your queries, staying within the website’s rate limits and reducing the likelihood of triggering anti-scraping measures.
Bypass Server Suspicions: Incorporating headers that mimic those of a typical browser, such as User-Agent, can make your scraping requests appear more like normal user requests. This can prevent the server from flagging your activities as suspicious.

Step 1: Preparing the Scraper

For this tutorial, we will use Python's requests library to download the web content, lxml for parsing HTML, and optionally, the json library for handling formatted data when needed. First, install the required libraries.

Installing the libraries

Before starting, you need to install the necessary Python libraries. Run the following command in your terminal to install them:

\npip install requests lxml\n\n

These libraries will be used to make HTTP requests, parse HTML content, and process the extracted data.

Configuring HTTP Request Headers

To make our requests resemble those from a real web browser, it’s crucial to set up the HTTP headers accordingly. Here is an example of how you might configure these headers in your script:

\nimport requests\n\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',  # Do Not Track header\n    'pragma': 'no-cache',\n    'sec-ch-ua': '\"Google Chrome\";v=\"129\", \"Not=A?Brand\";v=\"8\", \"Chromium\";v=\"129\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36',\n}\n\nresponse = requests.get('https://www.imdb.com/chart/top/', headers=headers)\n\n

Setting Up Proxies

Proxies are useful for large-scale scraping. They help you avoid getting blocked by distributing your requests across multiple IPs. Here’s how you can include a proxy:

\nproxies = {\n    \"http\": \"http://your_proxy_server\",\n    \"https\": \"https://your_proxy_server\"\n}\n\nresponse = requests.get('https://www.imdb.com/chart/top/', headers=headers, proxies=proxies)\n\n

Replace \"your_proxy_server\" with the actual proxy details you have access to. This ensures that your IP address is not exposed, and it helps to avoid getting blocked.

Step 2: Parsing the HTML Content

After fetching the webpage content, we need to parse it to extract movie details. We'll use lxml to parse the HTML and json to handle the structured data:

\nfrom lxml.html import fromstring\nimport json\n\n# Parse the HTML response\nparser = fromstring(response.text)\n\n# Extract the JSON-LD data (structured data) from the script tag\nraw_data = parser.xpath('//script[@type=\"application/ld+json\"]/text()')[0]\njson_data = json.loads(raw_data)\n\n# Now we have structured movie data in JSON format\n\n

Step 3: Extracting Movie Details

The IMDB Top 250 page includes structured data embedded in the HTML, which can be easily accessed using XPath and parsed as JSON. We’ll extract movie details such as name, description, ratings, genres, and more:

\nmovies_details = json_data.get('itemListElement')\n\n# Loop through the movie data\nmovies_data = []\nfor movie in movies_details:\n    movie_type = movie['item']['@type']\n    url = movie['item']['url']\n    name = movie['item']['name']\n    description = movie['item']['description']\n    image = movie['item']['image']\n    bestRating = movie['item']['aggregateRating']['bestRating']\n    worstRating = movie['item']['aggregateRating']['worstRating']\n    ratingValue = movie['item']['aggregateRating']['ratingValue']\n    ratingCount = movie['item']['aggregateRating']['ratingCount']\n    contentRating = movie['item'].get('contentRating')\n    genre = movie['item']['genre']\n    duration = movie['item']['duration']\n    \n    # Append each movie's data to the list\n    movies_data.append({\n        'movie_type': movie_type,\n        'url': url,\n        'name': name,\n        'description': description,\n        'image': image,\n        'bestRating': bestRating,\n        'worstRating': worstRating,\n        'ratingValue': ratingValue,\n        'ratingCount': ratingCount,\n        'contentRating': contentRating,\n        'genre': genre,\n        'duration': duration\n    })\n\n

Step 4: Storing the Data

Once the data is extracted, it’s important to store it in a format that’s easy to analyze. In this case, we'll save it to a CSV file using the pandas library:

\nimport pandas as pd\n\n# Convert the list of movies to a pandas DataFrame\ndf = pd.DataFrame(movies_data)\n\n# Save the data to a CSV file\ndf.to_csv('imdb_top_250_movies.csv', index=False)\n\nprint(\"IMDB Top 250 movies data saved to imdb_top_250_movies.csv\")\n\n

Complete Code

Here’s the complete code for scraping IMDB’s Top 250 movies:

\nimport requests\nfrom lxml.html import fromstring\nimport json\nimport pandas as pd\n\n# Define headers for the request\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'sec-ch-ua': '\"Google Chrome\";v=\"129\", \"Not=A?Brand\";v=\"8\", \"Chromium\";v=\"129\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36',\n}\n\n# Optionally, set up proxies\nproxies = {\n    \"http\": \"http://your_proxy_server\",\n    \"https\": \"https://your_proxy_server\"\n}\n\n# Send the request to the IMDB Top 250 page\nresponse = requests.get('https://www.imdb.com/chart/top/', headers=headers, proxies=proxies)\n\n# Parse the HTML response\nparser = fromstring(response.text)\n\n# Extract the JSON-LD data\nraw_data = parser.xpath('//script[@type=\"application/ld+json\"]/text()')[0]\njson_data = json.loads(raw_data)\n\n# Extract movie details\nmovies_details = json_data.get('itemListElement')\n\nmovies_data = []\nfor movie in movies_details:\n    movie_type = movie['item']['@type']\n    url = movie['item']['url']\n    name = movie['item']['name']\n    description = movie['item']['description']\n    image = movie['item']['image']\n    bestRating = movie['item']['aggregateRating']['bestRating']\n    worstRating = movie['item']['aggregateRating']['worstRating']\n    ratingValue = movie['item']['aggregateRating']['ratingValue']\n    ratingCount = movie['item']['aggregateRating']['ratingCount']\n    contentRating = movie['item'].get('contentRating')\n    genre = movie['item']['genre']\n    duration = movie['item']['duration']\n    \n    movies_data.append({\n        'movie_type': movie_type,\n        'url': url,\n        'name': name,\n        'description': description,\n        'image': image,\n        'bestRating': bestRating,\n        'worstRating': worstRating,\n        'ratingValue': ratingValue,\n        'ratingCount': ratingCount,\n        'contentRating': contentRating,\n        'genre': genre,\n        'duration': duration\n    })\n\n# Save the data to a CSV file\ndf = pd.DataFrame(movies_data)\ndf.to_csv('imdb_top_250_movies.csv', index=False)\nprint(\"IMDB Top 250 movies data saved to imdb_top_250_movies.csv\")\n\n

Ethical Considerations

Before scraping any website, it’s important to consider ethical and legal issues:

Respect the Robots.txt: Check IMDB’s robots.txt file to see what parts of the website are allowed for scraping. Always adhere to the website’s policies.
Avoid Overloading the Server: Scrape data responsibly by limiting the frequency of your requests to avoid putting unnecessary load on the server.
Respect Terms of Service: Ensure that scraping doesn’t violate IMDB’s terms of service.

Always be mindful of the rules and use web scraping for legitimate purposes.

","createdDate":"2025-01-21T10:33:00.000+00:00","description":"In today’s world, extracting data from online social platforms such as IMDB can be an effective road towards collecting much-needed movie-related information.","countOfViews":7,"attachments":[],"mainImg":{"id":"6a249b1d1079162e2e8bc24d","url":"https://proxy-seller.com/files/fs/image/6a249b1d1079162e2e8bc24d_how-to-scrape-imdb-data-using-python_mainImg.png","name":"how-to-scrape-imdb-data-using-python_mainImg.png","size":39038,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"how-to-scrape-imdb-data-using-python_img","metaTitle":"Scraping Data from IMDB Using Python","metaDescription":"This guide demonstrates how to extract data from the IMDB Top 250 movies list and presents the information in an organized format for future use."},{"id":"6a049d1835baad34714b00f9","tag":"how-to-scrape-indeed-job-listings-using-python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":7,"title":"How to Scrape Indeed Job Listings Using Python","body":"

For job hunters, employers, or any individuals monitoring trends in the job market, scraping Indeed's list of available jobs could provide helpful information. In this particular tutorial, we will combine Playwright for web scraping and lxml for HTML content parsing in order to collect the details of the job including its title, name of the recruiting company, location, job description, job posting link, and finally present the findings by saving the information in a CSV file.

Prerequisites

To successfully perform scraping, the following Python libraries need to be installed.

Playwright for browser automation:

\npip install playwright\n

lxml for parsing HTML:

\npip install lxml\n

pandas for saving data to a CSV file:

\npip install pandas\n

Install Playwright browsers:

After installing Playwright, run this command to install the necessary browser binaries:

\nplaywright install\n\n

Step 1: Setting up Playwright for web scraping

Playwright allows you to automate and interact with web browsers. We start by setting up Playwright to launch a Chromium browser, visit a webpage, and extract its content. Here we can also pass proxies through the playwright.

Why use proxies?

Websites often have rate-limiting or anti-scraping measures in place to block repeated requests from the same IP address. Proxies allow you to:

Avoid IP blocking: distribute your requests through different IPs to avoid detection.
Geolocation bypassing: access job listings that might be restricted based on geographical locations.
Anonymity: hide your actual IP and stay anonymous during the scraping process.

\nimport asyncio\nfrom playwright.async_api import async_playwright\n\nasync def get_page_content(url):\n    async with async_playwright() as p:\n        browser = await p.chromium.launch(\n            headless=False,\n            proxy = {\n                'server': '',\n                'username': '',\n                'password': ''\n            }\n        )  # Headed browser\n        page = await browser.new_page()\n        await page.goto(url)\n        \n        # Extract the page's content\n        content = await page.content()\n        \n        await browser.close()  # Close the browser once done\n        return content\n\n

In this code, async_playwright launches a headed browser, navigates to the specified URL, and fetches the page's content.

Parsing HTML content using lxml

Next, we will parse the page content to extract meaningful data. lxml is used for this purpose because it provides robust support for parsing and querying HTML content using XPath.

\nfrom lxml import html\n\ndef parse_job_listings(content):\n    # Parse HTML content\n    parser = html.fromstring(content)\n    \n    # Extract each job posting using XPath\n    job_posting = parser.xpath('//ul[@class=\"css-zu9cdh eu4oa1w0\"]/li')\n    \n    jobs_data = []\n    for element in job_posting[:-1]:  # Skip the last element if it's an ad or irrelevant\n        title = ''.join(element.xpath('.//h2/a/span/@title'))\n        if title:\n            link = ''.join(element.xpath('.//h2/a/@href'))\n            location = ''.join(element.xpath('.//div[@data-testid=\"text-location\"]/text()'))\n            description = ', '.join(element.xpath('.//div[@class=\"css-9446fg eu4oa1w0\"]/ul//li/text()'))\n            company_name = ''.join(element.xpath('.//span[@data-testid=\"company-name\"]/text()'))\n\n            # Append extracted data to the jobs_data list\n            jobs_data.append({\n                'Title': title,\n                'Link': f\"https://www.indeed.com{link}\",\n                'Location': location,\n                'Description': description,\n                'Company': company_name\n            })\n    \n    return jobs_data\n\n

Step 2: Scraping job listings

Now that we have both the browser automation and parsing steps set up, let’s combine them to scrape job listings from the Indeed page.

Explanation:

get_page_content(url): Fetches the page content using Playwright.
parse_job_listings(content): Parses the content using lxml and extracts job data.
main(): Orchestrates the scraping process, fetching data and saving it to a CSV file.

\nimport pandas as pd\n\nasync def scrape_indeed_jobs(url):\n    # Step 1: Get page content using Playwright\n    content = await get_page_content(url)\n    \n    # Step 2: Parse the HTML and extract job details\n    jobs_data = parse_job_listings(content)\n    \n    return jobs_data\n\n# URL to scrape\nurl = 'https://www.indeed.com/q-usa-jobs.html'\n\n# Scraping and saving data\nasync def main():\n    # Scrape job data from the specified URL\n    jobs = await scrape_indeed_jobs(url)\n    \n    # Step 3: Save data to CSV using pandas\n    df = pd.DataFrame(jobs)\n    df.to_csv('indeed_jobs.csv', index=False)\n    \n    print(\"Data saved to indeed_jobs.csv\")\n\n# Run the main function\nasyncio.run(main())\n\n

Step 3: Adding pagination support

Indeed paginates its job listings, and you can easily extend the scraper to handle multiple pages. The page URL is adjusted using a query parameter start, which increments by 10 for each new page.

To enhance your scraper's functionality for collecting data from multiple pages, you can implement a function called scrape_multiple_pages. This function will modify the base URL by incrementally adjusting the start parameter, enabling access to subsequent pages. By systematically progressing through each page, you can expand the scope and quantity of data collected, such as vacancies, ensuring a more comprehensive dataset.

\nasync def scrape_multiple_pages(base_url, pages=3):\n    all_jobs = []\n    \n    for page_num in range(pages):\n        # Update URL for pagination\n        url = f\"{base_url}&start={page_num * 10}\"\n        print(f\"Scraping page: {url}\")\n        \n        # Scrape job data from each page\n        jobs = await scrape_indeed_jobs(url)\n        all_jobs.extend(jobs)\n    \n    # Save all jobs to CSV\n    df = pd.DataFrame(all_jobs)\n    df.to_csv('indeed_jobs_all_pages.csv', index=False)\n    print(\"Data saved to indeed_jobs_all_pages.csv\")\n\n# Scrape multiple pages of job listings\nasyncio.run(scrape_multiple_pages('https://www.indeed.com/jobs?q=usa', pages=3))\n\n

Step 4: Customizing job search queries

To target specific job titles or keywords in your scraping efforts, you'll need to configure the query search parameter in the URL used by Indeed. This customization allows the scraper to collect data specific to particular jobs or sectors. For instance, if you're searching for Python developer positions on http://www.indeed.com, you would adjust the query parameter to include “Python+developer” or relevant keywords.

\nquery = \"python+developer\"\nbase_url = f\"https://www.indeed.com/jobs?q={query}\"\nasyncio.run(scrape_multiple_pages(base_url, pages=3))\n\n

By modifying this parameter according to your data collection needs, you can focus your scraping on specific jobs, enhancing the flexibility and efficiency of your data collection process. This approach is especially useful for adapting to the dynamic demands of the job market.

Complete code

\nimport asyncio\nfrom playwright.async_api import async_playwright\nfrom lxml import html\nimport pandas as pd\n\n# Step 1: Fetch page content using Playwright\nasync def get_page_content(url):\n    async with async_playwright() as p:\n        browser = await p.chromium.launch(\n            headless=False\n            proxy = {\n                'server': '',\n                'username': '',\n                'password': ''\n            }\n        )  # Run browser in headed mode\n        page = await browser.new_page()\n        await page.goto(url, wait_until='networkidle')\n        \n        # Extract page content\n        content = await page.content()\n        await browser.close()  # Close browser after use\n        return content\n\n# Step 2: Parse the HTML content using lxml\ndef parse_job_listings(content):\n    # Parse the HTML using lxml\n    parser = html.fromstring(content)\n    \n    # Select individual job postings using XPath\n    job_posting = parser.xpath('//ul[@class=\"css-zu9cdh eu4oa1w0\"]/li')\n    \n    # Extract job data\n    jobs_data = []\n    for element in job_posting[:-1]:\n        title = ''.join(element.xpath('.//h2/a/span/@title'))\n        if title:\n            link = ''.join(element.xpath('.//h2/a/@href'))\n            location = ''.join(element.xpath('.//div[@data-testid=\"text-location\"]/text()'))\n            description = ', '.join(element.xpath('.//div[@class=\"css-9446fg eu4oa1w0\"]/ul//li/text()'))\n            company_name = ''.join(element.xpath('.//span[@data-testid=\"company-name\"]/text()'))\n\n            # Append extracted data to the jobs_data list\n            jobs_data.append({\n                'Title': title,\n                'Link': f\"https://www.indeed.com{link}\",\n                'Location': location,\n                'Description': description,\n                'Company': company_name\n            })\n    \n    return jobs_data\n\n# Step 3: Scrape Indeed jobs for a single page\nasync def scrape_indeed_jobs(url):\n    # Get page content using Playwright\n    content = await get_page_content(url)\n    \n    # Parse HTML and extract job data\n    jobs_data = parse_job_listings(content)\n    \n    return jobs_data\n\n# Step 4: Handle pagination and scrape multiple pages\nasync def scrape_multiple_pages(base_url, query, pages=3):\n    all_jobs = []\n    \n    for page_num in range(pages):\n        # Update the URL to handle pagination and add the search query\n        url = f\"{base_url}?q={query}&start={page_num * 10}\"\n        print(f\"Scraping page: {url}\")\n        \n        # Scrape jobs for the current page\n        jobs = await scrape_indeed_jobs(url)\n        all_jobs.extend(jobs)\n    \n    # Save all jobs to a CSV file\n    df = pd.DataFrame(all_jobs)\n    df.to_csv(f'indeed_jobs_{query}.csv', index=False)\n    print(f\"Data saved to indeed_jobs_{query}.csv\")\n\n# Function to run the scraper with dynamic query input\nasync def run_scraper():\n    # Step 5: Ask user for input query and number of pages to scrape\n    query = input(\"Enter the job title or keywords to search (e.g., python+developer): \")\n    pages = int(input(\"Enter the number of pages to scrape: \"))\n    \n    # Scrape jobs across multiple pages based on the query\n    base_url = 'https://www.indeed.com/jobs'\n    await scrape_multiple_pages(base_url, query, pages)\n\n# Run the scraper\nasyncio.run(run_scraper())\n\n

To ensure a smooth scraping process and reduce the risk of blocks and CAPTCHA appearances, it's crucial to choose the right proxy server. The most optimal option for scraping are ISP proxies, which provide high speed and connection stability, as well as a high trust factor, making them rarely blocked by platforms. This type of proxy is static, so for large-scale scraping, it's necessary to create a pool of ISP proxies and configure IP rotation for their regular change. An alternative option would be residential proxies, which are dynamic and have the broadest geographic coverage compared to other types of proxy servers.

","createdDate":"2025-01-15T14:23:31.000+00:00","description":"For job hunters, employers, or any individuals monitoring trends in the job market, scraping Indeed's list of available jobs could provide helpful information.","countOfViews":7,"attachments":[],"mainImg":{"id":"6a249b741079162e2e8bd484","url":"https://proxy-seller.com/files/fs/image/6a249b741079162e2e8bd484_how-to-scrape-indeed-job-listings-using-python_mainImg.png","name":"how-to-scrape-indeed-job-listings-using-python_mainImg.png","size":39185,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"how-to-scrape-indeed-job-listings-using-python_img","metaTitle":"Scraping Data from Indeed Using Python","metaDescription":"This guide demonstrates the process of scraping job listings from Indeed using Playwright and lxml tools, as well as utilizing proxies for anonymity and data access."},{"id":"6a049d1a35baad34714b0116","tag":"guide-to-scraping-dynamic-websites-with-python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":6,"title":"Guide to scraping dynamic websites with Python","body":"

An important ability for obtaining data from web pages is web scraping. Pinterest and Instagram, which dynamically load content through user interaction with them are examples of these types of websites. Regular scraping methods are insufficient when handling JavaScript-based material. In this article, we will elaborate on Playwright as the automation tool while lxml will be used for data extraction from such dynamic sites that require Javascript to work properly. On this note, we may discuss utilizing proxies in Playwright to evade detection as bots. In this tutorial, we'll scrape the Instagram profile to retrieve all the post URLs by simulating user behavior, such as scrolling and waiting for posts to load.

Tools we’ll use in this guide:

Playwright (for browser automation);
lxml (for data extraction using XPath);
Python (as our programming language).

Step-by-step guide to scraping Instagram posts

We will illustrate the process using the example of scraping an Instagram profile to extract post URLs, simulating user actions such as scrolling through the page and waiting for new data to load. Dynamic websites asynchronously load their content via AJAX requests, which means not all page content is immediately accessible.

Step 1. Install required libraries

Before we start, install the necessary packages:

\npip install playwright\npip install lxml\n\n

You'll also need to install Playwright browsers:

\nplaywright install\n\n

Step 2. Playwright setup for dynamic website scraping

We'll use Playwright to automate the browser, load Instagram's dynamic content, and scroll through the page to load more posts. Let's create a basic automation script:

Automation script (Headless Browser):

\nimport asyncio\nfrom playwright.async_api import async_playwright\n\nasync def scrape_instagram():\n    async with async_playwright() as p:\n        browser = await p.chromium.launch(headless=True)  # Headless mode No visual feedback\n        page = await browser.new_page()\n        \n        # Visit the profile URL\n        await page.goto(\"https://www.instagram.com/profile name/\", wait_until=\"networkidle\")\n\n        # Click the button to load more posts \n        await page.get_by_role(\"button\", name=\"Show more posts from\").click()\n        \n        # Scroll the page to load dynamic content\n        scroll_count = 5  # Customize this based on how many times you want to scroll\n        for _ in range(scroll_count):\n            await page.evaluate('window.scrollBy(0, 700);')\n            await page.wait_for_timeout(3000)  # Wait for posts to load\n            await page.wait_for_load_state(\"networkidle\")\n        \n        # Get the page content\n        content = await page.content()\n        await browser.close()\n        \n        return content\n\n# Run the asynchronous function\nasyncio.run(scrape_instagram())\n\n

Step 3. Parsing the page with lxml and XPath

Once the content is loaded, we can use lxml to parse the HTML and extract data using XPath. In this case, we're extracting the URLs of all posts from the profile.

Parsing the page content and extracting post URLs:

\nfrom lxml import html\nimport json\n\ndef extract_post_urls(page_content):\n    # Parse the HTML content using lxml\n    tree = html.fromstring(page_content)\n    \n    # XPath for extracting post URLs\n    post_urls_xpath = '//a[contains(@href, \"/p/\")]/@href'\n    \n    # Extract URLs\n    post_urls = tree.xpath(post_urls_xpath)\n    \n    # Convert relative URLs to absolute\n    base_url = \"https://www.instagram.com\"\n    post_urls = [f\"{base_url}{url}\" for url in post_urls]\n    \n    return post_urls\n\n

Example function to save extracted data in JSON format:

\ndef save_data(profile_url, post_urls):\n    data = {profile_url: post_urls}\n    with open('instagram_posts.json', 'w') as json_file:\n        json.dump(data, json_file, indent=4)\n\n# Scrape and extract URLs\npage_content = asyncio.run(scrape_instagram())\npost_urls = extract_post_urls(page_content)\n\n# Save the extracted URLs in a JSON file\nsave_data(\"https://www.instagram.com/profile name/\", post_urls)\n\n

Step 4. Handling infinite scroll with Playwright

To scrape dynamic websites, you often need to simulate infinite scrolling. In our script, we scroll the page using JavaScript:

\n(window.scrollBy(0, 700))\n

And wait for new content to load using this command:

\n wait_for_load_state(\"networkidle\")\n

Step 5. Using proxies with Playwright

Instagram has strict rate limits and anti-bot measures. To avoid being blocked, you can use proxies to rotate IP addresses and distribute requests. Playwright makes it easy to integrate proxies into your scraping automation.

Implementing proxies in Playwright:

\nasync def scrape_with_proxy():\n    async with async_playwright() as p:\n        browser = await p.chromium.launch(\n            headless=False, \n            proxy={\"server\": \"http://your-proxy-server:port\"}\n        )\n        page = await browser.new_page()\n        await page.goto(\"https://www.instagram.com/profile name/\", wait_until=\"networkidle\")\n        # Continue scraping as before...\n\n

Playwright also supports proxy to be passed as username password and server example is given below.

\nasync def scrape_with_proxy():\n    async with async_playwright() as p:\n        browser = await p.chromium.launch(\n            headless=False, \n            proxy={\"server\": \"http://your-proxy-server:port\", \"username\": \"username\", \"password\": \"password\"}\n        )\n        page = await browser.new_page()\n        await page.goto(\"https://www.instagram.com/profile name/\", wait_until=\"networkidle\")\n        # Continue scraping as before...\n\n

Proxies help avoid IP bans, CAPTCHA challenges, and ensure smooth scraping of data-heavy or restricted websites like Instagram.

Complete code

\nimport asyncio\nfrom playwright.async_api import async_playwright\nfrom lxml import html\nimport json\n\n# Function to automate browser and scrape dynamic content with proxies\nasync def scrape_instagram(profile_url, proxy=None):\n    async with async_playwright() as p:\n        # Set up browser with proxy if provided\n        browser_options = {\n            'headless': True,  # Use headed browser to see the action (can set to True for headless mode)\n        }\n        if proxy:\n            browser_options['proxy'] = proxy\n\n        # Launch the browser\n        browser = await p.chromium.launch(**browser_options)\n        page = await browser.new_page()\n\n        # Visit the Instagram profile page\n        await page.goto(profile_url, wait_until=\"networkidle\")\n        \n        # Try clicking the \"Show more posts\" button (optional, might fail if button not found)\n        try:\n           await page.click('button:has-text(\"Show more posts from\")')\n        except Exception as e:\n           print(f\"No 'Show more posts' button found: {e}\")\n\n\n        # Scroll the page to load more posts\n        scroll_count = 5  # Number of scrolls to load posts\n        for _ in range(scroll_count):\n            await page.evaluate('window.scrollBy(0, 500);')\n            await page.wait_for_timeout(3000)  # Wait for new posts to load\n            await page.wait_for_load_state(\"networkidle\")\n\n        # Get the complete page content after scrolling\n        content = await page.content()\n        await browser.close()  # Close the browser once done\n        \n        return content\n\n# Function to parse the scraped page content and extract post URLs\ndef extract_post_urls(page_content):\n    # Parse the HTML content using lxml\n    tree = html.fromstring(page_content)\n    \n    # XPath for extracting post URLs\n    post_urls_xpath = '//a[contains(@href, \"/p/\")]/@href'\n    \n    # Extract post URLs using the XPath\n    post_urls = tree.xpath(post_urls_xpath)\n    \n    # Convert relative URLs to absolute URLs\n    base_url = \"https://www.instagram.com\"\n    post_urls = [f\"{base_url}{url}\" for url in post_urls]\n    \n    return post_urls\n\n# Function to save the extracted post URLs into a JSON file\ndef save_data(profile_url, post_urls):\n    # Structure the data in JSON format\n    data = {profile_url: post_urls}\n    \n    # Save the data to a file\n    with open('instagram_posts.json', 'w') as json_file:\n        json.dump(data, json_file, indent=4)\n    print(f\"Data saved to instagram_posts.json\")\n\n# Main function to run the scraper and save the data\nasync def main():\n    # Define the Instagram profile URL\n    profile_url = \"https://www.instagram.com/profile name/\"\n    \n    # Optionally, set up a proxy\n    proxy = {\"server\": \"server\", \"username\": \"username\", \"password\": \"password\"}  # Use None if no proxy is required\n    \n    # Scrape the Instagram page with proxies\n    page_content = await scrape_instagram(profile_url, proxy=proxy)\n    \n    # Extract post URLs from the scraped page content\n    post_urls = extract_post_urls(page_content)\n    \n    # Save the extracted post URLs into a JSON file\n    save_data(profile_url, post_urls)\n\nif __name__ == '__main__':\n   asyncio.run(main())\n\n

Alternative automation tools for web scraping

While Playwright is an excellent choice for scraping dynamic websites, other tools might be suitable for different scenarios:

Selenium: Selenium is one of the oldest browser automation frameworks and works similarly to Playwright. It's highly versatile but lacks some modern capabilities that Playwright offers, such as handling multiple browsers with a single API;
Puppeteer: Puppeteer is another popular tool for browser automation, especially for scraping JavaScript-heavy websites. Like Playwright, it controls headless browsers and allows interaction with dynamic content;
Requests + BeautifulSoup: for simpler websites that don’t require JavaScript to load content, the Requests library combined with BeautifulSoup is a lightweight alternative. However, it doesn’t handle dynamic content well.

Each tool offers unique strengths and can be chosen based on the specific needs and conditions of the project.

For successful scraping of dynamic websites that actively use JavaScript and AJAX requests, powerful tools capable of efficiently handling infinite scrolling and complex interactive elements are necessary. One such solution is Playwright—a tool from Microsoft that provides full browser automation, making it an ideal choice for platforms like Instagram. Combined with the lxml library for HTML parsing, Playwright greatly simplifies data extraction, allowing for the automation of interactions with page elements and the parsing process without manual intervention. Additionally, the use of proxy servers helps circumvent anti-bot protection and prevents IP blocking, ensuring stable and uninterrupted scraping operations.

","createdDate":"2025-01-10T14:32:11.000+00:00","description":"An important ability for obtaining data from web pages is web scraping.","countOfViews":8,"attachments":[],"mainImg":{"id":"6a249bab1079162e2e8bdd8d","url":"https://proxy-seller.com/files/fs/image/6a249bab1079162e2e8bdd8d_guide-to-scraping-dynamic-websites-with-python_mainImg.png","name":"guide-to-scraping-dynamic-websites-with-python_mainImg.png","size":45417,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"guide-to-scraping-dynamic-websites-with-python_img","metaTitle":"Scraping Data from Dynamic Websites Using Python","metaDescription":"In this article, we will closely examine the process of extracting data from dynamically updated pages, such as Instagram profiles, using Python."},{"id":"6a049d1d35baad34714b0187","tag":"guide_to_scraping_linkedin_data_with_python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":4,"title":"Guide to scraping LinkedIn data with Python","body":"

Scraping LinkedIn data can be incredibly valuable for several reasons:

Job market analysis: analyze trends in job listings, such as the most in-demand skills and industries;
Recruitment: gather data on job postings to inform hiring strategies;
Competitor research: monitor hiring patterns and strategies of competitors.

The article will emphasize important techniques and strategies, including the importance of avoiding detection via proxies and headers in the first place. The requests library will be used for making HTTP requests while lxml will be employed to parse HTML content.

Setting up the environment

Before you begin, make sure you have Python installed on your machine.

Install the required libraries using pip:

\npip install requests\npip install lxml\n\n

Getting started with the scraper

Here’s a comprehensive code example for scraping LinkedIn job listings using Python:

Import libraries

We'll need several Python libraries:

requests: For making HTTP requests to retrieve web pages.
lxml: For parsing HTML content.
csv: For writing the extracted data to a CSV file.

\nimport requests\nfrom lxml import html\nimport csv\nimport random\n\n

Define job search URL

Start by defining the LinkedIn job search URL that you want to scrape.

\nurl = 'https link'\n\n

User-agent strings and proxies

To scrape LinkedIn effectively, it’s crucial to use the correct headers, especially the User-Agent header, to mimic requests from an actual browser.

\nuser_agents = [\n    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',\n    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',\n    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'\n]\n\n

Modern-day proxy providers often support internal rotation, meaning they automatically rotate IP addresses for you. This eliminates the need to manually select proxies from a list. However, for illustrative purposes, here’s how1 you would handle proxy rotation if needed:

\nproxies = {\n    'http': random.choice(proxies),\n    'https': random.choice(proxies)\n}\n\n

Headers for requests

Successful LinkedIn scraping hinges on the correct setup of headers that emulate the behavior of a real browser. Properly configured headers not only aid in circumventing anti-bot protection systems but also diminish the chances of your scraping activities being blocked.

\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'dnt': '1',\n    'sec-ch-ua': '\"Not/A)Brand\";v=\"8\", \"Chromium\";v=\"126\", \"Google Chrome\";v=\"126\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n}\n\n

Initialize Data Storage

To effectively collect and store job posting information, you should start by initializing a data store. In Python, this typically involves creating an empty list. This list will serve as a storage container where you can add job details as they are extracted from the HTML content. This method ensures that the information is systematically collected and easily accessible for further processing or analysis.

job_details = []\n\n

Parsing the HTML content

After sending an HTTP GET request, the next step is to parse the HTML content using the lxml library. This will allow us to navigate through the HTML structure and identify the data we want to extract.

\n# Set random User-Agent and proxy with IP authorization method\nheaders['user-agent'] = random.choice(user_agents)\nproxies = {\n    'http': IP:PORT,\n    'https': IP:PORT\n}\n\n# Send an HTTP GET request to the URL\nresponse = requests.get(url=url, headers=headers, proxies=proxies)\nparser = html.fromstring(response.content)\n\n

Extracting job data

Once the HTML content is parsed, we can extract specific job details such as title, company name, location, and job URL using XPath queries. These details are stored in a dictionary and appended to a list.

\n# Extract job details from the HTML content\nfor job in parser.xpath('//ul[@class=\"jobs-search__results-list\"]/li'):\n    title = ''.join(job.xpath('.//div/a/span/text()')).strip()\n    company = ''.join(job.xpath('.//div/div[2]/h4/a/text()')).strip()\n    location = ''.join(job.xpath('.//div/div[2]/div/span/text()')).strip()\n    job_url = job.xpath('.//div/a/@href')[0]\n    \n    job_detail = {\n        'title': title,\n        'company': company,\n        'location': location,\n        'job_url': job_url\n    }\n    job_details.append(job_detail)\n\n

Save data to CSV

After collecting the job data, save it to a CSV file.

\nwith open('linkedin_jobs.csv', 'w', newline='', encoding='utf-8') as csvfile:\n    fieldnames = ['title', 'company', 'location', 'job_url']\n    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n    writer.writeheader()\n    for job_detail in job_details:\n        writer.writerow(job_detail)\n\n

Complete code

Here is the complete code, combining all the sections above:

\nfrom selenium.webdriver.chrome.options import Options\nfrom seleniumwire import webdriver as wiredriver\nfrom selenium.webdriver.common.by import By\nimport csv\n\n# Specify the proxy server address with username and password\nproxy_address = \"\"\nproxy_username = \"\"\nproxy_password = \"\"\n\n# Set up Chrome options with the proxy and authentication\nchrome_options = Options()\nchrome_options.add_argument(f'--proxy-server={proxy_address}')\nchrome_options.add_argument(f'--proxy-auth={proxy_username}:{proxy_password}')\n\n# Create a WebDriver instance with selenium-wire\ndriver = wiredriver.Chrome(options=chrome_options)\n\nurl = 'https link'\n\n# Perform your Selenium automation with the enhanced capabilities of selenium-wire\ndriver.get(url)\n\njob_details = []\n\nall_elements = driver.find_elements(By.XPATH,\n                                   '//*[@id=\"main-content\"]/section/ul/li')\n\nfor i in all_elements:\n   title = i.find_element(By.XPATH,\n                          './/div/div/h3').text\n   company = i.find_element(By.XPATH, './/div/div[2]/h4/a').text\n   location = i.find_element(By.XPATH, './/div/div[2]/div/span').text\n   job_url = i.find_element(By.XPATH,\n                            './/div/a').get_attribute('href')\n\n   job_detail = {\n       'title': title,\n       'company': company,\n       'location': location,\n       'job_url': job_url\n   }\n   job_details.append(job_detail)\n\nwith open('linkedin_jobs.csv', 'w', newline='', encoding='utf-8') as csvfile:\n   fieldnames = ['title', 'company', 'location', 'job_url']\n   writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n   writer.writeheader()\n   for job_detail in job_details:\n       writer.writerow(job_detail)\n\ndriver.quit()\n\n

Extracting data from LinkedIn using Python with the requests and lxml libraries offers a powerful way to analyze the job market and recruit personnel. To ensure a smooth scraping process, datacenter proxies with high speeds are utilized, as well as ISP proxies with a higher trust factor, which reduces the risk of blocks on automated actions.

","createdDate":"2025-01-06T09:15:00.000+00:00","description":"The article will emphasize important techniques and strategies, including the importance of avoiding detection via proxies and headers in the first place.","countOfViews":8,"attachments":[],"mainImg":{"id":"6a249bd41079162e2e8be3e9","url":"https://proxy-seller.com/files/fs/image/6a249bd41079162e2e8be3e9_guide_to_scraping_linkedin_data_with_python_mainImg.png","name":"guide_to_scraping_linkedin_data_with_python_mainImg.png","size":39790,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"guide_to_scraping_linkedin_data_with_python_img","metaTitle":"Step-by-Step Guide to Scraping LinkedIn Data with Python","metaDescription":"This guide covers the process of collecting job-related data on LinkedIn using Python, which includes the extraction of job titles, company information, and their addresses."},{"id":"6a049d2335baad34714b01b1","tag":"guide_to_scraping_yahoo_finance_data_with_python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":9,"title":"Guide to Scraping Yahoo Finance Data with Python","body":"

This guide demonstrates how to scrape data from Yahoo Finance using Python, employing the requests and lxml libraries. Yahoo Finance offers extensive financial data, such as stock prices and market trends, which are pivotal for real-time market analysis, financial modeling, and crafting automated investment strategies.

The procedure entails sending HTTP requests to retrieve the webpage content, parsing the HTML received, and extracting specific data using XPath expressions. This approach enables efficient and targeted data extraction, allowing users to access and utilize financial information dynamically.

Tools and Libraries

We'll be using the following Python libraries:

requests: To send HTTP requests to the Yahoo Finance website.
lxml: To parse the HTML content and extract data using XPath.

Before you begin, ensure you have these libraries installed:

pip install requests\npip install  lxml

How to Construct Scraping Requests Efficiently

To succeed at web scraping Yahoo Finance Python data, craft your requests carefully. Start with a complete Yahoo Finance URL. Include query parameters for the stock symbols you want; for example, https://finance.yahoo.com/quote/AMZN or https://finance.yahoo.com/quote/AAPL for Amazon and Apple data.

Set proper HTTP headers to mimic a real user’s browser. Include:

User-Agent: a recent browser signature such as “Mozilla/5.0…”
Accept-Language: to specify preferred language like “en-US”

When using scraping tools like Scrapingdog API, leverage its parameters fully:

url= to set the target page
autoparse=1 to get structured data automatically
premium_proxy=1 to enable better proxy routing
js_render=1 to handle JavaScript-loaded content

For example, a request URL can look like this:

https://api.scrapingdog.com/scrape?url=https://finance.yahoo.com/quote/AMZN&autoparse=1&...

Also, apply rate limiting in your code. Pause between requests to avoid detection and server overload. You can integrate scheduling tools like cron jobs or Python schedulers to run your scraping at controlled intervals, ensuring consistent and safe data collection.

To sum up, efficient scraping requests require:

accurate and full URLs with stock symbols;
realistic and complete headers;
use of API features for proxies and JavaScript rendering;
thoughtful timing with rate limiting and scheduled jobs.

The Process of Web Scraping with Python Explained

Below, we will explore the parsing process in a step-by-step manner, complete with code examples for clarity and ease of understanding.

Step 1: Sending a Request

The first step in web scraping is sending an HTTP request to the target URL. We will use the requests library to do this. It's crucial to include proper headers in the request to mimic a real browser, which helps in bypassing basic anti-bot measures.

import requests\nfrom lxml import html\n\n# Target URL\nurl = \"https://finance.yahoo.com/quote/AMZN/\"\n\n# Headers to mimic a real browser\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'priority': 'u=0, i',\n    'sec-ch-ua': '\"Not)A;Brand\";v=\"99\", \"Google Chrome\";v=\"127\", \"Chromium\";v=\"127\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36',\n}\n\n# Send the HTTP request\nresponse = requests.get(url, headers=headers)

Step 2: Extracting Data with XPath

After receiving the HTML content, the next step is to extract the desired data using XPath. XPath is a powerful query language for selecting nodes from an XML document, which is perfect for parsing HTML content.

Title and price:

$\"scraping.png\"$

More details:

$\"scraping2.png\"$

Below are the XPath expressions we'll use to extract different pieces of financial data:

# Parse the HTML content\nparser = html.fromstring(response.content)\n\n# Extracting data using XPath\ntitle = ' '.join(parser.xpath('//h1[@class=\"yf-3a2v0c\"]/text()'))\nlive_price = parser.xpath('//fin-streamer[@class=\"livePrice yf-mgkamr\"]/span/text()')[0]\ndate_time = parser.xpath('//div[@slot=\"marketTimeNotice\"]/span/text()')[0]\nopen_price = parser.xpath('//ul[@class=\"yf-tx3nkj\"]/li[2]/span[2]/fin-streamer/text()')[0]\nprevious_close = parser.xpath('//ul[@class=\"yf-tx3nkj\"]/li[1]/span[2]/fin-streamer/text()')[0]\ndays_range = parser.xpath('//ul[@class=\"yf-tx3nkj\"]/li[5]/span[2]/fin-streamer/text()')[0]\nweek_52_range = parser.xpath('//ul[@class=\"yf-tx3nkj\"]/li[6]/span[2]/fin-streamer/text()')[0]\nvolume = parser.xpath('//ul[@class=\"yf-tx3nkj\"]/li[7]/span[2]/fin-streamer/text()')[0]\navg_volume = parser.xpath('//ul[@class=\"yf-tx3nkj\"]/li[8]/span[2]/fin-streamer/text()')[0]\n\n# Print the extracted data\nprint(f\"Title: {title}\")\nprint(f\"Live Price: {live_price}\")\nprint(f\"Date & Time: {date_time}\")\nprint(f\"Open Price: {open_price}\")\nprint(f\"Previous Close: {previous_close}\")\nprint(f\"Day's Range: {days_range}\")\nprint(f\"52 Week Range: {week_52_range}\")\nprint(f\"Volume: {volume}\")\nprint(f\"Avg. Volume: {avg_volume}\")

Step 3: Handling Proxies and Headers

Websites like Yahoo Finance often employ anti-bot measures to prevent automated scraping. To avoid getting blocked, you can use proxies and rotate headers.

Using proxies

A proxy server acts as an intermediary between your machine and the target website. It helps mask your IP address, making it harder for websites to detect that you're scraping.

# Example of using a proxy with IP authorization model\nproxies = {\n    \"http\": \"http://your.proxy.server:port\",\n    \"https\": \"https://your.proxy.server:port\"\n}\n\nresponse = requests.get(url, headers=headers, proxies=proxies)

Rotating user-agent headers

Rotating the User-Agent header is another effective way to avoid detection. You can use a list of common User-Agent strings and randomly select one for each request.

import random\n\nuser_agents = [\n    \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3\",\n    \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0\",\n    # Add more User-Agent strings here\n]\n\nheaders[\"user-agent\"]: random.choice(user_agents)\n\nresponse = requests.get(url, headers=headers)

Step 4: Saving Data to a CSV File

Finally, you can save the scraped data into a CSV file for later use. This is particularly useful for storing large datasets or analyzing the data offline.

import csv\n\n# Data to be saved\ndata = [\n    [\"URL\", \"Title\", \"Live Price\", \"Date & Time\", \"Open Price\", \"Previous Close\", \"Day's Range\", \"52 Week Range\", \"Volume\", \"Avg. Volume\"],\n    [url, title, live_price, date_time, open_price, previous_close, days_range, week_52_range, volume, avg_volume]\n]\n\n# Save to CSV file\nwith open(\"yahoo_finance_data.csv\", \"w\", newline=\"\") as file:\n    writer = csv.writer(file)\n    writer.writerows(data)\n\nprint(\"Data saved to yahoo_finance_data.csv\")

Complete Code

Below is the complete Python script that integrates all the steps we’ve discussed (a full walkthrough about running Python scripts on Windows, check out this tutorial). This includes sending requests with headers, using proxies, extracting data with XPath, and saving the data to a CSV file.

import requests\nfrom lxml import html\nimport random\nimport csv\n\n# Example URL to scrape\nurl = \"https://finance.yahoo.com/quote/AMZN/\"\n\n# List of User-Agent strings for rotating headers\nuser_agents = [\n    \"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3\",\n    \"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0\",\n    # Add more User-Agent strings here\n]\n\n# Headers to mimic a real browser\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'priority': 'u=0, i',\n    'sec-ch-ua': '\"Not)A;Brand\";v=\"99\", \"Google Chrome\";v=\"127\", \"Chromium\";v=\"127\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'User-agent': random.choice(user_agents),\n}\n\n# Example of using a proxy\nproxies = {\n    \"http\": \"http://your.proxy.server:port\",\n    \"https\": \"https://your.proxy.server:port\"\n}\n\n# Send the HTTP request with headers and optional proxies\nresponse = requests.get(url, headers=headers, proxies=proxies)\n\n# Check if the request was successful\nif response.status_code == 200:\n    # Parse the HTML content\n    parser = html.fromstring(response.content)\n\n    # Extract data using XPath\n    title = ' '.join(parser.xpath('//h1[@class=\"yf-3a2v0c\"]/text()'))\n    live_price = parser.xpath('//fin-streamer[@class=\"livePrice yf-mgkamr\"]/span/text()')[0]\n    date_time = parser.xpath('//div[@slot=\"marketTimeNotice\"]/span/text()')[0]\n    open_price = parser.xpath('//ul[@class=\"yf-tx3nkj\"]/li[2]/span[2]/fin-streamer/text()')[0]\n    previous_close = parser.xpath('//ul[@class=\"yf-tx3nkj\"]/li[1]/span[2]/fin-streamer/text()')[0]\n    days_range = parser.xpath('//ul[@class=\"yf-tx3nkj\"]/li[5]/span[2]/fin-streamer/text()')[0]\n    week_52_range = parser.xpath('//ul[@class=\"yf-tx3nkj\"]/li[6]/span[2]/fin-streamer/text()')[0]\n    volume = parser.xpath('//ul[@class=\"yf-tx3nkj\"]/li[7]/span[2]/fin-streamer/text()')[0]\n    avg_volume = parser.xpath('//ul[@class=\"yf-tx3nkj\"]/li[8]/span[2]/fin-streamer/text()')[0]\n\n    # Print the extracted data\n    print(f\"Title: {title}\")\n    print(f\"Live Price: {live_price}\")\n    print(f\"Date & Time: {date_time}\")\n    print(f\"Open Price: {open_price}\")\n    print(f\"Previous Close: {previous_close}\")\n    print(f\"Day's Range: {days_range}\")\n    print(f\"52 Week Range: {week_52_range}\")\n    print(f\"Volume: {volume}\")\n    print(f\"Avg. Volume: {avg_volume}\")\n\n    # Save the data to a CSV file\n    data = [\n        [\"URL\", \"Title\", \"Live Price\", \"Date & Time\", \"Open Price\", \"Previous Close\", \"Day's Range\", \"52 Week Range\", \"Volume\", \"Avg. Volume\"],\n        [url, title, live_price, date_time, open_price, previous_close, days_range, week_52_range, volume, avg_volume]\n    ]\n\n    with open(\"yahoo_finance_data.csv\", \"w\", newline=\"\") as file:\n        writer = csv.writer(file)\n        writer.writerows(data)\n\n    print(\"Data saved to yahoo_finance_data.csv\")\nelse:\n    print(f\"Failed to retrieve data. Status code: {response.status_code}\")

Legal and Ethical Considerations When Scraping Yahoo Finance

Before you start web scraping Yahoo Finance Python data, you must consider the legal and ethical boundaries to avoid trouble.

First, review Yahoo Finance’s Terms of Service carefully. Yahoo does not allow unauthorized mass extraction or redistribution of their data for commercial purposes. Accessing data for personal use or academic research is generally acceptable but still requires responsible behavior.
Respecting Yahoo Finance’s robots.txt file is crucial. This file tells you which parts of the site are off-limits to automated scraping tools. Ignoring robots.txt can lead to your IP being blocked or legal warnings. Also, avoid sending too many requests too quickly. Overloading the server harms the website’s functioning and violates ethical scraping practices.
Use the data you scrape responsibly. Focus on personal projects or academic work, and don’t use the data for unauthorized commercial gain. This keeps you on the right side of the law and maintains trust in the scraping community.

To scrape ethically, follow these best practices:

Implement rate limiting: pause between requests to avoid overwhelming the server.
Identify your scraper clearly by setting a user-agent header that states your script’s name and contact info.
Avoid excessive requests: scrape only what you need and minimize hits to the same pages.
Monitor your scraping behavior constantly and adjust if you hit limits or face blocks.

By following these steps, your web scraping Yahoo Finance with Python will be both effective and respectful, minimizing risks and keeping your projects safe.

Common Challenges and How to Overcome Them When Scraping Yahoo Finance

You’ll face some common hurdles when web scraping Python Yahoo Finance data. The main issues are IP blocking, CAPTCHAs, and dynamic content loading via JavaScript, all designed to prevent automated access.

Dealing with Anti-Bot Measures

IP blocking happens when sites detect suspicious traffic from one address and block it. CAPTCHAs force human verification, stopping most bots. Yahoo Finance also loads some data dynamically with JavaScript, which requires special handling since static HTML scraping won’t capture it.

To solve these problems, use proxies and specialized tools. Proxies rotate your IP address to avoid blocks and CAPTCHAs. For example, the Scrapingdog API manages proxy rotation automatically and can bypass CAPTCHAs, making your scraper more reliable.

Handling HTML Structure Changes

Yahoo Finance updates its HTML structure often, breaking fixed scraping code. To handle this, build adaptive parsing logic that can adjust to changes without crashing. Use selectors that are less likely to change, like unique element IDs or classes.

Debugging Effectively

When your script fails, debug efficiently by:

Logging every request and response with timestamps.
Inspecting HTTP status codes for errors.
Catching and handling exceptions gracefully to avoid crashes.

Expert Proxy Help for Robust Scraping

Proxy-Seller offers a robust proxy solution perfect for web scraping Yahoo Finance Python tasks. They provide private SOCKS5 and HTTP(S) proxies, ensuring fast and stable connections. Their proxy pool covers over 800 subnets across 400 networks, supports both IPv4 and IPv6, and includes geo-targeting in 220+ countries.

Using Proxy-Seller’s proxies helps you:

Avoid IP bans by rotating addresses.
Reduce CAPTCHA challenges.
Maintain stable connections during heavy scraping.

They also back you up with a 24-hour refund and replacement policy, plus assistance configuring proxies for smooth integration. This makes your scraping project more reliable and less risky.

In Conclusion

Scraping Yahoo Finance data using Python is a powerful way to automate the collection of financial data. By using the requests and lxml libraries, along with proper headers, proxies, and anti-bot measures, you can efficiently scrape and store stock data for analysis. This guide covered the basics, but remember to adhere to legal and ethical guidelines when scraping websites. And if you need professional assistance, don't hesitate to contact 24/7 customer support.

","createdDate":"2024-12-20T14:47:01.000+00:00","description":"This guide demonstrates how to scrape data from Yahoo Finance using Python, employing the requests and lxml libraries.","countOfViews":9,"attachments":[{"id":"6a249c0e1079162e2e8beea6","url":"https://proxy-seller.com/files/fs/image/6a249c0e1079162e2e8beea6_omy52ctli3u0pp2en1sjy9x379z9aav2.png","name":"omy52ctli3u0pp2en1sjy9x379z9aav2.png","size":333140,"type":"image/png"},{"id":"6a249c0e1079162e2e8beea7","url":"https://proxy-seller.com/files/fs/image/6a249c0e1079162e2e8beea7_5s5z22s0cyxjjmm57kdg45jg93w15b86.png","name":"5s5z22s0cyxjjmm57kdg45jg93w15b86.png","size":294127,"type":"image/png"}],"mainImg":{"id":"6a249c0e1079162e2e8bee9e","url":"https://proxy-seller.com/files/fs/image/6a249c0e1079162e2e8bee9e_guide_to_scraping_yahoo_finance_data_with_python_mainImg.png","name":"guide_to_scraping_yahoo_finance_data_with_python_mainImg.png","size":43924,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"guide_to_scraping_yahoo_finance_data_with_python_img","metaTitle":"Scraping Data from Yahoo Finance Using Python","metaDescription":"Learn to build a Python script for Yahoo Finance data scraping and storage with our guide. Choose a perfect tool for market analysis and investment strategies."},{"id":"6a049d2735baad34714b01d6","tag":"best_web_scraping_tools_to_get_ahead_in_2025","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":17,"title":"Best Web Scraping Tools to Get Ahead in 2026","body":"

Web scraping tools are specialized software designed to automatically pull data from websites, organizing it into a usable format. These tools are essential for various tasks like data collection, digital archiving, and conducting in-depth analytics. With the ability to meticulously extract and analyze page data, advanced web scraping tools ensure the precision and relevance of the information they gather.

Their ability to handle large-scale data extraction makes them a critical resource for businesses engaged in competitor analysis, market research, and lead generation. These tools not only streamline processes but also provide significant competitive advantages by offering deep insights quickly.

In this article, we'll explore the top web scraping tools of 2026. We'll cover a range of options, including browser-based tools, programming frameworks, libraries, APIs, and software-as-a-service (SaaS) solutions.

Factors to Consider When Choosing Web Scraping Tools

When selecting a web scraping tool, there are several key factors to consider:

Scraping intervals: assess how often you need to extract data, as some tools offer real-time scraping while others are better suited for less frequent batch jobs.
Ease of use: look for tools with intuitive interfaces and clear documentation to ensure a smooth setup and operation.
Proxy support: ensure the tool manages proxies to avoid detection and IP blocking, which is vital for scraping at scale or from sites with anti-scraping measures.
Cost and effectiveness: balance your budget against the features offered; more expensive tools may provide advanced features that justify their cost.
Data input and export: choose a tool that can handle the types of data you're interested in and supports the formats you need for output (CSV, JSON, or direct database integration).
Data volume: consider tools that scale to handle the volume of data you plan to scrape, especially if dealing with large datasets or high-traffic sites.
Site complexity: evaluate the complexity of the target websites, as more complex sites with dynamic content may require more sophisticated tools like those capable of rendering JavaScript (additional information about disabling JavaScript).
Support and community: check if the tool has responsive customer support and an active user community for troubleshooting and advice.

The choice of a web scraping tool depends on the complexity of the task and the volume of data being processed. For simpler tasks, browser extensions are often sufficient. They are easy to install and do not require programming knowledge, making them a good choice for straightforward data collection tasks. For more complex and customizable solutions, frameworks are better suited as they offer more flexibility and control. If a high level of automation and management is required, API-oriented scrapers provide a fully managed service that can handle large volumes of data efficiently.

Top 19 Best Scrapers: Detailed Comparison

We have curated a list of the 19 best scrapers that cater to a variety of needs. This selection includes powerful programs designed for complex web scraping tasks, as well as universal tools that are user-friendly and do not require programming knowledge. Whether you're an experienced developer needing robust data extraction capabilities or a beginner looking to easily gather web data, this list has options to suit different levels of expertise and project demands.

You’ll learn how to choose the best web scraping tools 2026 by understanding core strengths, key features, pricing, and real-world use cases. Each tool suits different user needs, from business teams to AI experts. Here is a concise breakdown of the top tools.

Bright Data

Bright Data offers a robust, enterprise-grade web scraping platform that includes a Web Scraper IDE with ready-made code templates. These templates are managed and updated on a regular basis, ensuring that scraping operations remain effective even if the layout of the target website changes.

$\"image2.png\"$

Bright Data also uses proxy rotation and allows you to save scraped data in various formats, such as JSON and CSV, or directly to cloud storage solutions such as Google Cloud Storage or Amazon S3.

Features:

Web Scraper IDE;
cloud-based hosting environment;
ready-to-use web scraping templates;
extensive proxy network;
unblocker infrastructure to bypass anti-bot measures;
scheduling options for automatic data extraction;
supports a wide range of output formats;
managed service with live support;
search engine crawler;
compliance with data protection laws.

The scraper is available starting at $4.00 per month, and it offers a free trial version for users to test its capabilities. It is well-regarded on G2, where it has a rating of 4.6 out of 5.0.

Octoparse

Octoparse is a no-code, easy-to-use web scraping tool that simplifies scraping tasks without requiring any coding skills. Designed for both seasoned and novice users, it offers a visual approach to data extraction, requiring minimal to no coding skills.

$\"image1.png\"$

One of the standout features of Octoparse is its AI assistant. This feature assists users by auto-detecting data patterns on websites and offering handy tips for effective data extraction. In addition, Octoparse offers a library of preset templates for popular websites, which can be used to obtain data instantly.

Features:

user-friendly interface;
AI-assisted data pattern detection;
pre-built templates for popular websites;
IP rotation and data extraction tips;
infinite scrolling;
scheduled scraping and automation.

The scraper starts at $75.00 per month and includes a free trial. It is rated 4.5/5.0 on Capterra and 4.3/5.0 on G2.

WebScraper.io

WebScraper.io is a Chrome and Firefox extension that is designed for regular and scheduled use to extract large amounts of data either manually or automatically.

It's free for local use, with a paid cloud service available for scheduling and managing scraping jobs through an API. This tool also supports scraping of dynamic websites and saves data in structured formats like CSV, XLSX, or JSON.

$\"image4.png\"$

WebScraper.io facilitates web scraping through a point-and-click interface, allowing users to create Site Maps and select elements without any coding expertise. It’s also versatile for use cases like market research, lead generation, and academic projects.

Features:

point and click interface;
modular selector system;
handles pagination;
can extract data from dynamic websites;
multiple data export options.

The scraper is priced at $50 per month and offers a free trial. It has a Capterra rating of 4.7 out of 5.

ScraperAPI

Getting started with Scraper API is easy for non-developers, as all users need is an API key and URL to begin scraping. Besides supporting JavaScript renderings, Scraper API is fully customizable, allowing users to customize the request and header parameters in order to meet their needs.

$\"image3.png\"$

Features:

handles JavaScript rendering for dynamic content;
manages CAPTCHAs and uses proxies to prevent detection;
provides options for customizing headers and cookies;
no-code API playground;
offers geotargeting capabilities for scraping location-specific content;
offers a free trial version that allows you to send up to 5,000 requests to test its features.

You should format your requests to the API endpoint as follows:

import requests\npayload = {'api_key': 'APIKEY', 'url': 'https://httpbin.org/ip'}\nr = requests.get('http://api.scraperapi.com', params=payload)\nprint(r.text)

This scraper is available at an introductory price of $49 per month and comes with a free trial. It has a Capterra rating of 4.6 out of 5 and a G2 rating of 4.3 out of 5.

Scraping Dog

Scraping Dog stands out for its simplicity and ease of use, providing an API that can be quickly integrated into various applications and workflows. It's a solution that serves a broad spectrum of scraping requirements, from simple data collection tasks to more complex operations.

$\"image6.png\"$

Scrapingdog also supports JS rendering, which can be used for scraping websites that require multiple API calls to fully load.

Features:

proxy support including IP rotation for improved anonymity;
handles JavaScript-heavy websites;
offers a range of pricing plans;
webhooks.

Here's a basic example of how to use Scraping Dog’s API endpoint:

import requests

import requests\n\nurl = \"https://api.scrapingdog.com/scrape\"\n\nparams = {\n    \"api_key\": \"5e5a97e5b1ca5b194f42da86\",\n    \"url\": \"http://httpbin.org/ip\",\n    \"dynamic\": \"false\"\n}\n\nresponse = requests.get(url, params=params)\n\nprint(response.text)

The scraper is available starting at $30 per month and includes a free trial. It has a Trustpilot rating of 4.6 out of 5.

APIfy

Apify is an open software platform that makes it easy to develop and run data extraction, web automation, and web integration tools at scale. It is a versatile cloud-based platform that provides a comprehensive suite of web scraping and automation tools. It's designed for developers who need to build, run, and scale web scraping and data extraction tasks without managing servers.

$\"image5.png\"$

Apify also comes with an open-source web scraping library called Crawlee and is compatible both with Python and JavaScript. With Apify, you can integrate your content easily with third-party applications such as Google Drive, GitHub, and Slack, as well as create your own integrations with webhooks and APIs.

Features:

scale using a pool of proxies for reliable data collection;
access to a complete API for integration and automation;
host code anywhere;
cloud-based data storage and management;
pre-built scrapers for popular websites;
scheduling options for extraction tasks;
support for multiple data export formats.

The scraper starts at $49 per month and includes a free version. It has a rating of 4.8 out of 5 on both Capterra and G2.

ScrapingBee

ScrapingBee is a versatile web scraping API that's crafted to handle a wide range of web scraping tasks efficiently. It excels in areas such as real estate scraping, price monitoring, and review extraction, allowing users to gather data seamlessly without the fear of being blocked.

$\"image8.png\"$

The flexibility and effectiveness of ScrapingBee make it an invaluable resource for developers, marketers, and researchers who aim to automate and streamline the data collection process from various online sources.

Features:

handles JavaScript rendering;
manages CAPTCHAs, ensuring uninterrupted scraping operations;
IP rotation;
scraping search engine results pages;
direct API access for easy integration with your existing systems.

This scraper is available starting at $49 per month and includes a free version. It boasts a perfect rating of 5.0 out of 5 on Capterra.

Diffbot

Diffbot stands out with its advanced AI and machine learning capabilities, making it highly effective for content extraction from web pages. It's a fully automated solution that is great at extracting structured data.

$\"image7.png\"$

Diffbot is ideal for marketing teams and businesses focused on lead generation, market research, and sentiment analysis. Its ability to process and structure data on the fly makes it a powerful tool for those who need quick and accurate data extraction without the need for an extensive technical setup.

Features:

AI-driven analysis of web pages for automatic data extraction;
ability to extract a variety of content types, including articles, products, and discussions;
supports structured searches to filter results to only matching records;
visual processing for scraping non-English web pages;
data exports are available in JSON or CSV format;
offers a fully hosted SaaS platform, which means no infrastructure to manage.

The scraper is priced at $299 per month and includes a free trial. It has a Capterra rating of 4.5 out of 5.

Scrapy

Scrapy is a robust, open-source web crawling and scraping framework known for its speed and efficiency. Written in Python, Scrapy is compatible with multiple operating systems, including Linux, Windows, Mac, and BSD. The framework allows for the creation of custom search agents and offers flexibility in customizing its components without needing to alter the core system. This makes Scrapy a versatile tool for developers looking to tailor their scraping tools to specific requirements.

$\"image11.png\"$

Features:

asynchronous processing for handling large amounts of data and requests efficiently;
extensive selectors for data extraction using XPath and CSS;
built-in support for generating feed exports in various formats like JSON, CSV, and XML;
middleware support for adding custom functionalities and processing requests and responses;
robust error handling and logging features;
completely free.

Here's a simple example of how to use Scrapy to scrape data from a website:

import scrapy\n\nclass BlogSpider(scrapy.Spider):\n    name = 'blogspider'\n    start_urls = ['https://www.zyte.com/blog/']\n\n    def parse(self, response):\n        for title in response.css('.oxy-post-title'):\n            yield {'title': title.css('::text').get()}\n\n        for next_page in response.css('a.next'):\n            yield response.follow(next_page, self.parse)

Beautiful Soup

Beautiful Soup is a Python library that makes it easy to scrape information from web pages. It's a great tool for beginners and is often used for quick scraping projects or when you need to scrape a website with a simple HTML structure.

$\"image9.png\"$

Features:

simple methods for navigating and searching the parse tree;
parsing HTML or XML documents;
easily find and extract specific information;
modify the parse tree;
works well with multiple parsers like `lxml` and `html5lib`.

Here's a basic example of how to use Beautiful Soup:

from bs4 import BeautifulSoup\n\nhtml_doc =\"\"\"<html><head><title>The Dormouse's story</title></head>\n<body>\n<p class=\"title\"><b>The Dormouse's story</b></p>\n\n<p class=\"story\">Once upon a time there were three little sisters; and their names were\n<a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>,\n<a href=\"http://example.com/lacie\" class=\"sister\" id=\"link2\">Lacie</a> and\n<a href=\"http://example.com/tillie\" class=\"sister\" id=\"link3\">Tillie</a>;\nand they lived at the bottom of a well.</p>\n\n<p class=\"story\">...</p>\n\"\"\"\n\nsoup = BeautifulSoup(html_doc, 'html.parser')\nprint(soup.title.string)  # Outputs \"The Dormouse's story\"

Cheerio

Cheerio is a fast, flexible, and user-friendly library in Node.js that mimics the core functionality of jQuery. Utilizing the parse5 parser by default, Cheerio also offers the option to use the more error-tolerant htmlparser2. This library is capable of parsing almost any HTML or XML document, making it an excellent choice for developers who need efficient and versatile web scraping capabilities.

$\"image10.png\"$

Features:

uses familiar jQuery syntax for manipulating the DOM;
extremely fast and lightweight;
parses and manipulates HTML on the server side;
capable of handling a large number of pages efficiently.

Here's a simple Cheerio example:

const cheerio = require('cheerio');\n\n// some product webpage\nconst html = `\n<html>\n  <head>\n    <title>Sample Page</title>\n  </head>\n  <body>\n    <h1>Welcome to a Product Page</h1>\n    <div class=\"products\">\n      <div class=\"item\">Product 1</div>\n      <div class=\"item\">Product 2</div>\n      <div class=\"item\">Product 3</div>\n    </div>\n  </body>\n</html>\n`;\n\nconst $ = cheerio.load(html);\n\n$('.item').each(function () {\n  const product = $(this).text();\n  console.log(product);\n});

Visualping

$\"Visualping.png\"$

Best for business teams wanting easy automation without coding. Pricing starts at $100/month with custom enterprise tiers. It's good for monitoring website changes for e-commerce pricing, news updates, and social media content. However, it's not ideal for heavy technical scraping or sites with complex login flows. Visualping is user-friendly, has stable integrations and solid support, and handles complex modern sites.

Features:

visual point-and-click selectors;
screenshot diff alerts, scheduling;
Google Sheets integration;
API for batch jobs;
JS rendering;
bulk scraping;
notifications via email/SMS.

Oxylabs

$\"Oxylabs.png\"$

Best for enterprises needing high-scale scraping and robust proxy support. Starts at $49/month with addons and enterprise plans. It is used for market intelligence, retail price monitoring, ad verification at scale. Can boast a vast proxy pool, reliability, customizable workflows. Nevertheless, developer skills are needed when using Oxylabs, plus it has a higher cost at scale.

Features:

100+ million rotating residential/datacenter IPs;
scraping APIs;
geo-targeting;
browser automation;
CAPTCHA bypass.

Decodo

$\"Smartproxy.png\"$

Best for SEO and marketing teams focusing on SERP and social media data. The price starts at $50/month with a trial (flexible plans). Convenient for tracking keyword rankings, collecting social media metadata, brand monitoring. Decodo is marketing-oriented, affordable mid-tier pricing and boasts proxy diversity. Note that it has a limited UI scraping builder and requires developer knowledge.

Features:

65M+ proxies worldwide;
API-first, keyword tracking;
social media extraction;
local data centers, and user-agent control.

ParseHub

$\"ParseHub.png\"$

Best for individuals, freelancers, and small teams on a budget. It is used for small-scale data projects, academic research, price monitoring. The scraper provides thorough tutorials. It is desktop only. There is a free tier available, and paid plans start at $149/month.

Features:

desktop app with visual builder;
supports navigation;
forms,
AJAX;
exports CSV/JSON/Excel;
scheduled runs.

WebHarvy

$\"WebHarvy.png\"$

Best for visual scraping enthusiasts and Windows users. The one-time license price is around $139. It's good for classifieds, real estate listings, and news scraping. No programming needed. Like ParseHub, it is Windows only plus has limited cloud capabilities.

Features:

point-and-click interface;
automatic pattern detection;
image and text extraction;
proxy support.

Content Grabber

$\"Content$

Best for enterprises requiring powerful automation and extraction control. Pricing starts at $995 one-time, with custom enterprise pricing. It's convenient to use for financial data, e-commerce, and media monitoring. Content Grabber's drawback is high cost, and its complexity requires training.

Features:

advanced agent automation;
cloud integration;
error handling;
export formats;
scheduling.

Zyte (formerly Scrapinghub)

$\"Zyte$

Best for developers wanting cloud-based scraping with managed proxy services. Used for data mining, research, product info extraction. Zyte is scalable, developer-friendly but requires technical knowledge. It costs from $99/month with enterprise plans.

Features:

Scrapy Cloud;
rotating proxies;
smart APIs;
scalable infrastructure.

Import.io

$\"Import.io.png\"$

Best for enterprises seeking fast data extraction with ready connectors and APIs. It is beneficial for financial services, market data, sentiment analysis. It's cloud-scalable and easy to integrate. Pricing is available on request with an enterprise focus, but the plans are relatively more expensive.

Features:

web data integration platform;
scheduling;
cloud extraction;
pre-built connectors to major sites.

Adding Proxy-Seller to Your Scraping Toolkit

Proxy-Seller complements these tools by providing fast, stable, geo-diverse proxies essential for anti-blocking and IP rotation strategies. It offers IPv4, IPv6, residential, and mobile proxies across 220+ countries, plus SOCKS5 and HTTP(S) protocols. The flexible pricing starts around $0.08 per IP for IPv6 and up to $0.90 for residential and mobile proxies, with discounts for volume rentals. Its API supports Python, Node.js, PHP, Java, and Golang for smooth integration.

Use cases include SERP scraping with geo-targeting, e-commerce stock monitoring, social media metadata retrieval, and ad verification. The broad proxy network helps avoid IP bans and CAPTCHA challenges. Keep Proxy-Seller in mind for smart proxy rotation, especially when scaling web scraping projects.

Advanced Web Scraping Considerations

You’ll learn to scrape responsibly and technically efficiently. Consider legal, ethical, and technical best practices to sustain success with the best web scraping tools 2026.

Legal and Ethical Guidelines

Always respect robots.txt files and website terms of service.
Implement rate limiting to avoid server overload and IP bans.
Ensure compliance with data privacy laws like GDPR (explicit consent) and CCPA (opt-out options).
Anonymize data where necessary and credit source websites when redistributing data.
Stay updated on landmark legal cases (e.g., hiQ Labs vs. LinkedIn) affecting scraping legality.

Technical Best Practices

Use graceful error handling: retries, timeouts, and failure alerts.
Validate data with schema tools like JSON Schema or Cerberus to maintain quality.
Monitor scrapers with uptime alerts, logs, and change detection tools.
Secure data storage with encryption, access control, and regular backups.
Scale scraping infrastructure using cloud solutions such as AWS Lambda or Kubernetes clusters.
Integrate Proxy-Seller proxies for robust anti-blocking: IP rotation, SOCKS5/HTTP(S) protocols, and API-driven proxy management.

Common Challenges and Solutions

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

Challenge	Solution
Site Layout Changes	Adapt scrapers by using versioned selectors or ML-based scrapers.
Detection/Bans	Use rotating proxies, random user agents, and CAPTCHA-solving services. (Proxy-Seller enhances this).
JavaScript-rich Sites	Access sites via headless browsers like Puppeteer or Playwright.
Cost Optimization	Schedule scrapes intelligently, cache data, and perform incremental updates.

Use this checklist to maintain scraper health:

rate-limit your requests;
validate outputs;
monitor uptime;
use proxies wisely;
comply with legal standards.

What Frameworks and Libraries Are Popular for Custom Web Scraping?

You’ll find the best web scraping tools, Python and others, for custom builds. These open-source frameworks and libraries form the backbone of professional scraping projects.

Scrapy (Python): powerful, flexible, supports middleware and pipelines.
Puppeteer (Node.js): headless Chrome automation, great for JavaScript-heavy sites.
Playwright (multi-language): supports multiple browsers and languages, fast DOM control.
Selenium (multi-language): browser automation with wide compatibility.

Supporting tools and libraries:

2Captcha and Anti-Captcha APIs for automated CAPTCHA solving.
Proxy integration libraries supporting Proxy-Seller or others for easy proxy rotation.
Big Data/ETL Integration (e.g., Apache NiFi, Apache Airflow) to automate workflows, cleansing, and transfers between systems.

Deployment and Orchestration

Docker and Kubernetes enable containerized scraper deployment and scaling.
CI/CD pipelines allow frequent and automated scrape job updates.

Developer scrapers often combine frameworks with providers like Oxylabs, Decodo, and Proxy-Seller. Proxy-Seller is a cost-effective alternative offering broad geo coverage, fast proxies, and robust API support.

Use this practical list to pick your toolset: choose a scraping framework that fits your coding skills, add proxy support for reliability, automate workflows with orchestration tools, and solve CAPTCHAs as needed. This approach ensures efficient, scalable, and maintainable scraping operations in 2026.

Conclusion

In summary, each scraper brings unique features suited for different scraping needs. Cheerio and Beautiful Soup are HTML parsing libraries optimized for Node.js and Python, respectively. Scrapy, another Python-based tool, excels in handling complex scripts and managing large datasets as part of a comprehensive web scraping and parsing framework.

For those evaluating platforms or services for web scraping, here are tailored recommendations based on common selection criteria:

For those needing a simple interface without coding knowledge, Octoparse and WebScraper.io are ideal.
Cheerio, Beautiful Soup, and Scrapy offer completely free tools for those on a budget.
For scraping dynamically updated websites, Bright Data, ScraperAPI, Scraping Dog, and ScrapingBee are recommended.
Diffbot and APIfy specialize in providing APIs for integration and process automation, making them suitable for advanced users looking to streamline their operations.

","createdDate":"2024-12-16T13:59:51.000+00:00","description":"Web scraping tools are specialized software designed to automatically pull data from websites, organizing it into a usable format.","countOfViews":8,"attachments":[{"id":"6a249c391079162e2e8bf690","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf690_5wz2dyq6yswhaimisehyvs0g21giijh7.png","name":"5wz2dyq6yswhaimisehyvs0g21giijh7.png","size":86769,"type":"image/png"},{"id":"6a249c391079162e2e8bf691","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf691_xaao9crr058lyrm82lt5jt9tbh0j26o2.png","name":"xaao9crr058lyrm82lt5jt9tbh0j26o2.png","size":212962,"type":"image/png"},{"id":"6a249c391079162e2e8bf692","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf692_ipl5pvbj1gmirtets10q59hdav5znikp.png","name":"ipl5pvbj1gmirtets10q59hdav5znikp.png","size":122519,"type":"image/png"},{"id":"6a249c391079162e2e8bf693","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf693_hapq9i66s42bqi01zdwvvh380u0sap35.png","name":"hapq9i66s42bqi01zdwvvh380u0sap35.png","size":283858,"type":"image/png"},{"id":"6a249c391079162e2e8bf694","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf694_3azhzq1xsnuc8he106dl8mmylja544r3.png","name":"3azhzq1xsnuc8he106dl8mmylja544r3.png","size":144691,"type":"image/png"},{"id":"6a249c391079162e2e8bf695","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf695_4j1cg0tvxn2tvsl83c6yl476tttazeto.png","name":"4j1cg0tvxn2tvsl83c6yl476tttazeto.png","size":115291,"type":"image/png"},{"id":"6a249c391079162e2e8bf696","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf696_ayncbhqktizvhjdt3j0ttix2t6p5g7jf.png","name":"ayncbhqktizvhjdt3j0ttix2t6p5g7jf.png","size":110048,"type":"image/png"},{"id":"6a249c391079162e2e8bf697","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf697_9ad1gjjh535y42s1yelb40vga39jd3nh.png","name":"9ad1gjjh535y42s1yelb40vga39jd3nh.png","size":176878,"type":"image/png"},{"id":"6a249c391079162e2e8bf698","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf698_ibmac7sx8qmmnso4dw79q2lspgebim8d.png","name":"ibmac7sx8qmmnso4dw79q2lspgebim8d.png","size":69259,"type":"image/png"},{"id":"6a249c391079162e2e8bf699","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf699_4jfc5fizosn6c1zj0uxl4pci6q5w7ptv.png","name":"4jfc5fizosn6c1zj0uxl4pci6q5w7ptv.png","size":128052,"type":"image/png"},{"id":"6a249c391079162e2e8bf69a","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf69a_l5styjue3vh5y7irmi8jzsuout80qkwj.png","name":"l5styjue3vh5y7irmi8jzsuout80qkwj.png","size":148390,"type":"image/png"},{"id":"6a249c391079162e2e8bf69b","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf69b_3tgmx2x5jihokw5ovyfg2f5377b877tf.png","name":"3tgmx2x5jihokw5ovyfg2f5377b877tf.png","size":195859,"type":"image/png"},{"id":"6a249c391079162e2e8bf69c","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf69c_6o412s8wl2yt5x4sizwy8w3dk8bdd49s.png","name":"6o412s8wl2yt5x4sizwy8w3dk8bdd49s.png","size":170479,"type":"image/png"},{"id":"6a249c391079162e2e8bf69d","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf69d_xl4j363dmnk83qppc2ipg5azsbtolspa.png","name":"xl4j363dmnk83qppc2ipg5azsbtolspa.png","size":145144,"type":"image/png"},{"id":"6a249c391079162e2e8bf69f","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf69f_7mkosewaw3fq7fmqzf85fv8d85z251w6.png","name":"7mkosewaw3fq7fmqzf85fv8d85z251w6.png","size":155888,"type":"image/png"},{"id":"6a249c391079162e2e8bf6a0","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf6a0_z6jv3uyr16n9rxir5kmmsacua8wywpy6.png","name":"z6jv3uyr16n9rxir5kmmsacua8wywpy6.png","size":237305,"type":"image/png"},{"id":"6a249c391079162e2e8bf6a1","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf6a1_d7ha9kxap6unavvleoizuvm4iowua2pc.png","name":"d7ha9kxap6unavvleoizuvm4iowua2pc.png","size":233171,"type":"image/png"},{"id":"6a249c391079162e2e8bf6aa","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf6aa_97ochzchc8nagzvzcaldonv0bcd4jfx7.png","name":"97ochzchc8nagzvzcaldonv0bcd4jfx7.png","size":127770,"type":"image/png"},{"id":"6a249c391079162e2e8bf6ab","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf6ab_bva6upwrsn6sine9dcqxz8hl94ubc4v8.png","name":"bva6upwrsn6sine9dcqxz8hl94ubc4v8.png","size":307963,"type":"image/png"}],"mainImg":{"id":"6a249c391079162e2e8bf67e","url":"https://proxy-seller.com/files/fs/image/6a249c391079162e2e8bf67e_best_web_scraping_tools_to_get_ahead_in_2025_mainImg.png","name":"best_web_scraping_tools_to_get_ahead_in_2025_mainImg.png","size":41692,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"best_web_scraping_tools_to_get_ahead_in_2025_img","metaTitle":"Best Web Scraping Tools in 2026","metaDescription":"A selection of the best tools for data scraping. Find out which web scraper suits your needs best."},{"id":"6a049d2a35baad34714b020b","tag":"guide_to_scraping_google_maps_data_with_python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":4,"title":"Guide to scraping Google Maps data with Python","body":"

Scraping data from Google Maps using Python allows for the collection of valuable information about locations, businesses, and services, which is beneficial for market analysis, identifying optimal new venue locations, maintaining current directories, competitor analysis, and gauging the popularity of places. This guide provides a comprehensive walkthrough on how to extract information from Google Maps utilizing the Python libraries requests and lxml. It includes detailed instructions on making requests, handling responses, parsing structured data, and exporting it to a CSV file.

Setting up your environment

Ensure you have the following Python libraries installed:

requests;
lxml;
csv (standard library).

Install these libraries using pip if needed:

\npip install requests\npip install lxml\n

Below, we'll present a step-by-step process of scraping, complete with examples.

Step-by-step Guide to scraping data from Google Maps

In the following sections, we will walk through a detailed step-by-step process for scraping data from Google Maps, complete with visual examples to guide you through each stage.

Step 1. Define the target URL

Specify the URL from which you want to scrape data.

\nurl = \"https link\"\n

Step 2. Define headers and proxies

Setting up appropriate headers is crucial for mimicking the activities of a genuine user, significantly reducing the chances of the scraper being flagged as a bot. Additionally, integrating proxy servers helps maintain continuous scraping activities by circumventing any blocks that might arise from exceeding the request limits associated with a single IP address.

\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'priority': 'u=0, i',\n    'sec-ch-ua': '\"Not)A;Brand\";v=\"99\", \"Google Chrome\";v=\"127\", \"Chromium\";v=\"127\"',\n    'sec-ch-ua-arch': '\"x86\"',\n    'sec-ch-ua-bitness': '\"64\"',\n    'sec-ch-ua-full-version-list': '\"Not)A;Brand\";v=\"99.0.0.0\", \"Google Chrome\";v=\"127.0.6533.72\", \"Chromium\";v=\"127.0.6533.72\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-model': '\"\"',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-ch-ua-platform-version': '\"6.5.0\"',\n    'sec-ch-ua-wow64': '?0',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36',\n}\n\nproxies = {\n    \"http\": \"http://username:password@your_proxy_ip:port\",\n    \"https\": \"https://username:password@your_proxy_ip:port\",\n}\n\n

Step 3. Fetch the page content

Send a request to the Google Maps URL and get the page content:

\nimport requests\n\nresponse = requests.get(url, headers=headers, proxies=proxies)\nif response.status_code == 200:\n    page_content = response.content\nelse:\n    print(f\"Failed to retrieve the page. Status code: {response.status_code}\")\n\n

Step 4. Parse the HTML content

Use lxml to parse the HTML content:

\nfrom lxml import html\n\nparser = html.fromstring(page_content)\n\n

Identifying the data XPaths

Understanding the structure of the HTML document is crucial for extracting data correctly. You need to identify the XPath expressions for the data points you want to scrape. Here’s how you can do it:

Inspect the Web Page: Open the Google Maps page in a web browser and use the browser’s developer tools (right-click > Inspect) to examine the HTML structure.
Find the Relevant Elements: Look for the HTML elements that contain the data you want to scrape (e.g., restaurant names, addresses).
Write XPaths: Based on the HTML structure, write XPath expressions to extract the data. For this guide, the XPaths are:

Restaurant Name:

\n//div[@jscontroller=\"AtSb\"]/div/div/div/a/div/div/div/span[@class=\"OSrXXb\"]/text()\n

Address:

\n //div[@jscontroller=\"AtSb\"]/div/div/div/a/div/div/div[2]/text()\n

Options:

\n = ', '.join(result.xpath('.//div[@jscontroller=\"AtSb\"]/div/div/div/a/div/div/div[4]/div/span/span[1]//text()'))\n

Geo Latitude:

\n//div[@jscontroller=\"AtSb\"]/div/@data-lat\n

Geo Longitude:

\n //div[@jscontroller=\"AtSb\"]/div/@data-lng\n

Step 5. Extracting data

Extract the data using the identified XPaths:

\nresults = parser.xpath('//div[@jscontroller=\"AtSb\"]')\ndata = []\n\nfor result in results:\n    restaurant_name = result.xpath('.//div/div/div/a/div/div/div/span[@class=\"OSrXXb\"]/text()')[0]\n    address = result.xpath('.//div/div/div/a/div/div/div[2]/text()')[0]\n    options = ', '.join(result.xpath('.//div/div/div/a/div/div/div[4]/div/span/span[1]//text()'))\n    geo_latitude = result.xpath('.//div/@data-lat')[0]\n    geo_longitude = result.xpath('.//div/@data-lng')[0]\n\n    # Append to data list\n    data.append({\n        \"restaurant_name\": restaurant_name,\n        \"address\": address,\n        \"options\": options,\n        \"geo_latitude\": geo_latitude,\n        \"geo_longitude\": geo_longitude\n    })\n\n

Step 6. Save data to CSV

Save the extracted data to a CSV file:

\nimport csv\n\nwith open(\"google_maps_data.csv\", \"w\", newline='', encoding='utf-8') as csv_file:\n    writer = csv.DictWriter(csv_file, fieldnames=[\"restaurant_name\", \"address\", \"options\", \"geo_latitude\", \"geo_longitude\"])\n    writer.writeheader()\n    for entry in data:\n        writer.writerow(entry)\n\n

Complete code

Here is the complete code for scraping Google Maps data:

\nimport requests\nfrom lxml import html\nimport csv\n\n# Define the target URL and headers\nurl = \"https://www.google.com/search?sca_esv=04f11db33f1535fb&sca_upv=1&tbs=lf:1,lf_ui:4&tbm=lcl&sxsrf=ADLYWIIFVlh6WQCV6I2gi1yj8ZyvZgLiRA:1722843868819&q=google+map+restaurants+near+me&rflfq=1&num=10&sa=X&ved=2ahUKEwjSs7fGrd2HAxWh1DgGHbLODasQjGp6BAgsEAE&biw=1920&bih=919&dpr=1\"\nheaders = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'priority': 'u=0, i',\n    'sec-ch-ua': '\"Not)A;Brand\";v=\"99\", \"Google Chrome\";v=\"127\", \"Chromium\";v=\"127\"',\n    'sec-ch-ua-arch': '\"x86\"',\n    'sec-ch-ua-bitness': '\"64\"',\n    'sec-ch-ua-full-version-list': '\"Not)A;Brand\";v=\"99.0.0.0\", \"Google Chrome\";v=\"127.0.6533.72\", \"Chromium\";v=\"127.0.6533.72\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-model': '\"\"',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-ch-ua-platform-version': '\"6.5.0\"',\n    'sec-ch-ua-wow64': '?0',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36',\n}\nproxies = {\n    \"http\": \"http://username:password@your_proxy_ip:port\",\n    \"https\": \"https://username:password@your_proxy_ip:port\",\n}\n\n# Fetch the page content\nresponse = requests.get(url, headers=headers, proxies=proxies)\nif response.status_code == 200:\n    page_content = response.content\nelse:\n    print(f\"Failed to retrieve the page. Status code: {response.status_code}\")\n    exit()\n\n# Parse the HTML content\nparser = html.fromstring(page_content)\n\n# Extract data using XPath\nresults = parser.xpath('//div[@jscontroller=\"AtSb\"]')\ndata = []\n\nfor result in results:\n    restaurant_name = result.xpath('.//div/div/div/a/div/div/div/span[@class=\"OSrXXb\"]/text()')[0]\n    address = result.xpath('.//div/div/div/a/div/div/div[2]/text()')[0]\n    options = ', '.join(result.xpath('.//div/div/div/a/div/div/div[4]/div/span/span[1]//text()'))\n    geo_latitude = result.xpath('.//div/@data-lat')[0]\n    geo_longitude = result.xpath('.//div/@data-lng')[0]\n\n    # Append to data list\n    data.append({\n        \"restaurant_name\": restaurant_name,\n        \"address\": address,\n        \"options\": options,\n        \"geo_latitude\": geo_latitude,\n        \"geo_longitude\": geo_longitude\n    })\n\n# Save data to CSV\nwith open(\"google_maps_data.csv\", \"w\", newline='', encoding='utf-8') as csv_file:\n    writer = csv.DictWriter(csv_file, fieldnames=[\"restaurant_name\", \"address\", \"options\", \"geo_latitude\", \"geo_longitude\"])\n    writer.writeheader()\n    for entry in data:\n        writer.writerow(entry)\n\nprint(\"Data has been successfully scraped and saved to google_maps_data.csv.\")\n\n

For effective web scraping, it's crucial to use the right request headers and proxies. The optimal proxy choices are data center or ISP proxies, which offer high speeds and low latency. However, since these are static proxies, implementing IP rotation is necessary to prevent blocking effectively. An alternative and more user-friendly option is to use residential proxies. These dynamic proxies simplify the rotation process and have a higher trust factor, making them more effective at circumventing blocks.

","createdDate":"2024-12-12T09:35:23.000+00:00","description":"Scraping data from Google Maps using Python allows for the collection of valuable information about locations, businesses, and services.","countOfViews":7,"attachments":[],"mainImg":{"id":"6a249c5f1079162e2e8bfc9a","url":"https://proxy-seller.com/files/fs/image/6a249c5f1079162e2e8bfc9a_guide_to_scraping_google_maps_data_with_python_mainImg.png","name":"guide_to_scraping_google_maps_data_with_python_mainImg.png","size":43930,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"guide_to_scraping_google_maps_data_with_python_img","metaTitle":"Guide to scraping Google Maps data with Python","metaDescription":"Learn more about Google Maps data scraping with Python using proxies and a custom User-Agent. Effortlessly avoid blocks while gathering location info."},{"id":"6a049d2d35baad34714b025f","tag":"step_by_step_guide_to_scraping_google_news_with_python","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":6,"title":"Step-by-step guide to scraping Google News with Python","body":"

In gathering the most recent news headlines, monitoring news trends, and performing sentiment analysis on present matters, scraping Google News proves to be an invaluable tool. In this article, we will guide you through the process of scraping Google News by means of Python. We will employ a requests library to obtain page content; lxml for parsing HTML documents and extracting data needed. By the end of this tutorial, you will learn how to extract news headlines and their respective links from Google News into structured JSON format.

Step 1: Setting Up the Environment

Before we begin, make sure you have Python installed on your system. You can install the required libraries by running the following commands:

pip install requests \npip install lxml

These libraries will allow us to make HTTP requests and parse the HTML content of the webpage.

Step 2: Understanding the Target URL and XPath Structure

We'll be scraping the Google News page at the following URL:

URL = \"https://news.google.com/topics/CAAqKggKIiRDQkFTRlFvSUwyMHZNRGRqTVhZU0JXVnVMVWRDR2dKSlRpZ0FQAQ?hl=en-US&gl=US&ceid=US%3Aen\"

This page contains multiple news items, each with a main headline and related articles. The XPath structure for these items is as follows:

Main News Container: //c-wiz[@jsrenderer=\"ARwRbe\"]
Main News Title: //c-wiz[@jsrenderer=\"ARwRbe\"]/c-wiz/div/article/a/text()
Main News Link: //c-wiz[@jsrenderer=\"ARwRbe\"]/c-wiz/div/article/a/@href
Related News Container: //c-wiz[@jsrenderer=\"ARwRbe\"]/c-wiz/div/div/article/
Related News Title: //c-wiz[@jsrenderer=\"ARwRbe\"]/c-wiz/div/div/article/a/text()
Related News Link: //c-wiz[@jsrenderer=\"ARwRbe\"]/c-wiz/div/div/article/a/@href

The HTML structure of Google News remains consistent across different pages, ensuring that the specified XPath elements are applicable universally.

Step 3: Fetching Google News content

We'll start by fetching the content of the Google News page using the requests library. Here's the code to fetch the page content:

import requests\n\nurl = \"https://news.google.com/topics/CAAqKggKIiRDQkFTRlFvSUwyMHZNRGRqTVhZU0JXVnVMVWRDR2dKSlRpZ0FQAQ?hl=en-US&gl=US&ceid=US%3Aen\"\nresponse = requests.get(url)\n\nif response.status_code == 200:\n    page_content = response.content\nelse:\n    print(f\"Failed to retrieve the page. Status code: {response.status_code}\")

This code sends a GET request to the Google News URL and stores the HTML content of the page in the page_content variable.

Step 4: Parsing the HTML content with lxml

With the HTML content in hand, we can use lxml to parse the page and extract the news headlines and links.

from lxml import html\n\n# Parse the HTML content\nparser = html.fromstring(page_content)

Step 5: Extracting news data

Google News organizes its articles in specific containers. We'll first extract these containers using their XPath and then iterate through them to extract the individual news headlines and links.

Extracting main news articles

The main news articles are located under the following XPath:

main_news_elements = parser.xpath('//c-wiz[@jsrenderer=\"ARwRbe\"]')

We can now loop through first 10 valid elements and extract the titles and links:

news_data = []\n\nfor element in main_news_elements[:10]:\n    title = element.xpath('.//c-wiz/div/article/a/text()')[0]\n    link = \"https://news.google.com\" + element.xpath('.//c-wiz/div/article/a/@href')[0][1:]\n\n    \n    # Ensure data exists before appending to the list\n    if title and link:\n        news_data.append({\n        \"main_title\": title,\n        \"main_link\": link,\n    })

Extracting related articles within each main new element

The main news element has subsections where related news is present. We can extract these using a similar approach:

related_articles = []\nrelated_news_elements = element.xpath('.//c-wiz/div/div/article')\n\nfor related_element in related_news_elements:\n    related_title = related_element.xpath('.//a/text()')[0]\n    related_link = \"https://news.google.com\" + related_element.xpath('.//a/@href')[0][1:]\n    related_articles.append({\"title\": related_title, \"link\": related_link})\n\nnews_data.append({\n    \"main_title\": title,\n    \"main_link\": link,\n    \"related_articles\": related_articles\n})

Step 6: Saving the data as JSON

After extracting the data, we can save it in a JSON file for later use.

import json\n\nwith open('google_news_data.json', 'w') as f:\n    json.dump(news_data, f, indent=4)

This code will create a file named google_news_data.json containing all the scraped news headlines and their corresponding links.

Using proxies

When scraping large amounts of data, especially from high-traffic sites like Google News, you might encounter issues like IP blocking or rate limiting. To avoid this, you can use proxies. Proxies allow you to route your requests through different IP addresses, making it harder for the website to detect and block your scraping activities.

For this tutorial, you can use a proxy by modifying the requests.get call:

proxies = {\n    \"http\": \"http://your_proxy_ip:port\",\n    \"https\": \"https://your_proxy_ip:port\",\n}\n\nresponse = requests.get(url, proxies=proxies)

If you're working with a service provider that manages proxy rotation, you only need to configure the service in your requests. The provider will handle the rotation and IP pool management on their end.

Customizing request headers

Sometimes, websites may block requests that don't have the proper headers, such as a user-agent string that identifies the request as coming from a browser. You can customize your headers to avoid detection:

headers = {\n    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n    'accept-language': 'en-IN,en;q=0.9',\n    'cache-control': 'no-cache',\n    'dnt': '1',\n    'pragma': 'no-cache',\n    'priority': 'u=0, i',\n    'sec-ch-ua': '\"Not)A;Brand\";v=\"99\", \"Google Chrome\";v=\"127\", \"Chromium\";v=\"127\"',\n    'sec-ch-ua-arch': '\"x86\"',\n    'sec-ch-ua-bitness': '\"64\"',\n    'sec-ch-ua-full-version-list': '\"Not)A;Brand\";v=\"99.0.0.0\", \"Google Chrome\";v=\"127.0.6533.72\", \"Chromium\";v=\"127.0.6533.72\"',\n    'sec-ch-ua-mobile': '?0',\n    'sec-ch-ua-model': '\"\"',\n    'sec-ch-ua-platform': '\"Linux\"',\n    'sec-ch-ua-platform-version': '\"6.5.0\"',\n    'sec-ch-ua-wow64': '?0',\n    'sec-fetch-dest': 'document',\n    'sec-fetch-mode': 'navigate',\n    'sec-fetch-site': 'none',\n    'sec-fetch-user': '?1',\n    'upgrade-insecure-requests': '1',\n    'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36',\n}\n\nresponse = requests.get(url, headers=headers)

Complete code example

Here’s the full code, combining all the steps:

import requests\nimport urllib3\nfrom lxml import html\nimport json\n\nurllib3.disable_warnings()\n\n# URL and headers\nurl = \"https://news.google.com/topics/CAAqKggKIiRDQkFTRlFvSUwyMHZNRGRqTVhZU0JXVnVMVWRDR2dKSlRpZ0FQAQ?hl=en-US&gl=US&ceid=US%3Aen\"\nheaders = {\n   'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',\n   'accept-language': 'en-IN,en;q=0.9',\n   'cache-control': 'no-cache',\n   'dnt': '1',\n   'pragma': 'no-cache',\n   'priority': 'u=0, i',\n   'sec-ch-ua': '\"Not)A;Brand\";v=\"99\", \"Google Chrome\";v=\"127\", \"Chromium\";v=\"127\"',\n   'sec-ch-ua-arch': '\"x86\"',\n   'sec-ch-ua-bitness': '\"64\"',\n   'sec-ch-ua-full-version-list': '\"Not)A;Brand\";v=\"99.0.0.0\", \"Google Chrome\";v=\"127.0.6533.72\", \"Chromium\";v=\"127.0.6533.72\"',\n   'sec-ch-ua-mobile': '?0',\n   'sec-ch-ua-model': '\"\"',\n   'sec-ch-ua-platform': '\"Linux\"',\n   'sec-ch-ua-platform-version': '\"6.5.0\"',\n   'sec-ch-ua-wow64': '?0',\n   'sec-fetch-dest': 'document',\n   'sec-fetch-mode': 'navigate',\n   'sec-fetch-site': 'none',\n   'sec-fetch-user': '?1',\n   'upgrade-insecure-requests': '1',\n   'user-agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36',\n}\n\n# Proxies configuration (replace with your proxy details)\nproxy = 'ip:port'\nproxies = {\n   \"http\": f\"http://{proxy}\",\n   \"https\": f\"https://{proxy}\",\n}\n\n# Fetch the page content with the specified headers and proxies\nresponse = requests.get(url, headers=headers, proxies=proxies, verify=False)\n\n# Check if the request was successful\nif response.status_code == 200:\n   page_content = response.content\nelse:\n   print(f\"Failed to retrieve the page. Status code: {response.status_code}\")\n   exit()\n\n# Parse the HTML content using lxml\nparser = html.fromstring(page_content)\n\n# Extract the main news and related articles\nmain_news_elements = parser.xpath('//*[@id=\"i10-panel\"]/c-wiz/c-wiz')\n\n# Initialize a list to store the extracted news data\nnews_data = []\n\n# Loop through each main news element\nfor element in main_news_elements[:10]:\n   # Extract the main news title and link\n   title = element.xpath('.//c-wiz/div/article/a/text()')\n   link = element.xpath('.//c-wiz/div/article/a/@href')\n\n   # Initialize a list to store related articles for this main news\n   related_articles = []\n\n   # Extract related news elements within the same block\n   related_news_elements = element.xpath('.//c-wiz/div/div/article')\n\n   # Loop through each related news element and extract the title and link\n   for related_element in related_news_elements:\n       related_title = related_element.xpath('.//a/text()')[0]\n       related_link = \"https://news.google.com\" + related_element.xpath('.//a/@href')[0][1:]\n       related_articles.append({\"title\": related_title, \"link\": related_link})\n\n   # Append the main news and its related articles to the news_data list\n   if title is not None:\n       news_data.append({\n           \"main_title\": title,\n           \"main_link\": f'https://news.google.com{link}',\n           \"related_articles\": related_articles\n       })\n   else:\n       continue\n\n\n# Save the extracted data to a JSON file\nwith open(\"google_news_data.json\", \"w\") as json_file:\n   json.dump(news_data, json_file, indent=4)\n\nprint('Data extraction complete. Saved to google_news_data.json')

Scraping Google News using Python, along with the requests and lxml libraries, facilitates detailed analysis of news trends. Implementing proxies and configuring request headers is crucial to avoid blocks and maintain scraper stability. Ideal proxies for this purpose include IPv4 and IPv6 datacenter proxies and ISP proxies, which offer high speeds and low ping. Additionally, dynamic residential proxies are highly effective due to their superior trust factor.

","createdDate":"2024-12-06T11:36:11.000+00:00","description":"In gathering the most recent news headlines, monitoring news trends, and performing sentiment analysis on present matters, scraping Google News proves to be an invaluable tool.","countOfViews":7,"attachments":[],"mainImg":{"id":"6a249ca01079162e2e8c08e4","url":"https://proxy-seller.com/files/fs/image/6a249ca01079162e2e8c08e4_step_by_step_guide_to_scraping_google_news_with_python_mainImg.png","name":"step_by_step_guide_to_scraping_google_news_with_python_mainImg.png","size":44028,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"step_by_step_guide_to_scraping_google_news_with_python_img","metaTitle":"Step-by-step guide to scraping Google News with Python","metaDescription":"Develop a Python script for Google News scraping. Integrate proxies and User-Agent headers to bypass Google’s security blocks effectively."},{"id":"6a049d2f35baad34714b0270","tag":"openbullet_2_overview_of_the_web_scraping_tool","groups":["6aa7a82e264e7061a333704b"],"groupsDtos":[{"id":"6aa7a82e264e7061a333704b","tag":"web-scraping","namesByLocale":{"en":"Web Scraping","ru":"Веб-скрапинг","es":"Web Scraping","zh":"网页抓取","vn":"Web Scraping","de":"Web Scraping","ua":"Веб-скрапінг","pt":"Web Scraping","fr":"Web Scraping","hi":"वेब स्क्रैपिंग","tr":"Web Kazıma","ko":"웹 스크래핑","it":"Web Scraping","id":"Web Scraping","pl":"Web Scraping"},"fade":""}],"author":{"slug":"igor-efimenko","name":"Igor Efimenko","title":"","shortBio":"Head of Development @ Proxy-Seller","authorAbout":"Head of Development at Proxy-Seller with 12+ years in DevOps. Writes on Python scraping, browser automation, and engineering benchmarks — backed by production experience at scale.","photo":{"id":"6abcd31e0ffe3e314374c36c","url":"https://proxy-seller.com/files/fs/image/6abcd31e0ffe3e314374c36c_IE.jpg","name":"IE.jpg","size":32706,"type":"image/jpeg"}},"readTimeMinutes":9,"title":"OpenBullet 2: Overview of the Web Scraping Tool","body":"

OpenBullet 2 is a versatile web scraping tool known for its user-friendly interface, extensive functionality, support for multithreading, and customizable process settings. It's particularly popular among professionals in data parsing and analysis, SEO, cybersecurity, and software testing. This tool enables users to rapidly gather data and automate repetitive tasks, streamlining operations in various tech-driven fields. Its flexibility and ease of use make it an essential resource for specialists aiming to optimize their workflows and enhance productivity efficiently.

$\"1en.jpg\"$

Key features of OpenBullet 2 for web scraping and automation

OpenBullet 2 is developed on the lightweight, modular, and cross-platform .NET Core runtime environment, designed for running cloud applications and Internet services. It operates both as a native desktop application for Windows and as a web client for Linux, Windows, macOS, and Docker. The platform allows users to customize query parameters to extract the required data and utilize HTML and JSON to format the gathered information appropriately. Additionally, OpenBullet 2 offers features for sorting, filtering, and aggregating data, which facilitates the processing and analysis of large volumes of information.

Support for multithreading

OpenBullet 2 employs asynchronous methods for executing network requests alongside a thread pool. This setup facilitates the parallel execution of multiple tasks such as collecting and processing data from various resources, interacting with APIs, and filtering information. The result is a significant increase in the speed of data collection operations on the Internet. Users have the flexibility to configure the number of threads based on their requirements, optimizing load distribution on their devices and minimizing the risk of website blocks. Additionally, web scraping APIs can be integrated with OpenBullet 2 to handle multiple requests and enhance data extraction efficiency.

Configuring bot scrapers

OpenBullet 2 allows users to create custom configurations for bots that can send requests to target sites, test APIs, or collect data on complex websites. These configurations consist of scripts and parameters defining the bot's interactions with web resources, the data it collects, and its processing methods. Users can configure elements such as URLs, request headers, authentication methods, and filtering rules.

OpenBullet 2 also supports script customization in C#, enabling users to tailor request processing logic to their specific needs. The flexible configuration system allows for script logic adjustments, the addition of new features, and the integration of third-party frameworks, fostering the creation of bespoke solutions for various scenarios. Additionally, users can share their configurations and use templates created by others.

Processing and managing results (Hits)

After executing scripts and receiving data, OpenBullet 2 automatically classifies the results, allowing users to easily distinguish between successful and unsuccessful attempts. Users can set up filters and rules to sort results by various criteria such as request status, data type (e.g., logins, passwords), date and time of receipt, and data uniqueness (e.g., removing duplicates). Additionally, conditions can be set for specific values, enabling users to highlight or ignore records that meet certain criteria.

Broad support for browsers and libraries

OpenBullet 2 is compatible with major browsers such as Google Chrome, Mozilla Firefox, and Opera. The platform allows users to adapt script behavior to the specifics of each browser, enhancing the ability to bypass anti-fraud systems and improving compatibility with various web technologies. Users can adjust parameters like user agents and cookies and utilize special browser features for more effective website interaction.

Furthermore, OpenBullet 2 supports integration with popular libraries that extend its automation capabilities. HtmlAgilityPack facilitates HTML document parsing for easy data extraction from websites. Newtonsoft.Json aids in handling JSON by providing efficient serialization and deserialization, useful in API interactions. RestSharp is instrumental in working with RESTful APIs, enabling users to perform network requests and process responses efficiently, eliminating the need for extensive manual coding.

Bypassing CAPTCHA and anti-fraud filters

OpenBullet 2 provides robust features to effectively bypass CAPTCHA and anti-fraud filters, ensuring seamless access to required data:

Automatic bypass of CAPTCHA: the application integrates with several captcha-solving services, such as 2Captcha and Anti-Captcha. This integration automates the process of bypassing CAPTCHAs, facilitating uninterrupted access to necessary data.
Proxy server integration: OpenBullet 2 supports a variety of private proxy types including HTTP, HTTPS, SOCKS4, SOCKS4a, and SOCKS5. This functionality enables users to bypass geographical restrictions, conceal internet activities from anti-fraud systems, and conduct multi-threaded operations. Additionally, the platform offers automatic IP address rotation, enhancing anonymity and reducing the likelihood of detection.
Configuring User-Agent: to further reduce the risk of detection and blocking, OpenBullet 2 allows for the customization of user agents. This feature simulates requests from different browsers or devices, diversifying the digital footprint during automated tasks.

These capabilities collectively lower the risk of triggering anti-fraud mechanisms, helping to avoid blocks and failures during the data scraping process, thereby ensuring efficient and reliable data collection.

How to install OpenBullet 2

Since OpenBullet 2 is developed in C#, it requires specific .NET applications to operate effectively. Before installation, ensure you have the latest versions of the .NET Framework, .NET Desktop Runtime, and ASP.NET Core Runtime, which are available for download from the official Microsoft website.

Here’s how to install OpenBullet 2:

Navigate to the OpenBullet 2 documentation page and download the latest release version, which will typically be an .exe file. Save this file in an empty folder designated for the application's distribution after installation.\n
$\"2en.png\"$
\n
Execute the downloaded file. In the opening interface, use the arrow keys on your keyboard to select “stable builds” and press “Enter” to proceed with installing the stable version of OpenBullet 2.\n
$\"3en.png\"$
\n
When prompted, press “y” followed by “Enter” to confirm the installation of the latest version.\n
$\"4en.png\"$
\n
Allow some time for the distribution to download and unpack. Once completed, press any key to continue.\n
$\"5en.png\"$
\n
Locate and open the folder where you initially saved the .exe file and run the “OpenBullet2.Native” file to start the application.\n
$\"6en.png\"$
\n
If the program closes immediately after launching, it could indicate an issue with the .NET Desktop Runtime installation. In such a case, attempt reinstalling the .NET Desktop Runtime to resolve the problem and ensure the proper functioning of OpenBullet 2.

OpenBullet 2 client interface

The OpenBullet 2 client interface is managed through various tabs positioned at the top of the interface, allowing users to access and configure the application's functionalities easily.

Home

This tab serves as the central panel for navigation and control of all main functions. It provides quick access to created jobs, configurations, and scripts. The window displays general information about the status of active processes and key performance metrics, such as the number of completed tasks and current results.

$\"7en.png\"$

Jobs

This section is designed to create and manage jobs/bots for operations like data collection, parsing, and interacting with APIs. It allows users to configure job parameters, including configurations, used scripts, and flow parameters. It also displays information about the current status of job execution, results, and errors, enabling users to track progress and adjust settings in real-time.

$\"8en.png\"$

Monitor

Here, users can monitor active requests, flow status, and script execution results in real-time. It provides detailed information about each request, including status and response time, facilitating quick identification of issues and optimization of processes.

$\"9en.png\"$

Proxies

This tab allows users to add, edit, and delete proxy servers, as well as configure their parameters like protocol type and authentication. It shows information about proxy status, including availability and response time, assisting in selecting the most effective servers.

$\"10en.png\"$

Wordlists

This section manages lists of logins and passwords used in authentication and parsing processes. Users can create, import, edit, sort, and filter wordlists. It also offers analysis and verification tools to assess the effectiveness and relevance of the lists. Importantly, wordlists are not stored in the OpenBullet database; only links to their locations on the disk are saved, ensuring the security of user data.

$\"11en.png\"$

Configs

In the “Configs” tab, users can create, edit, and customize configurations and select from pre-prepared templates. Each configuration includes parameters for authentication, data processing, and various parsing methods. This window also allows checking configurations for correct operation, enhancing flexibility and efficiency in automating data collection processes.

$\"12en.png\"$

Hits

This window displays the results of previously launched tasks, including successful authentications, connections to proxy servers, and data extraction from web resources. It offers functions for filtering and sorting, simplifying the search for necessary information. Results can be exported to various formats, ensuring convenient data interaction in the future. All successful records are saved in the program database, securing them even when the application is closed.

$\"13en.png\"$

Plugins

Plugins extend the functionality of OpenBullet 2 by adding new features, improving performance, or integrating with external services. This tab allows users to view available plugins, install new ones, and manage installed plugins, including activation and deactivation, providing flexibility in tailoring the application to specific tasks and needs.

$\"14en.png\"$

OB Settings

This tab is dedicated to configuring the fundamental parameters and settings of the OpenBullet 2 application itself. Users can manage various settings including interface preferences, security configurations, software updates, and integration with external services.

$\"15en.png\"$

This section also offers customization options for the application's interface, such as changing the design theme, configuring notifications, and managing data storage options.

$\"16en.png\"$

RL Settings

The “RL Settings” tab is designed for configuring the parameters associated with render-life (RL) mechanisms that enable dynamic processing and real-time display of data. Here, users can adjust settings related to data processing and analysis, including timing, request limits, and resource management. These configurations help optimize data collection processes and enhance the efficiency of operations with web resources.

$\"17en.png\"$

About

The About section provides detailed information about the developers, testers, and contributors to the OpenBullet 2 open-source project. It also includes valuable resources such as links to communities and forums. These platforms offer a space for users to share experiences, seek advice, and receive support regarding the application.

$\"18en.png\"$

Setting up a proxy server in OpenBullet 2

To connect a new proxy server in the OpenBullet 2 application, you can follow these straightforward steps:

Navigate to the “Proxies” section and click on the “Import” icon.\n
$\"19en.png\"$
\n
Choose the “Paste” option and then select the connection type. Enter the proxy server details in the format “IP-address: port”, for example, “80.71.217.138:80”.\n
$\"20en.png\"$
\n
Input the username and password required for connecting to the proxy server and then click “ACCEPT” to save the new connection settings.\n
$\"21en.png\"$
\n

Choosing the Right Proxy Type

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n

\n Proxy Type \n	Best suited for
\n ISP Proxy \n	Long-running sessions, API access, stable integrations
\n Residential proxy \n	Traffic distribution, user-like behavior scenarios
\n Mobile proxies \n	Mobile environment testing
\n IPv6 proxy (Datacenter) \n	Large-scale technical testing
\n IPv4 (Datacenter) \n	Fast and simple checks

Conclusion

OpenBullet 2 is a versatile and robust tool that caters to professionals in web scraping, SEO, cybersecurity, and software testing. This free software excels in data collection and analysis, offering features such as multithreading, script customization, proxy and CAPTCHA support, and comprehensive logging. Users can effectively manage bot configurations, maintain lists of logins and passwords, and utilize plugins to expand the tool’s capabilities. Its extensive functionality and adaptability make OpenBullet 2 an ideal solution for automating complex web-based processes and enhancing scraping efficiency.

","createdDate":"2024-12-04T16:06:01.000+00:00","description":"OpenBullet 2 is a versatile web scraping tool known for its user-friendly interface, extensive functionality, support for multithreading, and customizable process settings.","countOfViews":10,"attachments":[{"id":"6a249cb41079162e2e8c0cec","url":"https://proxy-seller.com/files/fs/image/6a249cb41079162e2e8c0cec_tneq7fphy90atb35trevf9i59ymfv5lm.png","name":"tneq7fphy90atb35trevf9i59ymfv5lm.png","size":56586,"type":"image/png"},{"id":"6a249cb41079162e2e8c0cee","url":"https://proxy-seller.com/files/fs/image/6a249cb41079162e2e8c0cee_wiu76tsok320ppb9p1v4vp78ul4lq4pu.png","name":"wiu76tsok320ppb9p1v4vp78ul4lq4pu.png","size":57721,"type":"image/png"},{"id":"6a249cb41079162e2e8c0cf0","url":"https://proxy-seller.com/files/fs/image/6a249cb41079162e2e8c0cf0_ht9fvcr34e2pvtjjrhtn1yboa81zc0n6.png","name":"ht9fvcr34e2pvtjjrhtn1yboa81zc0n6.png","size":60174,"type":"image/png"},{"id":"6a249cb41079162e2e8c0cef","url":"https://proxy-seller.com/files/fs/image/6a249cb41079162e2e8c0cef_uy9okla35boiast9ke0ezeshjovzly0m.jpg","name":"uy9okla35boiast9ke0ezeshjovzly0m.jpg","size":30337,"type":"image/jpeg"},{"id":"6a249cb41079162e2e8c0cf3","url":"https://proxy-seller.com/files/fs/image/6a249cb41079162e2e8c0cf3_rmjxlbcn4asz8149siplu993l39yq71f.png","name":"rmjxlbcn4asz8149siplu993l39yq71f.png","size":172474,"type":"image/png"},{"id":"6a249cb41079162e2e8c0cf4","url":"https://proxy-seller.com/files/fs/image/6a249cb41079162e2e8c0cf4_md84lgzrszn57dfix1mptyifany9fdc8.png","name":"md84lgzrszn57dfix1mptyifany9fdc8.png","size":37214,"type":"image/png"},{"id":"6a249cb41079162e2e8c0cf6","url":"https://proxy-seller.com/files/fs/image/6a249cb41079162e2e8c0cf6_ot7zhe9hd3c2qsqypykh8he064pvp7pa.png","name":"ot7zhe9hd3c2qsqypykh8he064pvp7pa.png","size":35446,"type":"image/png"},{"id":"6a249cb41079162e2e8c0cf8","url":"https://proxy-seller.com/files/fs/image/6a249cb41079162e2e8c0cf8_gl1vn6k0sohysz946r0qyd33rdnpkeyh.png","name":"gl1vn6k0sohysz946r0qyd33rdnpkeyh.png","size":60881,"type":"image/png"},{"id":"6a249cb41079162e2e8c0cf9","url":"https://proxy-seller.com/files/fs/image/6a249cb41079162e2e8c0cf9_yyo9k4mxacowawud0d1bx5zo7d7xkec9.png","name":"yyo9k4mxacowawud0d1bx5zo7d7xkec9.png","size":58884,"type":"image/png"},{"id":"6a249cb41079162e2e8c0cfa","url":"https://proxy-seller.com/files/fs/image/6a249cb41079162e2e8c0cfa_mdfep7q17xz56wdtv79shkb82tvvnvwh.png","name":"mdfep7q17xz56wdtv79shkb82tvvnvwh.png","size":45202,"type":"image/png"},{"id":"6a249cb41079162e2e8c0cfc","url":"https://proxy-seller.com/files/fs/image/6a249cb41079162e2e8c0cfc_hgma680ghw3309i2mee7pm8lyqain5n2.png","name":"hgma680ghw3309i2mee7pm8lyqain5n2.png","size":76703,"type":"image/png"},{"id":"6a249cb41079162e2e8c0cfb","url":"https://proxy-seller.com/files/fs/image/6a249cb41079162e2e8c0cfb_2m0jq7haumkx1wa8jmb57jjsli4fzt1x.png","name":"2m0jq7haumkx1wa8jmb57jjsli4fzt1x.png","size":63703,"type":"image/png"},{"id":"6a249cb41079162e2e8c0cfd","url":"https://proxy-seller.com/files/fs/image/6a249cb41079162e2e8c0cfd_7bodw4ot6xqccbzpyy403kib9n4bgdfs.png","name":"7bodw4ot6xqccbzpyy403kib9n4bgdfs.png","size":23380,"type":"image/png"},{"id":"6a249cb51079162e2e8c0cff","url":"https://proxy-seller.com/files/fs/image/6a249cb51079162e2e8c0cff_3ok6yzgo8kve6a4onth8ck879vgc2yqd.png","name":"3ok6yzgo8kve6a4onth8ck879vgc2yqd.png","size":53676,"type":"image/png"},{"id":"6a249cb51079162e2e8c0d00","url":"https://proxy-seller.com/files/fs/image/6a249cb51079162e2e8c0d00_qyl859i4rkv15rg6ns5kg2b0wdtar782.png","name":"qyl859i4rkv15rg6ns5kg2b0wdtar782.png","size":68274,"type":"image/png"},{"id":"6a249cb51079162e2e8c0d01","url":"https://proxy-seller.com/files/fs/image/6a249cb51079162e2e8c0d01_vwe77b4qdr27e602l6ti8752qzrzovp3.png","name":"vwe77b4qdr27e602l6ti8752qzrzovp3.png","size":82369,"type":"image/png"},{"id":"6a249cb51079162e2e8c0d02","url":"https://proxy-seller.com/files/fs/image/6a249cb51079162e2e8c0d02_c0g53zxpig2iqr9ta78tw2klkdjp4hj6.png","name":"c0g53zxpig2iqr9ta78tw2klkdjp4hj6.png","size":87902,"type":"image/png"},{"id":"6a249cb51079162e2e8c0d03","url":"https://proxy-seller.com/files/fs/image/6a249cb51079162e2e8c0d03_d6w15fv3uvxlbscwc3ka4x0x9z6t4lbh.png","name":"d6w15fv3uvxlbscwc3ka4x0x9z6t4lbh.png","size":254353,"type":"image/png"},{"id":"6a249cb51079162e2e8c0d04","url":"https://proxy-seller.com/files/fs/image/6a249cb51079162e2e8c0d04_aag07ba25ln7b6ql5i6avzgcjggw1mqc.png","name":"aag07ba25ln7b6ql5i6avzgcjggw1mqc.png","size":19283,"type":"image/png"},{"id":"6a249cb51079162e2e8c0d05","url":"https://proxy-seller.com/files/fs/image/6a249cb51079162e2e8c0d05_m6qyq9olvwttivlclpb03lbfq58fa7fu.png","name":"m6qyq9olvwttivlclpb03lbfq58fa7fu.png","size":25675,"type":"image/png"},{"id":"6a249cb51079162e2e8c0d06","url":"https://proxy-seller.com/files/fs/image/6a249cb51079162e2e8c0d06_i5z6mvx9ollo2wb17q6v6vzsehfdp7cw.png","name":"i5z6mvx9ollo2wb17q6v6vzsehfdp7cw.png","size":77147,"type":"image/png"}],"mainImg":{"id":"6a249cb41079162e2e8c0cdf","url":"https://proxy-seller.com/files/fs/image/6a249cb41079162e2e8c0cdf_openbullet_2_overview_of_the_web_scraping_tool_mainImg.png","name":"openbullet_2_overview_of_the_web_scraping_tool_mainImg.png","size":40856,"type":"image/png"},"popular":true,"localeCode":"en","mainImageAlt":"openbullet_2_overview_of_the_web_scraping_tool_img","metaTitle":"OpenBullet Review: Web Scraping & Proxy Setup","metaDescription":"Master web scraping with Openbullet: A detailed review and proxy setup guide."}]},"cta":{"block":"cta","type":"ipv4","label":"PROXIES FROM $0.02/IP","title":"Ready to put this into practice?","description":"Pick a proxy type and location, and start using it within minutes.","buttonLabel":"See pricing","to":"/price/","features":["Instant delivery","24/7 support","Refund policy"]}}},"actionData":null,"errors":null}