How to scrape YouTube using Python: A Step-by-step guide

Scraping data from YouTube can be challenging due to its dynamic content and anti-scraping measures. However, with the right tools and techniques, you can efficiently extract useful information. In this article, we'll walk you through the process of scraping YouTube video data using Python, Playwright, and lxml.

Understanding Data Structure to Scrape Youtube with Python

Before we get to know how to scrape YouTube we need to understand its structure. It has so many features available that have an endless array of data types to choose from pertaining to user activities and video statistics. Some key parameters from the platform include video titles and descriptions, the tags added, amount of views, likes and comments, as well as the channel and playlist information. These elements are significant for content marketers and creators for assessing the videos’ performances and strategizing how to formulate video content.

With the YouTube Data API, developers can get access to most of the metrics programmatically. The API also allows access to subscriber counts as well as videos on the channel which provides a good amount of data for analysis and integration purposes.

Yet, there might be some particular elements that are impossible to get through the API and thus can be only retrieved via web scraping. For example, obtaining some detailed viewer engagement metric, such as the sentiment of their comments or the specific time when they engaged, would require some approaches to web scrape Youtube pages. This technique is usually more complicated and can have risks with platform’s ever-evolving content rendition as well as their strict regulations on data scraping.

In the following blocks we are going to show you how to build a script and how to scrape data from Youtube in Python efficiently.

Using Proxies to Avoid Detection While Scraping YouTube

In order to scrape Youtube videos with Python, the use of proxies is essential to evade IP bans and bots traversal prevention methods. Here are some types and their descriptions:

1. Residential proxies are connected to genuine real IP addresses and are used as authentic connections for the websites. To scrape Youtube data, where trust is required to a large extent in order to not get caught, proxies are the best option. They make it possible for the scraper to behave like a genuine user, hence the chances of being detected as a bot are minimized.
2. ISP proxies provide the middle ground between residential IPs and datacenter proxies. They are provided by internet service providers, who issue authentic IP addresses, which are notoriously difficult to flag as proxies. This quality makes ISP proxies very effective in cases when need to scrape Youtube search results, which need both authenticity and outstanding performance.
3. Even though Datacenter ones boast the highest speeds, they can easily be identified by platforms like YouTube due to coming from large data centers. The risk of being blocked while scraping is high, even though they are efficient and easy to use. These types are best when the need for rapid data processing outweighs the risks posed by detection.
4. Mobile proxies provide the most legitimate solution due to routing connections through mobile devices on cellular networks. Their use for scraping tasks is the most effective as they are less likely to be blocked, as mobile IPs are often rotated by service providers, making mobile proxies far less likely to get flagged. But, it needs to be noted that their speed might be much lower than other types.

When using these types strategically, it is possible to scrape data from Youtube without being detected, allowing for continued access to data while abiding to the platform's Terms of Service. Also, understanding them properly will help you a lot when you will need to find a proxy for scraping.

Environment setup

Install the necessary libraries using pip:

pip install playwright 
pip install lxml

Install the Playwright browser binaries:

playwright install

To just install Chromium browser binaries use the following command:

playwright install chromium

For web scraping YouTube data with Python, you'll primarily need the following libraries:

1. Playwright: A powerful library for automating headless browsers, enabling you to interact with web pages as if you were a real user;
2. lxml: A fast and feature-rich library for processing XML and HTML in Python, supporting XPath for querying documents;
3. CSV Module: A built-in Python library for saving extracted data into a CSV file.

Step 1: Import required libraries

import asyncio
from playwright.async_api import Playwright, async_playwright
from lxml import html
import csv

Step 2: Headless browser automation

Launch a headless browser with Playwright, navigate to the YouTube video URL, and wait for the page to fully load.

Scroll the page to load more comments.

browser = await playwright.chromium.launch(headless=True)
context = await browser.new_context()
page = await context.new_page()

# Navigating to the YouTube video URL
await page.goto("https://www.youtube.com/watch?v=Ct8Gxo8StBU", wait_until="networkidle")

# Scrolling down to load more comments
for _ in range(20):
    await page.mouse.wheel(0, 200)
    await asyncio.sleep(0.2)

# Giving some time for additional content to load
await page.wait_for_timeout(1000)

Step 3: HTML content parsing

Extract the page's HTML content using Playwright and parse it with lxml.

# Extracting the page content
page_content = await page.content()

# Parsing the HTML content
parser = html.fromstring(page_content)

Step 4: Data extraction

Extract the required data points (e.g., title, channel, comments) using XPath expressions.

Collect all relevant data, including video metadata and comments.

# Extracting video data
title = parser.xpath('//div[@id="title"]/h1/yt-formatted-string/text()')[0]
channel = parser.xpath('//yt-formatted-string[@id="text"]/a/text()')[0]
channel_link = 'https://www.youtube.com' + parser.xpath('//yt-formatted-string[@id="text"]/a/@href')[0]
posted = parser.xpath('//yt-formatted-string[@id="info"]/span/text()')[2]
total_views = parser.xpath('//yt-formatted-string[@id="info"]/span/text()')[0]
total_comments = parser.xpath('//h2[@id="count"]/yt-formatted-string/span/text()')[0]
comments_list = parser.xpath('//yt-attributed-string[@id="content-text"]/span/text()')

Step 5: Saving data

Save the extracted data into a CSV file for easy analysis and storage.

# Saving the data to a CSV file
with open('youtube_video_data.csv', 'w', newline='', encoding='utf-8') as file:
    writer = csv.writer(file)
    writer.writerow(["Title", "Channel", "Channel Link", "Posted", "Total Views", "Total Comments", "Comments"])
    writer.writerow([title, channel, channel_link, posted, total_views, total_comments, ", ".join(comments_list)])

Proxy implementation

Proxies play a crucial role in web scraping, especially when dealing with large-scale data extraction or sites with strict anti-bot measures like YouTube. Here's how proxies are implemented in the Playwright script:

Proxy setup:

1. The proxy parameter in playwright.chromium.launch() is used to route all browser traffic through a specified proxy server.
2. The proxy server details, including the server address, username, and password, must be configured.

browser = await playwright.chromium.launch(
        headless=True,
        proxy={"server": "http://your_proxy_ip:port", "username": "your_username", "password": "your_password"}
    )

Benefits of using proxies:

• IP masking: proxies hide your original IP address, reducing the likelihood of being blocked.
• Request distribution: by rotating proxies, you can distribute requests across different IP addresses, mimicking traffic from multiple users.
• Access restricted content: proxies can help bypass regional restrictions or access content that might be limited to certain IP ranges.

This implementation ensures your scraping activities are less likely to be detected and blocked by YouTube's anti-bot mechanisms.

Complete code implementation

Below is the complete code to scrape YouTube video data using Playwright and lxml, including proxy implementation.

import asyncio
from playwright.async_api import Playwright, async_playwright
from lxml import html
import csv

# Asynchronous function to run Playwright and extract data
async def run(playwright: Playwright) -> None:
    # Launching headless browser with proxy settings
    browser = await playwright.chromium.launch(
        headless=True,
        proxy={"server": "http://your_proxy_ip:port", "username": "your_username", "password": "your_password"}
    )
    context = await browser.new_context()
    page = await context.new_page()

    # Navigating to the YouTube video URL
    await page.goto("https://www.youtube.com/watch?v=Ct8Gxo8StBU", wait_until="networkidle")

    # Scrolling down to load more comments
    for _ in range(20):
        await page.mouse.wheel(0, 200)
        await asyncio.sleep(0.2)
    
    # Giving some time for additional content to load
    await page.wait_for_timeout(1000)
    
    # Extracting the page content
    page_content = await page.content()

    # Closing browser
    await context.close()
    await browser.close()

    # Parsing the HTML content
    parser = html.fromstring(page_content)

    # Extracting video data
    title = parser.xpath('//div[@id="title"]/h1/yt-formatted-string/text()')[0]
    channel = parser.xpath('//yt-formatted-string[@id="text"]/a/text()')[0]
    channel_link = 'https://www.youtube.com' + parser.xpath('//yt-formatted-string[@id="text"]/a/@href')[0]
    posted = parser.xpath('//yt-formatted-string[@id="info"]/span/text()')[2]
    total_views = parser.xpath('//yt-formatted-string[@id="info"]/span/text()')[0]
    total_comments = parser.xpath('//h2[@id="count"]/yt-formatted-string/span/text()')[0]
    comments_list = parser.xpath('//yt-attributed-string[@id="content-text"]/span/text()')

    # Saving the data to a CSV file
    with open('youtube_video_data.csv', 'w', newline='', encoding='utf-8') as file:
        writer = csv.writer(file)
        writer.writerow(["Title", "Channel", "Channel Link", "Posted", "Total Views", "Total Comments", "Comments"])
        writer.writerow([title, channel, channel_link, posted, total_views, total_comments, ", ".join(comments_list)])

# Running the asynchronous function
async def main():
    async with async_playwright() as playwright:
        await run(playwright)

asyncio.run(main())

When setting up an environment for scraping data from YouTube, it's crucial to focus on using proxies to circumvent platform restrictions effectively. Carefully selecting proxy servers is essential to minimize blocking risks and ensure operation anonymity. Static ISP proxies are highly recommended for their fast connection speeds and stability. Additionally, residential proxies offer dynamic IP addresses with a high trust factor, making them less likely to be flagged by YouTube’s security systems. It's also vital to adhere to ethical standards during data collection to prevent violations of YouTube's terms of service.