How to Scrape YouTube Using Python: A Step-by-Step Guide

Scraping data from YouTube can be challenging due to its dynamic content and anti-scraping measures. However, with the right tools and techniques, you can efficiently extract useful information. In this article, we'll walk you through the process of scraping YouTube video data using Python, Playwright, and lxml.

Understanding Data Structure to Scrape YouTube with Python

Before we get to know how to scrape YouTube, we need to understand its structure. It has so many features available that have an endless array of data types to choose from pertaining to user activities and video statistics.

Some key parameters from the platform include:

• video titles and descriptions;
• the tags added;
• the amount of views, likes and comments;
• the channel and playlist information.

These elements are significant for content marketers and creators for assessing the videos’ performances and strategizing how to formulate video content.

API vs. Web Scraping

With the YouTube Data API, developers can get access to most of the metrics programmatically. The API also allows access to subscriber counts as well as videos on the channel, which provides a good amount of data for analysis and integration purposes.

Yet, there might be some particular elements that are impossible to get through the API and thus can only be retrieved via web scraping. For example, obtaining some detailed viewer engagement metric, such as the sentiment of their comments or the specific time when they engaged, would require some approaches to web scrape YouTube pages. This technique is usually more complicated and can have risks with the platform’s ever-evolving content rendition as well as their strict regulations on data scraping.

Analyzing YouTube and Defining Scraping Strategy

YouTube’s Data API offers valuable access but comes with significant limits. You get 10,000 units per day, with restrictions on request types. This quota quickly caps large-scale data collection needs. For projects requiring more extensive data, crawling (scraping) becomes necessary to bypass these API restrictions.

YouTube’s interface uses infinite scroll. As you scroll, the page sends requests that load more video data via JSON payloads.

• To understand this, open Chrome DevTools or Firefox Developer Tools and watch the Network tab.
• Filter by XHR or fetch requests to spot the URLs, methods (usually GET), and query parameters that load videos dynamically.

Dive into these JSON responses to learn their structure. They often contain nested fields with complex key naming conventions.

Selecting Links and Selectors

To pick reliable video links, inspect the page’s HTML structure using the Elements tab. Look for anchor tags with href attributes containing "watch" — e.g., a[href*="watch"]. This selector covers most video links, including normal videos and shorts, though playlists need special handling.

Test your selectors live in the browser console using JavaScript like document.querySelectorAll('a[href*="watch"]'). This helps verify you’re capturing the right links before coding your scraper.

Parsing YouTube’s complex JSON can be difficult. Instead, consider using Playwright for automated browser navigation. This lets you scrape data directly from the DOM without diving deep into JSON formats. Playwright can manage infinite scroll, clicks, and waits, simulating real user behavior.

Compliance matters:

• Detect cookie consent banners with Playwright, then either simulate consent clicks or set consent cookies programmatically.
• For regional variations and GDPR popups, use Playwright’s context and cookie persistence features to handle these automatically, keeping scraping smooth and compliant.

In the following blocks we are going to show you how to build a script and how to scrape data from YouTube in Python efficiently.

Using Proxies to Avoid Detection While Scraping YouTube

In order to scrape YouTube videos with Python, the use of proxies is essential to evade IP bans and bots' traversal prevention methods. Here are some types and their descriptions:

Residential Proxies

These are connected to genuine real IP addresses and are used as authentic connections for the websites. To scrape YouTube data, where trust is required to a large extent in order to not get caught, proxies are the best option. They make it possible for the scraper to behave like a genuine user. Hence, the chances of being detected as a bot are minimized.

ISP

These proxies provide the middle ground between residential IPs and datacenter proxies. They are provided by internet service providers, who issue authentic IP addresses, which are notoriously difficult to flag as proxies. This quality makes ISP proxies very effective in cases when there is a need to scrape YouTube search results, which need both authenticity and outstanding performance.

Datacenter (IPv4/IPv6)

Even though Datacenter ones boast the highest speeds, they can easily be identified by platforms like YouTube due to coming from large data centers. The risk of being blocked while scraping is high, even though they are efficient and easy to use. These types are best when the need for rapid data processing outweighs the risks posed by detection.

Mobile Proxies

These provide the most legitimate solution due to routing connections through mobile devices on cellular networks. Their use for scraping tasks is the most effective, as they are less likely to be blocked, as mobile IPs are often rotated by service providers, making mobile proxies far less likely to get flagged. But, it needs to be noted that their speed might be much lower than other types.

When using these types strategically, it is possible to scrape data from YouTube without being detected, allowing for continued access to data while abiding by the platform's Terms of Service. Also, understanding them properly will help you a lot when you need to find a proxy for scraping.

Environment Setup

Install the necessary libraries using pip:

pip install playwright
pip install lxml

Install the Playwright browser binaries:

playwright install

To just install Chromium browser binaries, use the following command:

 playwright install chromium

For web scraping YouTube data with Python, you'll primarily need the following libraries:

1. Playwright: A powerful library for automating headless browsers, enabling you to interact with web pages as if you were a real user;
2. lxml: A fast and feature-rich library for processing XML and HTML in Python, supporting XPath for querying documents;
3. CSV Module: A built-in Python library for saving extracted data into a CSV file.

Step 1: Import Required Libraries

import asyncio
from playwright.async_api import Playwright, async_playwright
from lxml import html
import csv

Step 2: Headless Browser Automation

Launch a headless browser with Playwright, navigate to the YouTube video URL, and wait for the page to fully load.

Scroll the page to load more comments.

browser = await playwright.chromium.launch(headless=True)
context = await browser.new_context()
page = await context.new_page()


# Navigating to the YouTube video URL
await page.goto("https://www.YouTube.com/watch?v=Ct8Gxo8StBU", wait_until="networkidle")


# Scrolling down to load more comments
for _ in range(20):
    await page.mouse.wheel(0, 200)
    await asyncio.sleep(0.2)


# Giving some time for additional content to load
await page.wait_for_timeout(1000)

Step 3: HTML Content Parsing

Extract the page's HTML content using Playwright and parse it with lxml.

# Extracting the page content
page_content = await page.content()


# Parsing the HTML content
parser = html.fromstring(page_content)

Step 4: Data Extraction

Extract the required data points (e.g., title, channel, comments) using XPath expressions.

Collect all relevant data, including video metadata and comments.

# Extracting video data
title = parser.xpath('//div[@id="title"]/h1/yt-formatted-string/text()')[0]
channel = parser.xpath('//yt-formatted-string[@id="text"]/a/text()')[0]
channel_link = 'https://www.YouTube.com' + parser.xpath('//yt-formatted-string[@id="text"]/a/@href')[0]
posted = parser.xpath('//yt-formatted-string[@id="info"]/span/text()')[2]
total_views = parser.xpath('//yt-formatted-string[@id="info"]/span/text()')[0]
total_comments = parser.xpath('//h2[@id="count"]/yt-formatted-string/span/text()')[0]
comments_list = parser.xpath('//yt-attributed-string[@id="content-text"]/span/text()')

Step 5: Saving Data

Save the extracted data into a CSV file for easy analysis and storage.

# Saving the data to a CSV file
with open('YouTube_video_data.csv', 'w', newline='', encoding='utf-8') as file:
    writer = csv.writer(file)
    writer.writerow(["Title", "Channel", "Channel Link", "Posted", "Total Views", "Total Comments", "Comments"])
    writer.writerow([title, channel, channel_link, posted, total_views, total_comments, ", ".join(comments_list)])

Proxy Implementation

Proxies play a crucial role in web scraping, especially when dealing with large-scale data extraction or sites with strict anti-bot measures like YouTube. Here's how proxies are implemented in the Playwright script:

Proxy setup:

1. The proxy parameter in playwright.chromium.launch() is used to route all browser traffic through a specified proxy for YouTube.
2. The proxy server details, including the server address, username, and password, must be configured.

browser = await playwright.chromium.launch(
        headless=True,
        proxy={"server": "http://your_proxy_ip:port", "username": "your_username", "password": "your_password"}
    )

Benefits of using proxies:

• IP masking: proxies hide your original IP address, reducing the likelihood of being blocked.
• Request distribution: by rotating proxies, you can distribute requests across different IP addresses, mimicking traffic from multiple users.
• Access restricted content: proxies can help bypass regional restrictions or access content that might be limited to certain IP ranges.

This implementation ensures your scraping activities are less likely to be detected and blocked by YouTube's anti-bot mechanisms.

Complete Code

Below is the complete code to scrape YouTube video data using Playwright and lxml, including proxy implementation.

import asyncio
from playwright.async_api import Playwright, async_playwright
from lxml import html
import csv


# Asynchronous function to run Playwright and extract data
async def run(playwright: Playwright) -> None:
    # Launching headless browser with proxy settings
    browser = await playwright.chromium.launch(
        headless=True,
        proxy={"server": "http://your_proxy_ip:port", "username": "your_username", "password": "your_password"}
    )
    context = await browser.new_context()
    page = await context.new_page()


    # Navigating to the YouTube video URL
    await page.goto("https://www.YouTube.com/watch?v=Ct8Gxo8StBU", wait_until="networkidle")


    # Scrolling down to load more comments
    for _ in range(20):
        await page.mouse.wheel(0, 200)
        await asyncio.sleep(0.2)
    
    # Giving some time for additional content to load
    await page.wait_for_timeout(1000)
    
    # Extracting the page content
    page_content = await page.content()


    # Closing browser
    await context.close()
    await browser.close()


    # Parsing the HTML content
    parser = html.fromstring(page_content)


    # Extracting video data
    title = parser.xpath('//div[@id="title"]/h1/yt-formatted-string/text()')[0]
    channel = parser.xpath('//yt-formatted-string[@id="text"]/a/text()')[0]
    channel_link = 'https://www.YouTube.com' + parser.xpath('//yt-formatted-string[@id="text"]/a/@href')[0]
    posted = parser.xpath('//yt-formatted-string[@id="info"]/span/text()')[2]
    total_views = parser.xpath('//yt-formatted-string[@id="info"]/span/text()')[0]
    total_comments = parser.xpath('//h2[@id="count"]/yt-formatted-string/span/text()')[0]
    comments_list = parser.xpath('//yt-attributed-string[@id="content-text"]/span/text()')


    # Saving the data to a CSV file
    with open('YouTube_video_data.csv', 'w', newline='', encoding='utf-8') as file:
        writer = csv.writer(file)
        writer.writerow(["Title", "Channel", "Channel Link", "Posted", "Total Views", "Total Comments", "Comments"])
        writer.writerow([title, channel, channel_link, posted, total_views, total_comments, ", ".join(comments_list)])


# Running the asynchronous function
async def main():
    async with async_playwright() as playwright:
        await run(playwright)


asyncio.run(main())

Configuring Crawlee for YouTube

Setting up Crawlee correctly ensures your YouTube scraper python runs efficiently and safely.

Concurrency and Rate Limits

• Use Crawlee’s ConcurrencySettings to limit your request rate.
• Set max_tasks_per_minute=50 to mimic human behavior and avoid IP bans.
• Control data volume and runtime with the max_items parameter, limiting videos per channel. This balances thoroughness with performance.

PlaywrightCrawler Settings

Configure PlaywrightCrawler with these key settings:

• Run headless for speed.
• Set request timeout to 120 seconds to handle slow loads.
• Spoof user agents to appear as various browsers.

Optimization via Request Interception

Boost performance by blocking non-essential requests, such as:

• Images (.webp, .jpg, .png, .svg).
• Fonts (.woff, .ttf).
• PDF and ZIP files.
• Analytics scripts.

Use request interception to drop these early, saving bandwidth and time.

Session Persistence and Data Transfer

Leverage pre-navigation hooks to restore cookies and local storage from previous runs. This bypasses GDPR prompts, speeding up repeated crawls.

Transport max_items and the channel list using the user_data field in Request objects. It keeps data consistent between crawler stages.

Set up proxies with Crawlee’s ProxyConfiguration class for IP rotation and geo-targeting. Integrate Proxy-Seller’s proxy services here. Proxy-Seller offers:

• huge proxy network with residential, ISP, datacenter IPv4/IPv6, and mobile proxies.
• High speeds up to 1 Gbps and 99% uptime.
• Coverage across 220+ countries with flexible MIX packages.
• Authentication via username/password or IP whitelist.
• SOCKS5 and HTTP(S) support for seamless PlaywrightCrawler integration.
• 24/7 customer support and API access for automation.

Using Proxy-Seller helps you avoid IP bans and geographic blocks, vital for large-scale YouTube comment scraper Python or YouTube channel scraper Python projects. It supports ethical sourcing and compliance with privacy regulations, reducing risks during data scraping.

Extracting YouTube Data

To truly harness a YouTube video scraper Python, implement infinite scrolling within Crawlee’s async task handlers.

Scrolling and Link Extraction

• Use Crawlee’s built-in infinite_scroll helper to scroll down, wait for new content, then repeat until reaching max_items or no more videos load.
• Extract video links with extract_links using the selector a[href*="watch"]. Label these links as "video" to organize the queue.
• You might encounter URLs starting with consent.youtube.com. Use transform_request_function to rewrite these dynamically to www.youtube.com, preventing authorization or redirect errors.
• Enqueue only new, unique video URLs. Keep track of visited URLs to avoid duplicates. Also, limit videos enqueued per channel by counting links against max_items.

Metadata Extraction

In the video page handler, extract metadata from window.ytInitialPlayerResponse, available in the page context. Gather:

• Title, description, author.
• Video ID, channel ID.
• Duration (parse ISO 8601 format).
• Keywords.
• View count.
• Like count (via additional DOM checks or XHR requests).
• Shorts eligibility.
• Publish date.

Transcript Scraping

For transcripts, simulate clicking the subtitles button.

• Intercept network requests with Playwright event listeners to capture transcript URLs.
• Rewrite these URLs by removing the fmt parameter to get XML-formatted transcripts, which parse more robustly.
• Enqueue transcript URL requests with video metadata in user_data for context association.

In the transcript handler:

• Parse the XML response using xml.etree.ElementTree.
• Extract all <text> elements, handle HTML entities within.
• Concatenate text into a clean transcript string.
• Append transcripts to the video metadata dictionary.

Data Storage and Export

Save combined data using Crawlee’s Dataset API via Dataset.push_data() or local storage hooks. Handle XML parsing errors gracefully with warnings or empty transcripts fallback.

For data export, choose one:

• Save JSON files locally with timestamped filenames for version control.
• Use Apify platform Dataset storage for automatic persistence and UI export options.

Below is an example data schema combining video metadata and transcripts for easy downstream analysis:

JSON
{
"videoId": "abc123",
"title": "Example Video Title",
"description": "Video description here.",
"author": "Channel Name",
"channelId": "channel456",
"duration": "PT10M15S",
"keywords": ["keyword1", "keyword2"],
"viewCount": 12345,
"likeCount": 678,
"isShort": false,
"publishDate": "2024-01-15",
"transcript": "Full transcript text concatenated here."
}

Following these steps, you’ll build a powerful YouTube scraper Python, suitable for extracting large volumes of video and comment data with accuracy and compliance.

In Conclusion

When setting up an environment for scraping data from YouTube, it's crucial to focus on using proxies to circumvent platform restrictions effectively. Carefully selecting proxy servers is essential to minimize blocking risks and ensure operation anonymity. Static ISP proxies are highly recommended for their fast connection speeds and stability. Additionally, residential proxies offer dynamic IP addresses with a high trust factor, making them less likely to be flagged by YouTube’s security systems. It's also vital to adhere to ethical standards during data collection to prevent violations of YouTube's terms of service.