Understanding Why Your Scrapy Spider Crawls Without Yielding Data

If your Scrapy spider connects successfully to target pages but finishes with 'items_per_minute': 0.0 and returns zero items, you are not alone. This is a common issue for developers learning web scraping with Scrapy. Inspecting the Scrapy stats log usually reveals the root cause quickly: a silent or caught exception inside your parsing loop.

The Core Problem: Unhandled Exceptions in the Loop

Look closely at your terminal output stats:

'log_count/ERROR': 1,
'spider_exceptions/KeyError': 1,

Scrapy encountered an unhandled KeyError inside your parse method while processing the very first item. When an exception occurs inside the for book in books: loop, execution halts immediately for that response, preventing any items from being yielded.

Analyzing the Buggy Code

There are two primary issues in the original code causing this behavior:

1. Incorrect CSS Selector and Attribute Lookup

In the original code snippet, the URL extraction line looks like this:

'url': book.css('next.a::text').attrib['href']

There are two problems here:

  • Invalid CSS Selector: next.a::text is attempting to select a element named <next> with a child <a>, which does not exist on books.toscrape.com. Inside article.product_pod, the title and book link are located inside h3 > a.
  • Attribute Access on Text Nodes: Using ::text selects the text inside an HTML element. Text nodes do not have HTML attributes like href. Calling .attrib['href'] on a text selection or empty selection raises a KeyError.

2. Indentation Issue with Pagination

Notice the pagination logic:

if next_page is not None:
    if 'catalogue/' in next_page:
        next_page_url = 'https://books.toscrape.com/catalogue/' + next_page
    else:
        next_page_url = 'https://books.toscrape.com/' + next_page
        yield response.follow(next_page_url, callback=self.parse)

The yield response.follow(...) statement is nested inside the else block. If the next_page string contains 'catalogue/', the URL is constructed, but response.follow is never executed!

The Solution: Corrected Scrapy Spider

Here is the clean, fixed version of the spider using robust standard selectors and response.follow for simple relative links:

import scrapy

class BookspiderSpider(scrapy.Spider):
    name = "bookspider"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com/"]

    def parse(self, response):
        books = response.css('article.product_pod')
        for book in books:
            yield {
                'title': book.css('h3 a::attr(title)').get() or book.css('h3 a::text').get(),
                'price': book.css('.product_price .price_color::text').get(),
                'url': book.css('h3 a::attr(href)').get(),
            }

        next_page = response.css('li.next a::attr(href)').get()
        if next_page is not None:
            # response.follow automatically handles relative URLs safely
            yield response.follow(next_page, callback=self.parse)

Key Takeaways for Scrapy Debugging

  • Check Stats for Exceptions: Always review the Scrapy summary log at the end of a run for spider_exceptions or ERROR counts.
  • Use Safe Extraction Methods: Use ::attr(href) with .get() rather than direct dictionary access on .attrib when elements might be missing.
  • Leverage response.follow: You don't need manual string concatenation to build full pagination URLs; Scrapy's response.follow handles relative path resolution automatically.