Python Scrapy: Extracting Data — XPath Method

Besides CSS, Scrapy selectors also support using XPath expressions:

response.xpath("//title")
[<Selector query='//title' data=''>]
response.xpath("//title/text()").get()
'Quotes to Scrape'

XPath expressions are a powerful tool for selecting elements within HTML or XML documents. They provide a more flexible and expressive language compared to CSS selectors.

While CSS selectors are often used due to their simplicity and familiarity, XPath offers several advantages:

• Content-based selection: XPath allows you to select elements based on their text content, attributes, or other properties beyond just their structure.
• Hierarchical navigation: XPath provides a hierarchical way to traverse the document tree, making it easier to select elements within nested structures.
• XPath is the foundation: Scrapy Selectors are built on top of XPath, so understanding XPath can provide for a deeper understanding of how Scrapy works. In fact, CSS selectors are converted to XPath under the hood, a detail you can observe by closely examining the text representation of selector objects in the shell.