使用Python的Scrapy框架编写web爬虫的简单示例

[ ... Scrapy log here ... ]

[s] Available Scrapy objects:
[s] 2010-08-19 21:45:59-0300 [default] INFO: Spider closed (finished)
[s] hxs  <HtmlXPathSelector (http://www.dmoz.org/Computers/Programming/Languages/Python/Books/) xpath=None>
[s] item  Item()
[s] request <GET http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
[s] response <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
[s] spider  <BaseSpider 'default' at 0x1b6c2d0>
[s] xxs  <XmlXPathSelector (http://www.dmoz.org/Computers/Programming/Languages/Python/Books/) xpath=None>
[s] Useful shortcuts:
[s] shelp()   Print this help
[s] fetch(req_or_url) Fetch a new request or URL and update shell objects
[s] view(response) View response in a browser

In [1]:

In [1]: hxs.select('//title')
Out[1]: [<HtmlXPathSelector (title) xpath=//title>]

In [2]: hxs.select('//title').extract()
Out[2]: [u'<title>Open Directory - Computers: Programming: Languages: Python: Books</title>']

In [3]: hxs.select('//title/text()')
Out[3]: [<HtmlXPathSelector (text) xpath=//title/text()>]

In [4]: hxs.select('//title/text()').extract()
Out[4]: [u'Open Directory - Computers: Programming: Languages: Python: Books']

In [5]: hxs.select('//title/text()').re('(\w+):')
Out[5]: [u'Computers', u'Programming', u'Languages', u'Python']

hxs.select('//ul/li')

hxs.select('//ul/li/text()').extract()

hxs.select('//ul/li/a/text()').extract()

hxs.select('//ul/li/a/@href').extract()

sites = hxs.select('//ul/li')
for site in sites:
 title = site.select('a/text()').extract()
 link = site.select('a/@href').extract()
 desc = site.select('text()').extract()
 print title, link, desc

Note

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
 
class DmozSpider(BaseSpider):
 name = "dmoz"
 allowed_domains = ["dmoz.org"]
 start_urls = [
  "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
  "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
 ]
 
 def parse(self, response):
  hxs = HtmlXPathSelector(response)
  sites = hxs.select('//ul/li')
  for site in sites:
   title = site.select('a/text()').extract()
   link = site.select('a/@href').extract()
   desc = site.select('text()').extract()
   print title, link, desc

scrapy crawl dmoz

> item = DmozItem()
> item['title'] = 'Example title'
> item['title']
'Example title'

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
 
from tutorial.items import DmozItem
 
class DmozSpider(BaseSpider):
 name = "dmoz"
 allowed_domains = ["dmoz.org"]
 start_urls = [
  "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
  "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
 ]
 
 def parse(self, response):
  hxs = HtmlXPathSelector(response)
  sites = hxs.select('//ul/li')
  items = []
  for site in sites:
   item = DmozItem()
   item['title'] = site.select('a/text()').extract()
   item['link'] = site.select('a/@href').extract()
   item['desc'] = site.select('text()').extract()
   items.append(item)
  return items

[dmoz] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
  {'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text, ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.\n],
  'link': [u'http://gnosis.cx/TPiP/'],
  'title': [u'Text Processing in Python']}
[dmoz] DEBUG: Scraped from <200 http://www.dmoz.org/Computers/Programming/Languages/Python/Books/>
  {'desc': [u' - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n'],
  'link': [u'http://www.informit.com/store/product.aspx"_blank" href="http://doc.scrapy.org/en/0.16/topics/feed-exports.html#topics-feed-exports">Feed exports,使用如下的命令:

 




scrapy crawl dmoz -o items.json -t json



它将会产生一个items.json文件,它包含所有抓取的items(序列化的JSON)。
在一些小的项目里(例如我们的教程中),那就足够啦。然而,假如你想要执行更多复杂的抓取items,你能够写一个 Item Pipeline。 因为在项目创建的时候，一个专门用于Item Pipelines的占位符文件已经随着项目一起被建立，目录在tutorial/pipelines.py。如果你只需要存取这些抓取后的items的话，就不需要去实现任何的条目管道。