04-crawler-beautifulsoup.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "\n",
    "# 第三章 数据抓取\n",
    "\n",
    "Requests和Beautifulsoup简介\n",
    "\n",
    "\n",
    "![image.png](./images/author.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-06-08T14:27:07.092873Z",
     "start_time": "2019-06-08T14:27:07.089169Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 基本原理\n",
    "\n",
    "爬虫就是请求网站并提取数据的自动化程序。其中请求，提取，自动化是爬虫的关键！爬虫的基本流程：\n",
    "\n",
    "- 发起请求\n",
    "    - 通过HTTP库向目标站点发起请求，也就是发送一个Request，请求可以包含额外的header等信息，等待服务器响应\n",
    "\n",
    "- 获取响应内容\n",
    "    - 如果服务器能正常响应，会得到一个Response。Response的内容便是所要获取的页面内容，类型可能是HTML、Json字符串、二进制数据（图片或者视频）等类型\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- 解析内容\n",
    "    - 得到的内容可能是HTML,可以用页面解析库、正则表达式进行解析；可能是Json,可以直接转换为Json对象解析；可能是二进制数据，可以做保存或者进一步的处理\n",
    "\n",
    "- 保存数据\n",
    "    - 保存形式多样，可以存为文本，也可以保存到数据库，或者保存特定格式的文件\n",
    "\n",
    "浏览器发送消息给网址所在的服务器，这个过程就叫做**Http Request**;服务器收到浏览器发送的消息后，能够根据浏览器发送消息的内容，做相应的处理，然后把消息回传给浏览器，这个过程就是**Http Response**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 需要解决的问题 \n",
    "\n",
    "- 页面解析\n",
    "- 获取Javascript隐藏源数据\n",
    "- 自动翻页\n",
    "- 自动登录\n",
    "- 连接API接口\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "一般的数据抓取，使用requests和beautifulsoup配合就可以了。\n",
    "- 尤其是对于翻页时url出现规则变化的网页，只需要处理规则化的url就可以了。\n",
    "- 以简单的例子是抓取天涯论坛上关于某一个关键词的帖子。\n",
    "    - 在天涯论坛，关于雾霾的帖子的第一页是：\n",
    "http://bbs.tianya.cn/list.jsp?item=free&nextid=0&order=8&k=雾霾\n",
    "    - 第二页是：\n",
    "http://bbs.tianya.cn/list.jsp?item=free&nextid=1&order=8&k=雾霾\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 第一个爬虫\n",
    "\n",
    "![](images/alice.png)\n",
    "\n",
    "Beautifulsoup Quick Start \n",
    "\n",
    "http://www.crummy.com/software/BeautifulSoup/bs4/doc/\n",
    "\n",
    "\n",
    "https://socratesacademy.github.io/bigdata/data/test.html\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "\n",
    "'Once upon a time there were three little sisters,' the Dormouse began in a great hurry; 'and their names were Elsie, Lacie, and Tillie; and they lived at the bottom of a well--'\n",
    "\n",
    "<img src=\"./images/alice2.png\" align='right'>\n",
    "\n",
    "'What did they live on?' said Alice, who always took a great interest in questions of eating and drinking.\n",
    "\n",
    "'They lived on treacle,' said the Dormouse, after thinking a minute or two.\n",
    "\n",
    "'They couldn't have done that, you know,' Alice gently remarked; 'they'd have been ill.'\n",
    "\n",
    "'So they were,' said the Dormouse; 'very ill.'\n",
    "\n",
    "**Alice's Adventures in Wonderland** CHAPTER VII A Mad Tea-Party http://www.gutenberg.org/files/928/928-h/928-h.htm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:39:04.976348Z",
     "start_time": "2023-10-27T06:39:03.752852Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-11-03T03:32:12.203081Z",
     "start_time": "2020-11-03T03:32:09.078050Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```\n",
    "import requests\n",
    "from bs4 import BeautifulSoup \n",
    "\n",
    "url = 'https://vp.fact.qq.com/home'\n",
    "content = requests.get(url)\n",
    "soup = BeautifulSoup(content.text, 'html.parser') \n",
    "\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-11-03T03:49:07.016890Z",
     "start_time": "2020-11-03T03:49:07.013641Z"
    },
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function get in module requests.api:\n",
      "\n",
      "get(url, params=None, **kwargs)\n",
      "    Sends a GET request.\n",
      "    \n",
      "    :param url: URL for the new :class:`Request` object.\n",
      "    :param params: (optional) Dictionary, list of tuples or bytes to send\n",
      "        in the query string for the :class:`Request`.\n",
      "    :param \\*\\*kwargs: Optional arguments that ``request`` takes.\n",
      "    :return: :class:`Response <Response>` object\n",
      "    :rtype: requests.Response\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(requests.get)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:42:15.950230Z",
     "start_time": "2023-10-27T06:42:15.200863Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "url = 'https://socratesclub.github.io/bigdata/data/test.html'\n",
    "content = requests.get(url)\n",
    "#help(content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:42:26.158027Z",
     "start_time": "2023-10-27T06:42:26.154643Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html><head><title>The Dormouse's story</title></head>\n",
      "<body>\n",
      "<p class=\"title\"><b>The Dormouse's story</b></p>\n",
      "\n",
      "<p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
      "<a href=\"http://example.com/elsie\" class=\"sister\" id=\"link1\">Elsie</a>,\n",
      "<a href=\"http://example.com/lacie\" class=\"sister\" id=\"link2\">Lacie</a> and\n",
      "<a href=\"http://example.com/tillie\" class=\"sister\" id=\"link3\">Tillie</a>;\n",
      "and they lived at the bottom of a well.</p>\n",
      "\n",
      "<p class=\"story\">...</p>\n"
     ]
    }
   ],
   "source": [
    "print(content.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:43:00.786372Z",
     "start_time": "2023-10-27T06:43:00.777585Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'utf-8'"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "content.encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Beautiful Soup\n",
    "> Beautiful Soup is a Python library designed for quick turnaround projects like screen-scraping. Three features make it powerful:\n",
    "\n",
    "- Beautiful Soup provides a few simple methods. It doesn't take much code to write an application\n",
    "- Beautiful Soup automatically converts incoming documents to Unicode and outgoing documents to UTF-8. Then you just have to specify the original encoding.\n",
    "- Beautiful Soup sits on top of popular Python parsers like `lxml` and `html5lib`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Install beautifulsoup4\n",
    "\n",
    "open your terminal/cmd\n",
    "\n",
    "<del> $ pip install beautifulsoup4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### html.parser\n",
    "Beautiful Soup supports the html.parser included in Python’s standard library\n",
    "\n",
    "### lxml\n",
    "but it also supports a number of third-party Python parsers. One is the lxml parser `lxml`. Depending on your setup, you might install lxml with one of these commands:\n",
    "\n",
    "> $ apt-get install python-lxml\n",
    "\n",
    "> $ easy_install lxml\n",
    "\n",
    "> $ pip install lxml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### html5lib\n",
    "Another alternative is the pure-Python html5lib parser `html5lib`, which parses HTML the way a web browser does. Depending on your setup, you might install html5lib with one of these commands:\n",
    "\n",
    "> $ apt-get install python-html5lib\n",
    "\n",
    "> $ easy_install html5lib\n",
    "\n",
    "> $ pip install html5lib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:45:23.747336Z",
     "start_time": "2023-10-27T06:45:23.740775Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<html><head><title>The Dormouse's story</title></head>\n",
       "<body>\n",
       "<p class=\"title\"><b>The Dormouse's story</b></p>\n",
       "<p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
       "<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n",
       "<a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a> and\n",
       "<a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>;\n",
       "and they lived at the bottom of a well.</p>\n",
       "<p class=\"story\">...</p></body></html>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# url = 'http://socratesacademy.github.io/bigdata/data/test.html'\n",
    "# content = requests.get(url)\n",
    "content = content.text\n",
    "soup = BeautifulSoup(content, 'html.parser') \n",
    "soup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:45:50.190354Z",
     "start_time": "2023-10-27T06:45:50.186885Z"
    },
    "scrolled": true,
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html>\n",
      " <head>\n",
      "  <title>\n",
      "   The Dormouse's story\n",
      "  </title>\n",
      " </head>\n",
      " <body>\n",
      "  <p class=\"title\">\n",
      "   <b>\n",
      "    The Dormouse's story\n",
      "   </b>\n",
      "  </p>\n",
      "  <p class=\"story\">\n",
      "   Once upon a time there were three little sisters; and their names were\n",
      "   <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">\n",
      "    Elsie\n",
      "   </a>\n",
      "   ,\n",
      "   <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">\n",
      "    Lacie\n",
      "   </a>\n",
      "   and\n",
      "   <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">\n",
      "    Tillie\n",
      "   </a>\n",
      "   ;\n",
      "and they lived at the bottom of a well.\n",
      "  </p>\n",
      "  <p class=\"story\">\n",
      "   ...\n",
      "  </p>\n",
      " </body>\n",
      "</html>\n"
     ]
    }
   ],
   "source": [
    "print(soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- html\n",
    "    - head\n",
    "        - title\n",
    "    - body\n",
    "        - p (class = 'title', 'story' )\n",
    "            - a (class = 'sister')\n",
    "                - href/id"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Select 方法\n",
    "\n",
    "\n",
    "- 标签名不加任何修饰\n",
    "- 类名前加点\n",
    "- id名前加 #\n",
    "\n",
    "我们也可以利用这种特性，使用soup.select()方法筛选元素，返回类型是 list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "Select方法三步骤\n",
    "\n",
    "- Inspect (检查)\n",
    "- Copy\n",
    "    - Copy Selector\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "- 鼠标选中标题`The Dormouse's story`, 右键检查Inspect\n",
    "- 鼠标移动到选中的源代码\n",
    "- 右键Copy-->Copy Selector \n",
    "\n",
    "`body > p.title > b`\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:54:02.243539Z",
     "start_time": "2023-10-27T06:54:02.238944Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The Dormouse's story\""
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select('body > p.title > b')[0].text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Select 方法: 通过标签名查找"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:54:58.162310Z",
     "start_time": "2023-10-27T06:54:58.157979Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The Dormouse's story\""
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select('title')[0].text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:55:09.473384Z",
     "start_time": "2023-10-27T06:55:09.469013Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select('a') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:55:24.216571Z",
     "start_time": "2023-10-27T06:55:24.212141Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<b>The Dormouse's story</b>]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select('b')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Select 方法: 通过类名查找"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:56:10.151624Z",
     "start_time": "2023-10-27T06:56:10.146748Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
       " <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a> and\n",
       " <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>;\n",
       " and they lived at the bottom of a well.</p>,\n",
       " <p class=\"story\">...</p>]"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select('.story')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:56:28.766310Z",
     "start_time": "2023-10-27T06:56:28.761632Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select('.sister')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:56:38.557921Z",
     "start_time": "2023-10-27T06:56:38.553664Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p class=\"title\"><b>The Dormouse's story</b></p>]"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select('.title')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Select 方法: 通过id名查找"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:57:23.636680Z",
     "start_time": "2023-10-27T06:57:23.632347Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>]"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select('#link1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:57:49.380331Z",
     "start_time": "2023-10-27T06:57:49.376333Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://example.com/elsie'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select('#link1')[0]['href']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Select 方法: 组合查找\n",
    "\n",
    "将标签名、类名、id名进行组合\n",
    "\n",
    "- 例如查找 p 标签中，id 等于 link1的内容\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:58:36.289056Z",
     "start_time": "2023-10-27T06:58:36.284848Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select('p #link1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Select 方法:属性查找\n",
    "\n",
    "加入属性元素\n",
    "- 属性需要用大于号`>`连接\n",
    "- 属性和标签属于同一节点，中间不能加空格。\n",
    " \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:59:05.510790Z",
     "start_time": "2023-10-27T06:59:05.506829Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<title>The Dormouse's story</title>]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select(\"head > title\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T06:59:09.488507Z",
     "start_time": "2023-10-27T06:59:09.483832Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p class=\"title\"><b>The Dormouse's story</b></p>,\n",
       " <p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
       " <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a> and\n",
       " <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>;\n",
       " and they lived at the bottom of a well.</p>,\n",
       " <p class=\"story\">...</p>]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.select(\"body > p\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## find_all方法\n",
    "\n",
    "- 找到所有的：\n",
    "    - `find_all`\n",
    "    - `()`\n",
    "- 只找一个：\n",
    "    - `find`\n",
    "    - `.`\n",
    "- `find_all('tag name', {'class or id': 'value'})`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:00:21.412286Z",
     "start_time": "2023-10-27T07:00:21.408110Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p class=\"title\"><b>The Dormouse's story</b></p>,\n",
       " <p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
       " <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a> and\n",
       " <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>;\n",
       " and they lived at the bottom of a well.</p>,\n",
       " <p class=\"story\">...</p>]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all('p')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:00:30.564143Z",
     "start_time": "2023-10-27T07:00:30.559833Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p class=\"title\"><b>The Dormouse's story</b></p>,\n",
       " <p class=\"story\">Once upon a time there were three little sisters; and their names were\n",
       " <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a> and\n",
       " <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>;\n",
       " and they lived at the bottom of a well.</p>,\n",
       " <p class=\"story\">...</p>]"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup('p')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:00:36.932827Z",
     "start_time": "2023-10-27T07:00:36.928226Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<p class=\"title\"><b>The Dormouse's story</b></p>"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find('p')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:00:40.387915Z",
     "start_time": "2023-10-27T07:00:40.383875Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<p class=\"title\"><b>The Dormouse's story</b></p>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:16:10.609548Z",
     "start_time": "2023-10-27T07:16:10.605052Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"The Dormouse's story\",\n",
       " 'Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.',\n",
       " '...']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[i.text.replace('\\n', ' ') for i in soup('p')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:16:22.279642Z",
     "start_time": "2023-10-27T07:16:22.276497Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Dormouse's story\n",
      "Once upon a time there were three little sisters; and their names were Elsie, Lacie and Tillie; and they lived at the bottom of a well.\n",
      "...\n"
     ]
    }
   ],
   "source": [
    "for i in soup('p'):\n",
    "    print(i.text.replace('\\n', ' '))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:16:39.366957Z",
     "start_time": "2023-10-27T07:16:39.362896Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "html\n",
      "head\n",
      "title\n",
      "body\n",
      "p\n",
      "b\n",
      "p\n",
      "a\n",
      "a\n",
      "a\n",
      "p\n"
     ]
    }
   ],
   "source": [
    "for tag in soup.find_all(True):\n",
    "    print(tag.name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:16:48.535162Z",
     "start_time": "2023-10-27T07:16:48.531087Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<title>The Dormouse's story</title>]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup('title')  # or  soup.title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:17:04.982434Z",
     "start_time": "2023-10-27T07:17:04.978507Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'title'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.title.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:17:08.776478Z",
     "start_time": "2023-10-27T07:17:08.772583Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The Dormouse's story\""
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.title.string"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:17:15.294390Z",
     "start_time": "2023-10-27T07:17:15.290238Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\"The Dormouse's story\""
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.title.text\n",
    "# 推荐使用text方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:18:45.938662Z",
     "start_time": "2023-10-27T07:18:45.933985Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'head'"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.title.parent.name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:18:50.954972Z",
     "start_time": "2023-10-27T07:18:50.950915Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>]"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all('a')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:18:57.875477Z",
     "start_time": "2023-10-27T07:18:57.870101Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>]"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all('a', {'class':'sister'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:19:13.019608Z",
     "start_time": "2023-10-27T07:19:13.014093Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all('a', {'class': 'sister'})[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:19:16.587642Z",
     "start_time": "2023-10-27T07:19:16.583377Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Elsie'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all('a', {'class': 'sister'})[0].text "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:19:26.695222Z",
     "start_time": "2023-10-27T07:19:26.690211Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://example.com/elsie'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all('a', {'class':'sister'})[0]['href']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:19:31.648926Z",
     "start_time": "2023-10-27T07:19:31.644890Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'link1'"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all('a', {'class': 'sister'})[0]['id']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:19:47.945161Z",
     "start_time": "2023-10-27T07:19:47.940166Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>]"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all('a', {'id':'link1'})#[0]['id']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:19:56.409445Z",
     "start_time": "2023-10-27T07:19:56.404915Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<b>The Dormouse's story</b>,\n",
       " <a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/lacie\" id=\"link2\">Lacie</a>,\n",
       " <a class=\"sister\" href=\"http://example.com/tillie\" id=\"link3\">Tillie</a>]"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find_all([\"a\", \"b\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:20:05.096341Z",
     "start_time": "2023-10-27T07:20:05.091965Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<a class=\"sister\" href=\"http://example.com/elsie\" id=\"link1\">Elsie</a>"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "soup.find(id=\"link1\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-27T07:20:19.318473Z",
     "start_time": "2023-10-27T07:20:19.315615Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Dormouse's story\n",
      "\n",
      "The Dormouse's story\n",
      "Once upon a time there were three little sisters; and their names were\n",
      "Elsie,\n",
      "Lacie and\n",
      "Tillie;\n",
      "and they lived at the bottom of a well.\n",
      "...\n"
     ]
    }
   ],
   "source": [
    "print(soup.get_text())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![image.png](./images/end.png)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "幻灯片",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 0,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "100px",
    "left": "1287.36px",
    "top": "0px",
    "width": "130.656px"
   },
   "toc_section_display": false,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}