04-crawler-gov-report.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "\n",
    "\n",
    "# 抓取历届政府工作报告\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:31:27.015108Z",
     "start_time": "2021-11-01T06:31:26.783392Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "www.hprc.org.cn/wxzl/wxysl/lczf/\n",
    "\n",
    "另外一个镜像\n",
    "\n",
    "http://hprc.cssn.cn/wxzl/wxysl/lczf/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## Inspect\n",
    "\n",
    "<td width=\"274\" class=\"bl\">·&nbsp;<a href=\"./d12qgrdzfbg/201603/t20160318_369509.html\" target=\"_blank\" title=\"2016年政府工作报告\">2016年政府工作报告</a></td>\n",
    "\n",
    "    <td width=\"274\" class=\"bl\">·&nbsp;<a href=\"./d12qgrdzfbg/201603/t20160318_369509.html\" target=\"_blank\" title=\"2016年政府工作报告\">2016年政府工作报告</a></td>\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:32:01.111759Z",
     "start_time": "2021-11-01T06:32:00.817730Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ISO-8859-1'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# get the link for each year\n",
    "url = \"http://www.hprc.org.cn/wxzl/wxysl/lczf/\" \n",
    "content = requests.get(url)\n",
    "content.encoding"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Encoding\n",
    "\n",
    "- ASCII\n",
    "    - 7位字符集\n",
    "    - 美国标准信息交换代码（American Standard Code for Information Interchange）的缩写, 为美国英语通信所设计。\n",
    "    - 它由128个字符组成，包括大小写字母、数字0-9、标点符号、非打印字符（换行符、制表符等4个）以及控制字符（退格、响铃等）组成。\n",
    "- iso8859-1 通常叫做Latin-1。\n",
    "    - 和ascii编码相似。\n",
    "    - 属于单字节编码，最多能表示的字符范围是0-255，应用于英文系列。比如，字母a的编码为0x61=97。 \n",
    "    - 无法表示中文字符。\n",
    "    - 单字节编码，和计算机最基础的表示单位一致，所以很多时候，仍旧使用iso8859-1编码来表示。在很多协议上，默认使用该编码。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- gb2312/gbk/gb18030\n",
    "    - 是汉字的国标码，专门用来表示汉字，是双字节编码，而英文字母和iso8859-1一致（兼容iso8859-1编码）。\n",
    "    - 其中gbk编码能够用来同时表示繁体字和简体字,K 为汉语拼音 Kuo Zhan（扩展）中“扩”字的声母\n",
    "    - gb2312只能表示简体字，gbk是兼容gb2312编码的。 \n",
    "    - gb18030，全称：国家标准 GB 18030-2005《信息技术中文编码字符集》，是中华人民共和国现时最新的内码字集"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "- unicode \n",
    "    - 最统一的编码，用来表示所有语言的字符。\n",
    "    - 占用更多的空间，定长双字节（也有四字节的）编码，包括英文字母在内。\n",
    "    - 不兼容iso8859-1等其它编码。相对于iso8859-1编码来说，uniocode编码只是在前面增加了一个0字节，比如字母a为\"00 61\"。 \n",
    "    - 定长编码便于计算机处理（注意GB2312/GBK不是定长编码），unicode又可以用来表示所有字符，所以在很多软件内部是使用unicode编码来处理的，比如java。 \n",
    "- UTF \n",
    "    - unicode不便于传输和存储，产生了utf编码\n",
    "    - utf编码兼容iso8859-1编码，同时也可以用来表示所有语言的字符\n",
    "    - utf编码是不定长编码，每一个字符的长度从1-6个字节不等。\n",
    "    - 其中，utf8（8-bit Unicode Transformation Format）是一种针对Unicode的可变长度字符编码，又称万国码。\n",
    "        - 由Ken Thompson于1992年创建。现在已经标准化为RFC 3629。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### decode\n",
    "<del>urllib2.urlopen(url).read().decode('gb18030') </del>\n",
    "\n",
    "    content.encoding = 'gb18030'\n",
    "    \n",
    "    content = content.text\n",
    "  \n",
    "Or\n",
    "\n",
    "    content = content.text.encode(content.encoding).decode('gb18030')\n",
    "\n",
    "\n",
    "\n",
    "### html.parser\n",
    "BeautifulSoup(content, 'html.parser')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:33:18.765890Z",
     "start_time": "2021-11-01T06:33:18.763331Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "# Specify the encoding\n",
    "content.encoding = 'utf8' # 'gb18030'\n",
    "content = content.text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:34:43.405715Z",
     "start_time": "2021-11-01T06:34:43.359495Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<a href=\"./dssjqgrmdbdh_1/202203/t20220313_5398415.html\" target=\"_blank\" title=\"2022年政府工作报告\">2022年政府工作报告</a>\n"
     ]
    }
   ],
   "source": [
    "soup = BeautifulSoup(content, 'html.parser') \n",
    "# links = soup.find_all('td', {'class', 'bl'})  \n",
    "# print(links[0]('a')[0]['href'])\n",
    "links = soup.select('.bl a')\n",
    "print(links[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:35:21.742972Z",
     "start_time": "2021-11-01T06:35:21.739110Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "53"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(links)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://www.hprc.org.cn/wxzl/wxysl/lczf/'"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:35:26.389635Z",
     "start_time": "2021-11-01T06:35:26.385247Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'./dishiyijie_10/200908/t20090818_3955459.html'"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links[-1]['href']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:35:36.538538Z",
     "start_time": "2021-11-01T06:35:36.534896Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'dssjqgrmdbdh_1/202203/t20220313_5398415.html'"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "links[0]['href'].split('./')[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:35:52.130583Z",
     "start_time": "2021-11-01T06:35:52.127404Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://www.hprc.org.cn/wxzl/wxysl/lczf/dssjqgrmdbdh_1/202203/t20220313_5398415.html\n"
     ]
    }
   ],
   "source": [
    "print(url + links[0]['href'].split('./')[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:36:35.683836Z",
     "start_time": "2021-11-01T06:36:35.678756Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['http://www.hprc.org.cn/wxzl/wxysl/lczf/dssjqgrmdbdh_1/202203/t20220313_5398415.html',\n",
       " 'http://www.hprc.org.cn/wxzl/wxysl/lczf/dssjqgrmdbdh_1/202103/t20210316_5318622.html',\n",
       " 'http://www.hprc.org.cn/wxzl/wxysl/lczf/dssjqgrmdbdh_1/202006/t20200604_5138985.html',\n",
       " 'http://www.hprc.org.cn/wxzl/wxysl/lczf/dssjqgrmdbdh_1/201903/t20190318_4849567.html',\n",
       " 'http://www.hprc.org.cn/wxzl/wxysl/lczf/dssjqgrmdbdh_1/201803/t20180323_4240852.html']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hyperlinks = [url + i['href'].split('./')[1] for i in links]\n",
    "hyperlinks[:5] "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:36:40.584311Z",
     "start_time": "2021-11-01T06:36:40.580807Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['http://www.hprc.org.cn/wxzl/wxysl/lczf/dishiyijie_9/200908/t20090818_3955464.html',\n",
       " 'http://www.hprc.org.cn/wxzl/wxysl/lczf/dishiyijie_10/200908/t20090818_3955462.html',\n",
       " 'http://www.hprc.org.cn/wxzl/wxysl/lczf/dishiyijie_10/200908/t20090818_3955461.html',\n",
       " 'http://www.hprc.org.cn/wxzl/wxysl/lczf/dishiyijie_10/200908/t20090818_3955460.html',\n",
       " 'http://www.hprc.org.cn/wxzl/wxysl/lczf/dishiyijie_10/200908/t20090818_3955459.html']"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hyperlinks[-5:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:36:51.848212Z",
     "start_time": "2021-11-01T06:36:51.845174Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://www.hprc.org.cn/wxzl/wxysl/lczf/dishiyijie_1/200908/t20090818_3955570.html\n"
     ]
    }
   ],
   "source": [
    "print(hyperlinks[15]) # 2007年有分页"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "### Inspect 下一页\n",
    "\n",
    "<a href=\"t20090818_27775_1.html\"><span style=\"color:#0033FF;font-weight:bold\">下一页</span></a>\n",
    "\n",
    "    <a href=\"t20090818_27775_1.html\"><span style=\"color:#0033FF;font-weight:bold\">下一页</span></a>\n",
    "    \n",
    "- a\n",
    "    - script\n",
    "        - td"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:39:57.985895Z",
     "start_time": "2021-11-01T06:39:57.648987Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "url_i = 'http://www.hprc.org.cn/wxzl/wxysl/lczf/dishiyijie_1/200908/t20090818_3955570.html'\n",
    "content = requests.get(url_i)\n",
    "content.encoding = 'utf8'\n",
    "content = content.text\n",
    "#content = content.text.encode(content.encoding).decode('gb18030')\n",
    "soup = BeautifulSoup(content, 'html.parser') \n",
    "#scripts = soup.find_all('script')\n",
    "#scripts[0]\n",
    "scripts = soup.select('td script')[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:40:02.091527Z",
     "start_time": "2021-11-01T06:40:02.087478Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<script>\n",
       "\tvar currentPage = 0;//所在页从0开始\n",
       "\tvar prevPage = currentPage-1//上一页\n",
       "\tvar 下一页Page = currentPage+1//下一页\n",
       "\tvar countPage = 4//共多少页\n",
       "\t//document.write(\"共\"+countPage+\"页&nbsp;&nbsp;\");\n",
       "\t\n",
       "\t//循环\n",
       "\tvar num = 17;\n",
       "\tfor(var i=0+(currentPage-1-(currentPage-1)%num) ; i<=(num+(currentPage-1-(currentPage-1)%num))&&(i<countPage) ; i++){\n",
       "\t\tif(countPage >1){\n",
       "\t\t\tif(currentPage==i)\n",
       "\t\t\t\tdocument.write(\"【<span style=\\\"color:#FF0000;\\\" class=\\\"hui14_30_h\\\">\"+(i+1)+\"</span>】&nbsp;\");\n",
       "\t\t\telse if(i==0)\n",
       "\t\t\t\tdocument.write(\"<a href=\\\"t20090818_3955570.html\\\" class=\\\"hui14_30_h\\\">【\"+(i+1)+\"】</a>&nbsp;\");\n",
       "\t\t\telse\n",
       "\t\t\t\tdocument.write(\"<a href=\\\"t20090818_3955570\"+\"_\" + i + \".\"+\"html\\\" class=\\\"hui14_30_h\\\">【\"+(i+1)+\"】</a>&nbsp;\");\n",
       "\t\t}\t\n",
       "\t}\n",
       "\t\n",
       "\tdocument.write(\"<br><br>\");\n",
       "\t//设置上一页代码\n",
       "\tif(countPage>1&&currentPage!=0&&currentPage!=1)\n",
       "\t\tdocument.write(\"<a href=\\\"t20090818_3955570\"+\"_\" + prevPage + \".\"+\"html\\\"><span style=\\\"color:#0033FF;font-weight:bold\\\">上一页</span></a>&nbsp;\");\n",
       "\telse if(countPage>1&&currentPage!=0&&currentPage==1)\n",
       "\t\tdocument.write(\"<a href=\\\"t20090818_3955570.html\\\"><span style=\\\"color:#0033FF;font-weight:bold\\\">上一页</span></a>&nbsp;\");\n",
       "\t//else\n",
       "\t//\tdocument.write(\"上一页 &nbsp;\");\n",
       "\t\n",
       "\t\n",
       "\t//设置下一页代码 \n",
       "\tif(countPage>1&&currentPage!=(countPage-1))\n",
       "\t\tdocument.write(\"<a href=\\\"t20090818_3955570\"+\"_\" + 下一页Page + \".\"+\"html\\\" ><span style=\\\"color:#0033FF;font-weight:bold\\\">下一页</span></a> &nbsp;\");\n",
       "\t//else\n",
       "\t//\tdocument.write(\"下一页 &nbsp;\");\n",
       "\t\t\t\t\t \n",
       "\t</script>"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scripts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:40:18.673197Z",
     "start_time": "2021-11-01T06:40:18.669232Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n\\tvar currentPage = 0;//所在页从0开始\\n\\tvar prevPage = currentPage-1//上一页\\n\\tvar 下一页Page = currentPage+1//下一页\\n\\tvar countPage = 4//共多少页\\n\\t//document.write(\"共\"+countPage+\"页&nbsp;&nbsp;\");\\n\\t\\n\\t//循环\\n\\tvar num = 17;\\n\\tfor(var i=0+(currentPage-1-(currentPage-1)%num) ; i<=(num+(currentPage-1-(currentPage-1)%num))&&(i<countPage) ; i++){\\n\\t\\tif(countPage >1){\\n\\t\\t\\tif(currentPage==i)\\n\\t\\t\\t\\tdocument.write(\"【<span style=\\\\\"color:#FF0000;\\\\\" class=\\\\\"hui14_30_h\\\\\">\"+(i+1)+\"</span>】&nbsp;\");\\n\\t\\t\\telse if(i==0)\\n\\t\\t\\t\\tdocument.write(\"<a href=\\\\\"t20090818_3955570.html\\\\\" class=\\\\\"hui14_30_h\\\\\">【\"+(i+1)+\"】</a>&nbsp;\");\\n\\t\\t\\telse\\n\\t\\t\\t\\tdocument.write(\"<a href=\\\\\"t20090818_3955570\"+\"_\" + i + \".\"+\"html\\\\\" class=\\\\\"hui14_30_h\\\\\">【\"+(i+1)+\"】</a>&nbsp;\");\\n\\t\\t}\\t\\n\\t}\\n\\t\\n\\tdocument.write(\"<br><br>\");\\n\\t//设置上一页代码\\n\\tif(countPage>1&&currentPage!=0&&currentPage!=1)\\n\\t\\tdocument.write(\"<a href=\\\\\"t20090818_3955570\"+\"_\" + prevPage + \".\"+\"html\\\\\"><span style=\\\\\"color:#0033FF;font-weight:bold\\\\\">上一页</span></a>&nbsp;\");\\n\\telse if(countPage>1&&currentPage!=0&&currentPage==1)\\n\\t\\tdocument.write(\"<a href=\\\\\"t20090818_3955570.html\\\\\"><span style=\\\\\"color:#0033FF;font-weight:bold\\\\\">上一页</span></a>&nbsp;\");\\n\\t//else\\n\\t//\\tdocument.write(\"上一页 &nbsp;\");\\n\\t\\n\\t\\n\\t//设置下一页代码 \\n\\tif(countPage>1&&currentPage!=(countPage-1))\\n\\t\\tdocument.write(\"<a href=\\\\\"t20090818_3955570\"+\"_\" + 下一页Page + \".\"+\"html\\\\\" ><span style=\\\\\"color:#0033FF;font-weight:bold\\\\\">下一页</span></a> &nbsp;\");\\n\\t//else\\n\\t//\\tdocument.write(\"下一页 &nbsp;\");\\n\\t\\t\\t\\t\\t \\n\\t'"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scripts.text#.split('countPage = ')[1].split('//')[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:40:37.548831Z",
     "start_time": "2021-11-01T06:40:37.544703Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "4"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# countPage = int(''.join(scripts).split('countPage = ')\\\n",
    "#                 [1].split('//')[0])\n",
    "# countPage\n",
    "\n",
    "countPage = int(scripts.text.split('countPage = ')[1].split('//')[0])\n",
    "countPage"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:48:16.422277Z",
     "start_time": "2021-11-01T06:48:16.413623Z"
    },
    "code_folding": [],
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "    \n",
    "def crawler(url_i):\n",
    "    content = requests.get(url_i)\n",
    "    content.encoding = 'utf8'  \n",
    "    content = content.text\n",
    "    soup = BeautifulSoup(content, 'html.parser') \n",
    "    year = soup.find('span', {'class', 'huang16c'}).text[:4]\n",
    "    year = int(year)\n",
    "    report = ''.join(s.text for s in soup('p'))\n",
    "    # 找到分页信息\n",
    "    scripts = soup.find_all('script')\n",
    "    countPage = int(''.join(scripts[1]).split('countPage = ')[1].split('//')[0])\n",
    "    if countPage == 1:\n",
    "        pass\n",
    "    else:\n",
    "        for i in range(1, countPage):\n",
    "            url_child = url_i.split('.html')[0] +'_'+str(i)+'.html'\n",
    "            content = requests.get(url_child)\n",
    "            content.encoding = 'utf8'\n",
    "            content = content.text\n",
    "            soup = BeautifulSoup(content, 'html.parser') \n",
    "            report_child = ''.join(s.text for s in soup('p'))\n",
    "            report = report + report_child\n",
    "    return year, report\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:46:40.279095Z",
     "start_time": "2021-11-01T06:46:40.275620Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://www.hprc.org.cn/wxzl/wxysl/lczf/dishiyijie_1/200908/t20090818_3955570.html'"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hyperlinks[15] # 2007"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:48:23.480599Z",
     "start_time": "2021-11-01T06:48:23.045514Z"
    }
   },
   "outputs": [],
   "source": [
    "year, report = crawler(hyperlinks[15])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:48:48.913057Z",
     "start_time": "2021-11-01T06:48:48.909842Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2007"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:49:31.088908Z",
     "start_time": "2021-11-01T06:49:31.085349Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'和高素质人才。引导跨国公司把高端制造和研发环节转移到我国，吸引外资加快向中西部、东北地区等老工业基地和符合产业政策的领域扩展。大力承接国际服务外包，提高我国服务业发展水平。加强对外资并购的引导和规范。优化投资环境，规范招商引资行为，纠正一些地方违法违规变相给予优惠政策和层层下达分解指标的做法。\\u3000\\u3000引导和规范企业对外投资合作。完善财税、信贷、外汇、保险等政策措施，支持有实力、有信誉、有竞争力的各种所有制企业走出去。加强引导和协调，避免企业在境外盲目投资和恶性竞争。发展对外承包工程与劳务合作。办好境外经济贸易合作区。推动多哈回合谈判进程，积极参与多边贸易规则制定，稳步推进双边和区域自由贸易区建设。'"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "report[-300:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:52:35.977662Z",
     "start_time": "2021-11-01T06:52:11.802085Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1954\r"
     ]
    }
   ],
   "source": [
    "# 抓取53年政府工作报告内容\n",
    "reports = {}\n",
    "for link in hyperlinks:\n",
    "    year, report = crawler(link)\n",
    "    print(year, end= '\\r') # flush and print\n",
    "    reports[year] = report "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:53:43.577604Z",
     "start_time": "2021-11-01T06:53:43.565060Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "with open('./data/gov_reports1954-2022.txt', 'w', encoding = 'utf8') as f:\n",
    "    for r in reports:\n",
    "        line = str(r)+'\\t'+reports[r].replace('\\n', ' ').replace('\\t', ' ') +'\\n'\n",
    "        f.write(line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:55:13.149030Z",
     "start_time": "2021-11-01T06:55:12.409780Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_table('./data/gov_reports1954-2022.txt', sep = '\\t', names = ['year', 'report'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.read_table?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-11-01T06:55:14.277715Z",
     "start_time": "2021-11-01T06:55:14.265910Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>report</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2022</td>\n",
       "      <td>各位代表：　　现在，我代表国务院，向大会报告政府工作，请予审议，并请全国政协委员提出意见。　...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2021</td>\n",
       "      <td>各位代表：　　现在，我代表国务院，向大会报告政府工作，请予审议，并请全国政协委员提出意见。　...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2020</td>\n",
       "      <td>新华社北京5月29日电　　政府工作报告　　——2020年5月22日在第十三届全国人民代表...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2019</td>\n",
       "      <td>新华社北京3月16日电　　 政府工作报告　　——2019年3月5日在第十三届全国人民代表...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2018</td>\n",
       "      <td>各位代表：　　现在，我代表国务院，向大会报告过去五年政府工作，对今年工作提出建议，请予审...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   year                                             report\n",
       "0  2022  各位代表：　　现在，我代表国务院，向大会报告政府工作，请予审议，并请全国政协委员提出意见。　...\n",
       "1  2021  各位代表：　　现在，我代表国务院，向大会报告政府工作，请予审议，并请全国政协委员提出意见。　...\n",
       "2  2020  　　新华社北京5月29日电　　政府工作报告　　——2020年5月22日在第十三届全国人民代表...\n",
       "3  2019  　　新华社北京3月16日电　　 政府工作报告　　——2019年3月5日在第十三届全国人民代表...\n",
       "4  2018  　　各位代表：　　现在，我代表国务院，向大会报告过去五年政府工作，对今年工作提出建议，请予审..."
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](./images/end.png)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 0,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": false,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "48px",
    "left": "1539px",
    "top": "303px",
    "width": "168px"
   },
   "toc_section_display": false,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}