04-crawler-netease-music.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "\n",
    "\n",
    "# 抓取网易云音乐热门评论\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "https://github.com/RitterHou/music-163\n",
    "\n",
    "爬取网易云音乐的所有的歌曲的评论数。以下为主要思路：\n",
    "\n",
    "- 爬取所有的歌手信息（artists.py）；\n",
    "- 根据上一步爬取到的歌手信息去爬取所有的专辑信息（album_by _artist.py）；\n",
    "- 根据专辑信息爬取所有的歌曲信息（music_by _album.py）；\n",
    "- 根据歌曲信息爬取其评论条数（comments_by _music.py）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 爬取所有的歌手信息（artists.py）"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-10T03:06:06.662665Z",
     "start_time": "2018-05-10T03:06:06.655299Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "观察网易云音乐官网页面HTML结构\n",
    "\n",
    "http://music.163.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-10T05:06:03.732955Z",
     "start_time": "2018-05-10T05:06:03.727816Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "\n",
    "\n",
    "http://music.163.com/#/discover/artist/cat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "http://music.163.com/#/discover/artist/cat?id=4003&initial=0\n",
    "\n",
    "\n",
    "![](./images/netease.png)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:20:06.224337Z",
     "start_time": "2021-05-15T06:20:06.018102Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:20:19.092104Z",
     "start_time": "2021-05-15T06:20:19.088327Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "headers = {\n",
    "    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',\n",
    "    'Accept-Encoding': 'gzip, deflate, sdch',\n",
    "    'Accept-Language': 'zh-CN,zh;q=0.8,en;q=0.6',\n",
    "    'Cache-Control': 'no-cache',\n",
    "    'Connection': 'keep-alive',\n",
    "    'Cookie': '_ntes_nnid=7eced19b27ffae35dad3f8f2bf5885cd,1476521011210; _ntes_nuid=7eced19b27ffae35dad3f8f2bf5885cd; usertrack=c+5+hlgB7TgnsAmACnXtAg==; Province=025; City=025; NTES_PASSPORT=6n9ihXhbWKPi8yAqG.i2kETSCRa.ug06Txh8EMrrRsliVQXFV_orx5HffqhQjuGHkNQrLOIRLLotGohL9s10wcYSPiQfI2wiPacKlJ3nYAXgM; P_INFO=hourui93@163.com|1476523293|1|study|11&12|jis&1476511733&mail163#jis&320100#10#0#0|151889&0|g37_client_check&mailsettings&mail163&study&blog|hourui93@163.com; NTES_SESS=Fa2uk.YZsGoj59AgD6tRjTXGaJ8_1_4YvGfXUkS7C1NwtMe.tG1Vzr255TXM6yj2mKqTZzqFtoEKQrgewi9ZK60ylIqq5puaG6QIaNQ7EK5MTcRgHLOhqttDHfaI_vsBzB4bibfamzx1.fhlpqZh_FcnXUYQFw5F5KIBUmGJg7xdasvGf_EgfICWV; S_INFO=1476597594|1|0&80##|hourui93; NETEASE_AUTH_SOURCE=space; NETEASE_AUTH_USERNAME=hourui93; _ga=GA1.2.1405085820.1476521280; JSESSIONID-WYYY=cbd082d2ce2cffbcd5c085d8bf565a95aee3173ddbbb00bfa270950f93f1d8bb4cb55a56a4049fa8c828373f630c78f4a43d6c3d252c4c44f44b098a9434a7d8fc110670a6e1e9af992c78092936b1e19351435ecff76a181993780035547fa5241a5afb96e8c665182d0d5b911663281967d675ff2658015887a94b3ee1575fa1956a5a%3A1476607977016; _iuqxldmzr_=25; __utma=94650624.1038096298.1476521011.1476595468.1476606177.8; __utmb=94650624.20.10.1476606177; __utmc=94650624; __utmz=94650624.1476521011.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)',\n",
    "    'DNT': '1',\n",
    "    'Host': 'music.163.com',\n",
    "    'Pragma': 'no-cache',\n",
    "    'Referer': 'http://music.163.com/',\n",
    "    'Upgrade-Insecure-Requests': '1',\n",
    "    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36'\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:21:46.111901Z",
     "start_time": "2021-05-15T06:21:45.461602Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "group_id = 1001\n",
    "initial = 67\n",
    "params = {'id': group_id, 'initial': initial}\n",
    "r = requests.get('http://music.163.com/discover/artist/cat', params=params, headers=headers)\n",
    "\n",
    "# 网页解析\n",
    "soup = BeautifulSoup(r.content.decode(), 'html.parser')\n",
    "body = soup.body"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:21:51.306889Z",
     "start_time": "2021-05-15T06:21:51.293214Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "hotartist_dic = {}\n",
    "hot_artists = body.find_all('a', attrs={'class': 'msk'})\n",
    "for artist in hot_artists:\n",
    "    artist_id = artist['href'].replace('/artist?id=', '').strip()\n",
    "    artist_name = artist['title'].replace('的音乐', '')\n",
    "    try:\n",
    "        hotartist_dic[artist_id] = artist_name\n",
    "    except Exception as e:\n",
    "        # 打印错误日志\n",
    "        print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:21:55.473763Z",
     "start_time": "2021-05-15T06:21:55.459681Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "artist_dic = {}\n",
    "artists = body.find_all('a', attrs={'class': 'nm nm-icn f-thide s-fc0'})\n",
    "for artist in artists:\n",
    "        artist_id = artist['href'].replace('/artist?id=', '').strip()\n",
    "        artist_name = artist['title'].replace('的音乐', '')\n",
    "        try:\n",
    "            artist_dic[artist_id] = artist_name\n",
    "        except Exception as e:\n",
    "            # 打印错误日志\n",
    "            print(e)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:21:56.902254Z",
     "start_time": "2021-05-15T06:21:56.890811Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'2116': '陈奕迅',\n",
       " '32897995': 'Copy',\n",
       " '28083351': 'CORSAK胡梦周',\n",
       " '34172520': 'C.HØPE',\n",
       " '12157336': 'CMJ',\n",
       " '28602303': '陈阳',\n",
       " '1058228': '陈鸿宇',\n",
       " '2112': '陈小春',\n",
       " '2127': '陈柏宇',\n",
       " '12932368': '蔡徐坤',\n",
       " '963431': '陈致逸',\n",
       " '12147170': '澄海伯伯',\n",
       " '29234568': '陈亦洺',\n",
       " '32720055': 'Crazy Donkey',\n",
       " '30377220': '陈麒名',\n",
       " '12519065': '程安',\n",
       " '32973587': '陈大本事',\n",
       " '34097618': 'ChenXi',\n",
       " '1140028': '承利',\n",
       " '2115': '陈百强',\n",
       " '12193174': 'Capper',\n",
       " '189955': '蔡翊昇',\n",
       " '14420803': '陈雪燃',\n",
       " '2110': '曹格',\n",
       " '34918869': 'Cheng橙zzzz',\n",
       " '12174110': '崔开潮',\n",
       " '12276430': 'Cubi',\n",
       " '12131566': 'CLOUDWANG 王云',\n",
       " '2118': '成龙',\n",
       " '2117': '侧田',\n",
       " '14608268': 'Corki宗鑫',\n",
       " '12095299': '陈柯宇',\n",
       " '1084178': 'Candy_Wind',\n",
       " '13447762': '陈亦云',\n",
       " '13056440': '陈立农',\n",
       " '13681463': '崔铭嘉',\n",
       " '31005762': '冲矢昴',\n",
       " '2331': '陈翔',\n",
       " '31252052': '春野杉卉',\n",
       " '33238413': '陈智凌',\n",
       " '33320567': '苍弟（半吨兄弟）',\n",
       " '34318343': 'Cc椿',\n",
       " '2124': '陈楚生',\n",
       " '12434106': '陈尤利',\n",
       " '47136983': '陈恻',\n",
       " '2201': '陈势安',\n",
       " '34564181': '菜篮中的菜',\n",
       " '1038074': '茶茶',\n",
       " '33950188': 'CHENYI',\n",
       " '31052925': '川屿',\n",
       " '32440237': '蔡少铭',\n",
       " '34318335': 'ChenYueLong.',\n",
       " '46632613': '陈小潇',\n",
       " '963409': '陈学冬',\n",
       " '12536599': '迟柏林',\n",
       " '12079044': '陈硕子',\n",
       " '2153': '常石磊',\n",
       " '2150': '陈冠蒲',\n",
       " '32264375': '从和文',\n",
       " '47060485': '晨晨没错了',\n",
       " '30000267': '蔡耀轩',\n",
       " '31087528': '陈奕辰Kane',\n",
       " '4092': '春风哥',\n",
       " '2130': '蔡国权',\n",
       " '47383136': '陈宇鸣',\n",
       " '12177051': 'C.A.R.L',\n",
       " '1047298': 'CADY',\n",
       " '2121': '蔡旻佑',\n",
       " '2111': '崔健',\n",
       " '2113': '车继铃',\n",
       " '31665914': 'Cackle Joyful',\n",
       " '29414807': '陈冬霖',\n",
       " '12185665': 'C.vo',\n",
       " '12291202': '陈曦',\n",
       " '33394182': '陈志林Mega',\n",
       " '33123519': '潮音锦式',\n",
       " '34038223': '初梦',\n",
       " '30437389': 'Chanlex',\n",
       " '2294': '陈光荣',\n",
       " '12490206': 'Credit',\n",
       " '30117214': 'CrapBoy',\n",
       " '12193309': 'CREAM D',\n",
       " '33390699': '潮音铭帝',\n",
       " '2338': '船长',\n",
       " '33711050': 'CFrie木焱',\n",
       " '34915098': '常泽浩',\n",
       " '31093206': 'Cas14',\n",
       " '27728315': '唇釉',\n",
       " '1043290': '传琦SAMA',\n",
       " '12459163': '陈嘉俊是这个嘉',\n",
       " '2159': '曹轩宾',\n",
       " '31055': 'Cee',\n",
       " '36621645': 'CHANGAN',\n",
       " '12700520': 'CJ 周密',\n",
       " '2135': '陈勋奇',\n",
       " '46507954': '川上島',\n",
       " '31050529': '陈茗',\n",
       " '2122': '陈冠希',\n",
       " '2125': '陈晓东',\n",
       " '12226597': '痴笑'}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "artist_dic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:24:25.630337Z",
     "start_time": "2021-05-15T06:24:25.623497Z"
    },
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def save_artist(group_id, initial, hot_artist_dic, artisti_dic):\n",
    "    params = {'id': group_id, 'initial': initial}\n",
    "    r = requests.get('http://music.163.com/discover/artist/cat', params=params)\n",
    "\n",
    "    # 网页解析\n",
    "    soup = BeautifulSoup(r.content.decode(), 'html.parser')\n",
    "    body = soup.body\n",
    "\n",
    "    hot_artists = body.find_all('a', attrs={'class': 'msk'})\n",
    "    artists = body.find_all('a', attrs={'class': 'nm nm-icn f-thide s-fc0'})\n",
    "    for artist in hot_artists:\n",
    "        artist_id = artist['href'].replace('/artist?id=', '').strip()\n",
    "        artist_name = artist['title'].replace('的音乐', '')\n",
    "        try:\n",
    "            hot_artist_dic[artist_id] = artist_name\n",
    "        except Exception as e:\n",
    "            # 打印错误日志\n",
    "            print(e)\n",
    "\n",
    "    for artist in artists:\n",
    "        artist_id = artist['href'].replace('/artist?id=', '').strip()\n",
    "        artist_name = artist['title'].replace('的音乐', '')\n",
    "        try:\n",
    "            artist_dic[artist_id] = artist_name\n",
    "        except Exception as e:\n",
    "            # 打印错误日志\n",
    "            print(e)\n",
    "    #return artist_dic, hot_artist_dic\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:24:39.517392Z",
     "start_time": "2021-05-15T06:24:39.209485Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "gg = 1001\n",
    "initial = 67\n",
    "artist_dic = {}\n",
    "hot_artist_dic = {} \n",
    "save_artist(gg, initial, hot_artist_dic, artist_dic  )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:24:42.525464Z",
     "start_time": "2021-05-15T06:24:42.518558Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'2116': '陈奕迅',\n",
       " '32897995': 'Copy',\n",
       " '28083351': 'CORSAK胡梦周',\n",
       " '34172520': 'C.HØPE',\n",
       " '12157336': 'CMJ',\n",
       " '28602303': '陈阳',\n",
       " '1058228': '陈鸿宇',\n",
       " '2112': '陈小春',\n",
       " '2127': '陈柏宇',\n",
       " '12932368': '蔡徐坤',\n",
       " '963431': '陈致逸',\n",
       " '12147170': '澄海伯伯',\n",
       " '29234568': '陈亦洺',\n",
       " '32720055': 'Crazy Donkey',\n",
       " '30377220': '陈麒名',\n",
       " '12519065': '程安',\n",
       " '32973587': '陈大本事',\n",
       " '34097618': 'ChenXi',\n",
       " '1140028': '承利',\n",
       " '2115': '陈百强',\n",
       " '12193174': 'Capper',\n",
       " '189955': '蔡翊昇',\n",
       " '14420803': '陈雪燃',\n",
       " '2110': '曹格',\n",
       " '34918869': 'Cheng橙zzzz',\n",
       " '12174110': '崔开潮',\n",
       " '12276430': 'Cubi',\n",
       " '12131566': 'CLOUDWANG 王云',\n",
       " '2118': '成龙',\n",
       " '2117': '侧田',\n",
       " '14608268': 'Corki宗鑫',\n",
       " '12095299': '陈柯宇',\n",
       " '1084178': 'Candy_Wind',\n",
       " '13447762': '陈亦云',\n",
       " '13056440': '陈立农',\n",
       " '13681463': '崔铭嘉',\n",
       " '31005762': '冲矢昴',\n",
       " '2331': '陈翔',\n",
       " '31252052': '春野杉卉',\n",
       " '33238413': '陈智凌',\n",
       " '33320567': '苍弟（半吨兄弟）',\n",
       " '34318343': 'Cc椿',\n",
       " '2124': '陈楚生',\n",
       " '12434106': '陈尤利',\n",
       " '47136983': '陈恻',\n",
       " '2201': '陈势安',\n",
       " '34564181': '菜篮中的菜',\n",
       " '1038074': '茶茶',\n",
       " '33950188': 'CHENYI',\n",
       " '31052925': '川屿',\n",
       " '32440237': '蔡少铭',\n",
       " '34318335': 'ChenYueLong.',\n",
       " '46632613': '陈小潇',\n",
       " '963409': '陈学冬',\n",
       " '12536599': '迟柏林',\n",
       " '12079044': '陈硕子',\n",
       " '2153': '常石磊',\n",
       " '2150': '陈冠蒲',\n",
       " '32264375': '从和文',\n",
       " '47060485': '晨晨没错了',\n",
       " '30000267': '蔡耀轩',\n",
       " '31087528': '陈奕辰Kane',\n",
       " '4092': '春风哥',\n",
       " '2130': '蔡国权',\n",
       " '47383136': '陈宇鸣',\n",
       " '12177051': 'C.A.R.L',\n",
       " '1047298': 'CADY',\n",
       " '2121': '蔡旻佑',\n",
       " '2111': '崔健',\n",
       " '2113': '车继铃',\n",
       " '31665914': 'Cackle Joyful',\n",
       " '29414807': '陈冬霖',\n",
       " '12185665': 'C.vo',\n",
       " '12291202': '陈曦',\n",
       " '33394182': '陈志林Mega',\n",
       " '33123519': '潮音锦式',\n",
       " '34038223': '初梦',\n",
       " '30437389': 'Chanlex',\n",
       " '2294': '陈光荣',\n",
       " '12490206': 'Credit',\n",
       " '30117214': 'CrapBoy',\n",
       " '12193309': 'CREAM D',\n",
       " '33390699': '潮音铭帝',\n",
       " '2338': '船长',\n",
       " '33711050': 'CFrie木焱',\n",
       " '34915098': '常泽浩',\n",
       " '31093206': 'Cas14',\n",
       " '27728315': '唇釉',\n",
       " '1043290': '传琦SAMA',\n",
       " '12459163': '陈嘉俊是这个嘉',\n",
       " '2159': '曹轩宾',\n",
       " '31055': 'Cee',\n",
       " '36621645': 'CHANGAN',\n",
       " '12700520': 'CJ 周密',\n",
       " '2135': '陈勋奇',\n",
       " '46507954': '川上島',\n",
       " '31050529': '陈茗',\n",
       " '2122': '陈冠希',\n",
       " '2125': '陈晓东',\n",
       " '12226597': '痴笑'}"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "artist_dic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:24:56.706234Z",
     "start_time": "2021-05-15T06:24:48.120196Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "65\n",
      "66\n",
      "67\n",
      "68\n",
      "69\n",
      "70\n",
      "71\n",
      "72\n",
      "73\n",
      "74\n",
      "75\n",
      "76\n",
      "77\n",
      "78\n",
      "79\n",
      "80\n",
      "81\n",
      "82\n",
      "83\n",
      "84\n",
      "85\n",
      "86\n",
      "87\n",
      "88\n",
      "89\n",
      "90\n"
     ]
    }
   ],
   "source": [
    "artist_dic = {}\n",
    "hot_artist_dic = {} \n",
    "for i in range(65, 91):\n",
    "    print(i)\n",
    "    save_artist(gg, i, hot_artist_dic, artist_dic  )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:25:07.338660Z",
     "start_time": "2021-05-15T06:25:07.334924Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "260"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(hot_artist_dic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:25:17.988195Z",
     "start_time": "2021-05-15T06:25:17.984224Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "2600"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(artist_dic)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 爬取所有的专辑信息（album_by _artist.py）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:25:36.915447Z",
     "start_time": "2021-05-15T06:25:36.911493Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'12174057'"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(hot_artist_dic.keys())[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-09T10:11:23.224190Z",
     "start_time": "2018-05-09T10:11:23.220284Z"
    },
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "source": [
    "http://music.163.com/#/artist/album?id=89659&limit=400"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "headers = {\n",
    "    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',\n",
    "    'Accept-Encoding': 'gzip, deflate, sdch',\n",
    "    'Accept-Language': 'zh-CN,zh;q=0.8,en;q=0.6',\n",
    "    'Cache-Control': 'no-cache',\n",
    "    'Connection': 'keep-alive',\n",
    "    'Cookie': '_ntes_nnid=7eced19b27ffae35dad3f8f2bf5885cd,1476521011210; _ntes_nuid=7eced19b27ffae35dad3f8f2bf5885cd; usertrack=c+5+hlgB7TgnsAmACnXtAg==; Province=025; City=025; _ga=GA1.2.1405085820.1476521280; NTES_PASSPORT=6n9ihXhbWKPi8yAqG.i2kETSCRa.ug06Txh8EMrrRsliVQXFV_orx5HffqhQjuGHkNQrLOIRLLotGohL9s10wcYSPiQfI2wiPacKlJ3nYAXgM; P_INFO=hourui93@163.com|1476523293|1|study|11&12|jis&1476511733&mail163#jis&320100#10#0#0|151889&0|g37_client_check&mailsettings&mail163&study&blog|hourui93@163.com; JSESSIONID-WYYY=189f31767098c3bd9d03d9b968c065daf43cbd4c1596732e4dcb471beafe2bf0605b85e969f92600064a977e0b64a24f0af7894ca898b696bd58ad5f39c8fce821ec2f81f826ea967215de4d10469e9bd672e75d25f116a9d309d360582a79620b250625859bc039161c78ab125a1e9bf5d291f6d4e4da30574ccd6bbab70b710e3f358f%3A1476594130342; _iuqxldmzr_=25; __utma=94650624.1038096298.1476521011.1476588849.1476592408.6; __utmb=94650624.11.10.1476592408; __utmc=94650624; __utmz=94650624.1476521011.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)',\n",
    "    'DNT': '1',\n",
    "    'Host': 'music.163.com',\n",
    "    'Pragma': 'no-cache',\n",
    "    'Referer': 'http://music.163.com/',\n",
    "    'Upgrade-Insecure-Requests': '1',\n",
    "    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36'\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:27:28.447720Z",
     "start_time": "2021-05-15T06:27:28.443068Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def save_albums(artist_id, albume_dic):\n",
    "    params = {'id': artist_id, 'limit': '200'}\n",
    "    # 获取歌手个人主页\n",
    "    r = requests.get('http://music.163.com/artist/album', headers=headers, params=params)\n",
    "\n",
    "    # 网页解析\n",
    "    soup = BeautifulSoup(r.content.decode(), 'html.parser')\n",
    "    body = soup.body\n",
    "\n",
    "    albums = body.find_all('a', attrs={'class': 'tit s-fc0'})  # 获取所有专辑\n",
    "\n",
    "    for album in albums:\n",
    "        albume_id = album['href'].replace('/album?id=', '')\n",
    "        albume_dic[albume_id] = artist_id"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:27:39.333362Z",
     "start_time": "2021-05-15T06:27:38.708490Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "albume_dic = {}\n",
    "save_albums('2116', albume_dic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:27:42.651661Z",
     "start_time": "2021-05-15T06:27:42.645362Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'98517711': '2116',\n",
       " '97079895': '2116',\n",
       " '95509227': '2116',\n",
       " '92392526': '2116',\n",
       " '87540898': '2116',\n",
       " '80538760': '2116',\n",
       " '80235030': '2116',\n",
       " '74268947': '2116',\n",
       " '38296010': '2116',\n",
       " '36304576': '2116',\n",
       " '35835294': '2116',\n",
       " '35643233': '2116',\n",
       " '35520072': '2116',\n",
       " '34961173': '2116',\n",
       " '34923261': '2116',\n",
       " '34881554': '2116',\n",
       " '34735139': '2116',\n",
       " '3404003': '2116',\n",
       " '3319407': '2116',\n",
       " '3279818': '2116',\n",
       " '3184340': '2116',\n",
       " '3279543': '2116',\n",
       " '35406784': '2116',\n",
       " '3109376': '2116',\n",
       " '2801259': '2116',\n",
       " '2793003': '2116',\n",
       " '2786670': '2116',\n",
       " '2767144': '2116',\n",
       " '2732645': '2116',\n",
       " '2518003': '2116',\n",
       " '2532179': '2116',\n",
       " '2499005': '2116',\n",
       " '2465020': '2116',\n",
       " '2302128': '2116',\n",
       " '2261058': '2116',\n",
       " '2261147': '2116',\n",
       " '2261091': '2116',\n",
       " '6335': '2116',\n",
       " '6338': '2116',\n",
       " '6339': '2116',\n",
       " '6341': '2116',\n",
       " '6343': '2116',\n",
       " '2692242': '2116',\n",
       " '2692238': '2116',\n",
       " '6355': '2116',\n",
       " '35663692': '2116',\n",
       " '6360': '2116',\n",
       " '6362': '2116',\n",
       " '6365': '2116',\n",
       " '6375': '2116',\n",
       " '6378': '2116',\n",
       " '35398900': '2116',\n",
       " '6388': '2116',\n",
       " '3211014': '2116',\n",
       " '3102567': '2116',\n",
       " '34611604': '2116',\n",
       " '6394': '2116',\n",
       " '3170625': '2116',\n",
       " '2339617': '2116',\n",
       " '6410': '2116',\n",
       " '2692239': '2116',\n",
       " '6404': '2116',\n",
       " '3391071': '2116',\n",
       " '6423': '2116',\n",
       " '6429': '2116',\n",
       " '6434': '2116',\n",
       " '6437': '2116',\n",
       " '6451': '2116',\n",
       " '6452': '2116',\n",
       " '6454': '2116',\n",
       " '6462': '2116',\n",
       " '6475': '2116',\n",
       " '6479': '2116',\n",
       " '6491': '2116',\n",
       " '6483': '2116',\n",
       " '6498': '2116',\n",
       " '35411774': '2116',\n",
       " '6510': '2116',\n",
       " '6522': '2116',\n",
       " '6530': '2116',\n",
       " '3070638': '2116',\n",
       " '6543': '2116',\n",
       " '6548': '2116',\n",
       " '6551': '2116',\n",
       " '6555': '2116',\n",
       " '6559': '2116',\n",
       " '6562': '2116',\n",
       " '6565': '2116',\n",
       " '6567': '2116',\n",
       " '6572': '2116',\n",
       " '6581': '2116',\n",
       " '2374009': '2116',\n",
       " '6584': '2116',\n",
       " '2374010': '2116',\n",
       " '6590': '2116',\n",
       " '2336647': '2116',\n",
       " '6595': '2116',\n",
       " '6599': '2116',\n",
       " '2621232': '2116',\n",
       " '2374011': '2116',\n",
       " '6604': '2116',\n",
       " '2374012': '2116',\n",
       " '6607': '2116',\n",
       " '6612': '2116',\n",
       " '2374013': '2116',\n",
       " '2374014': '2116',\n",
       " '6620': '2116',\n",
       " '6624': '2116',\n",
       " '2332713': '2116'}"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "albume_dic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 根据专辑信息爬取所有的歌曲信息（music_by _album.py）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:29:22.259438Z",
     "start_time": "2021-05-15T06:29:22.254686Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "def save_music(album_id, music_dic):\n",
    "    params = {'id': album_id}\n",
    "    # 获取专辑对应的页面\n",
    "    r = requests.get('http://music.163.com/album', headers=headers, params=params)\n",
    "    # 网页解析\n",
    "    soup = BeautifulSoup(r.content.decode(), 'html.parser')\n",
    "    body = soup.body\n",
    "    musics = body.find('ul', attrs={'class': 'f-hide'}).find_all('li')  # 获取专辑的所有音乐\n",
    "    for music in musics:\n",
    "        music = music.find('a')\n",
    "        music_id = music['href'].replace('/song?id=', '')\n",
    "        music_name = music.getText()\n",
    "        music_dic[music_id] = [music_name, album_id]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:29:25.389423Z",
     "start_time": "2021-05-15T06:29:25.385376Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'98517711'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(albume_dic.keys())[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:29:34.171287Z",
     "start_time": "2021-05-15T06:29:33.534275Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "music_dic = {}\n",
    "save_music('6423', music_dic)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:29:34.595565Z",
     "start_time": "2021-05-15T06:29:34.590339Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'65321': ['兄妹', '6423'],\n",
       " '65326': ['十年', '6423'],\n",
       " '65334': ['你的背包', '6423'],\n",
       " '65337': ['K歌之王', '6423'],\n",
       " '65342': ['Shall We Talk', '6423'],\n",
       " '65347': ['低等动物', '6423'],\n",
       " '65350': ['寂寞让你更快乐', '6423'],\n",
       " '65355': ['圣诞结', '6423'],\n",
       " '65360': ['想哭', '6423'],\n",
       " '65365': ['不如这样', '6423'],\n",
       " '65369': ['你会不会', '6423'],\n",
       " '65373': ['Last Order', '6423'],\n",
       " '65377': ['冤家', '6423'],\n",
       " '65381': ['全世界失眠', '6423'],\n",
       " '65385': ['我们都寂寞', '6423'],\n",
       " '65389': ['阿怪', '6423'],\n",
       " '65393': ['谢谢侬', '6423'],\n",
       " '65397': ['爱是怀疑', '6423'],\n",
       " '65400': [\"Because You're Good To Me\", '6423'],\n",
       " '65403': ['Good Times', '6423'],\n",
       " '65406': ['要你的', '6423'],\n",
       " '65410': ['像一句广告', '6423'],\n",
       " '65414': ['我也不会那样做', '6423'],\n",
       " '65418': ['人造卫星', '6423'],\n",
       " '65421': ['狂人日记', '6423'],\n",
       " '65425': ['没有手机的日子', '6423'],\n",
       " '65429': ['跳蚤市场', '6423'],\n",
       " '65433': ['故事', '6423'],\n",
       " '65437': ['男人的错', '6423'],\n",
       " '65441': ['没有你', '6423']}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "music_dic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 根据歌曲信息爬取其评论条数（comments_by _music.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "http://music.163.com/#/song?id=516997458\n",
    "\n",
    "\n",
    "很遗憾的是评论数虽然也在详情页内，但是网易云音乐做了防爬处理，\n",
    "- 采用AJAX调用评论数API的方式填充评论相关数据，\n",
    "- 异步的特性导致我们爬到的页面中评论数是空，\n",
    "\n",
    "我们就找一找这个API吧，通关观察XHR请求发现是下面这个家伙..\n",
    "\n",
    "响应结果很丰富呢，所有评论相关的数据都有，不过经过观察发现这个API是经过加密处理的，不过没关系...\n",
    "\n",
    "https://blog.csdn.net/python233/article/details/72825003\n",
    "\n",
    "https://www.zhihu.com/question/36081767\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:38:55.004179Z",
     "start_time": "2021-05-15T06:38:55.001241Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [],
   "source": [
    "params = {\n",
    "    'csrf_token': ''\n",
    "}\n",
    "\n",
    "data = {\n",
    "    'params': '5L+s/X1qDy33tb2sjT6to2T4oxv89Fjg1aYRkjgzpNPR6hgCpp0YVjNoTLQAwWu9VYvKROPZQj6qTpBK+sUeJovyNHsnU9/StEfZwCOcKfECFFtAvoNIpulj1TDOtBir',\n",
    "    'encSecKey': '59079f3e07d6e240410018dc871bf9364f122b720c0735837d7916ac78d48a79ec06c6307e6a0e576605d6228bd0b377a96e1a7fc7c7ddc8f6a3dc6cc50746933352d4ec5cbe7bddd6dcb94de085a3b408d895ebfdf2f43a7c72fc783512b3c9efb860679a88ef21ccec5ff13592be450a1edebf981c0bf779b122ddbd825492'\n",
    "    \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:39:37.078787Z",
     "start_time": "2021-05-15T06:39:36.888408Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['isMusician', 'userId', 'topComments', 'moreHot', 'hotComments', 'commentBanner', 'code', 'comments', 'total', 'more'])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "offset = 0\n",
    "music_id = '65337'\n",
    "url = 'http://music.163.com/api/v1/resource/comments/R_SO_4_'+ music_id + '?limit=20&offset=' + str(offset)\n",
    "response = requests.post(url, headers=headers, data=data)\n",
    "cj = response.json()\n",
    "cj.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:39:41.538163Z",
     "start_time": "2021-05-15T06:39:41.534087Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(18942, 20, 15, 0)"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cj['total'],len(cj['comments']), len(cj['hotComments']), len(cj['topComments'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2021-05-15T06:44:16.866621Z",
     "start_time": "2021-05-15T06:44:16.861645Z"
    },
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'user': {'locationInfo': None,\n",
       "  'liveInfo': None,\n",
       "  'anonym': 0,\n",
       "  'userId': 130378562,\n",
       "  'avatarDetail': None,\n",
       "  'userType': 0,\n",
       "  'followed': False,\n",
       "  'mutual': False,\n",
       "  'remarkName': None,\n",
       "  'vipRights': None,\n",
       "  'nickname': '富士山下钟无艳-',\n",
       "  'avatarUrl': 'http://p2.music.126.net/AlUvuBnyf32Qs1bhtWQpvw==/109951163540816255.jpg',\n",
       "  'authStatus': 0,\n",
       "  'expertTags': None,\n",
       "  'experts': None,\n",
       "  'vipType': 0,\n",
       "  'commonIdentity': None},\n",
       " 'beReplied': [],\n",
       " 'pendantData': None,\n",
       " 'showFloorComment': None,\n",
       " 'status': 0,\n",
       " 'commentId': 310484149,\n",
       " 'content': '这是那天她哭着唱的歌  因此这是我一辈子记得的歌@豆芽小张 ',\n",
       " 'time': 1487199594599,\n",
       " 'likedCount': 1811,\n",
       " 'expressionUrl': None,\n",
       " 'commentLocationType': 0,\n",
       " 'parentCommentId': 0,\n",
       " 'decoration': None,\n",
       " 'repliedMark': None,\n",
       " 'liked': False}"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cj['hotComments'][11]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "subslide"
    }
   },
   "source": [
    "## 翻页的实现\n",
    "\n",
    "limit是一页的数量，offset往后的偏移。\n",
    "- 比如limit是20，offset是40，就展示第三页的\n",
    "\n",
    "http://music.163.com/api/v1/resource/comments/R_SO_4_516997458?limit=20&offset=0\n",
    "\n",
    "http://music.163.com/api/v1/resource/comments/R_SO_4_516997458?limit=20&offset=20\n",
    "\n",
    "http://music.163.com/api/v1/resource/comments/R_SO_4_516997458?limit=20&offset=40"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## 另外一种方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-10T04:38:38.262593Z",
     "start_time": "2018-05-10T04:38:38.133110Z"
    }
   },
   "outputs": [],
   "source": [
    "from Crypto.Cipher import AES\n",
    "import base64\n",
    "import requests\n",
    "import json\n",
    "import time\n",
    "\n",
    "# headers\n",
    "headers = {\n",
    "    'Host': 'music.163.com',\n",
    "    'Connection': 'keep-alive',\n",
    "    'Content-Length': '484',\n",
    "    'Cache-Control': 'max-age=0',\n",
    "    'Origin': 'http://music.163.com',\n",
    "    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.84 Safari/537.36',\n",
    "    'Content-Type': 'application/x-www-form-urlencoded',\n",
    "    'Accept': '*/*',\n",
    "    'DNT': '1',\n",
    "    'Accept-Encoding': 'gzip, deflate',\n",
    "    'Accept-Language': 'zh-CN,zh;q=0.8,en;q=0.6,zh-TW;q=0.4',\n",
    "    'Cookie': 'JSESSIONID-WYYY=b66d89ed74ae9e94ead89b16e475556e763dd34f95e6ca357d06830a210abc7b685e82318b9d1d5b52ac4f4b9a55024c7a34024fddaee852404ed410933db994dcc0e398f61e670bfeea81105cbe098294e39ac566e1d5aa7232df741870ba1fe96e5cede8372ca587275d35c1a5d1b23a11e274a4c249afba03e20fa2dafb7a16eebdf6%3A1476373826753; _iuqxldmzr_=25; _ntes_nnid=7fa73e96706f26f3ada99abba6c4a6b2,1476372027128; _ntes_nuid=7fa73e96706f26f3ada99abba6c4a6b2; __utma=94650624.748605760.1476372027.1476372027.1476372027.1; __utmb=94650624.4.10.1476372027; __utmc=94650624; __utmz=94650624.1476372027.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)',\n",
    "}\n",
    "\n",
    "\n",
    "\n",
    "#获取params\n",
    "def get_params(first_param, forth_param):\n",
    "    iv = \"0102030405060708\"\n",
    "    first_key = forth_param\n",
    "    second_key = 16 * 'F'\n",
    "    h_encText = AES_encrypt(first_param, first_key.encode(), iv.encode())\n",
    "    h_encText = AES_encrypt(h_encText.decode(), second_key.encode(), iv.encode())\n",
    "    return h_encText.decode()\n",
    "\n",
    "\n",
    "# 获取encSecKey\n",
    "def get_encSecKey():\n",
    "    encSecKey = \"257348aecb5e556c066de214e531faadd1c55d814f9be95fd06d6bff9f4c7a41f831f6394d5a3fd2e3881736d94a02ca919d952872e7d0a50ebfa1769a7a62d512f5f1ca21aec60bc3819a9c3ffca5eca9a0dba6d6f7249b06f5965ecfff3695b54e1c28f3f624750ed39e7de08fc8493242e26dbc4484a01c76f739e135637c\"\n",
    "    return encSecKey\n",
    "\n",
    "\n",
    "# 解AES秘\n",
    "def AES_encrypt(text, key, iv):\n",
    "    pad = 16 - len(text) % 16\n",
    "    text = text + pad * chr(pad)\n",
    "    encryptor = AES.new(key, AES.MODE_CBC, iv)\n",
    "    encrypt_text = encryptor.encrypt(text.encode())\n",
    "    encrypt_text = base64.b64encode(encrypt_text)\n",
    "    return encrypt_text\n",
    "\n",
    "\n",
    "# 获取json数据\n",
    "def get_json(url, data):\n",
    "    response = requests.post(url, headers=headers, data=data)\n",
    "    return response.content\n",
    "\n",
    "\n",
    "# 传入post数据\n",
    "def crypt_api(id, offset):\n",
    "    url = \"http://music.163.com/weapi/v1/resource/comments/R_SO_4_%s/?csrf_token=\" % id\n",
    "    first_param = \"{rid:\\\"\\\", offset:\\\"%s\\\", total:\\\"true\\\", limit:\\\"20\\\", csrf_token:\\\"\\\"}\" % offset\n",
    "    forth_param = \"0CoJUm6Qyw8W8jud\"\n",
    "    params = get_params(first_param, forth_param)\n",
    "    encSecKey = get_encSecKey()\n",
    "    data = {\n",
    "        \"params\": params,\n",
    "        \"encSecKey\": encSecKey\n",
    "    }\n",
    "    return url, data\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-10T04:41:55.356484Z",
     "start_time": "2018-05-10T04:41:55.251451Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8054"
      ]
     },
     "execution_count": 138,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "offset = 0\n",
    "id = '516997458'\n",
    "url, data = crypt_api(id, offset)\n",
    "json_text = get_json(url, data)\n",
    "json_dict = json.loads(json_text.decode(\"utf-8\"))\n",
    "comments_sum = json_dict['total']\n",
    "comments_sum"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-10T04:41:56.302874Z",
     "start_time": "2018-05-10T04:41:56.298243Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 139,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(json_dict['comments'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-10T04:41:57.399920Z",
     "start_time": "2018-05-10T04:41:57.394647Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'beReplied': [],\n",
       " 'commentId': 1112523641,\n",
       " 'content': '喜欢双笙，喜欢这首歌',\n",
       " 'isRemoveHotComment': False,\n",
       " 'liked': False,\n",
       " 'likedCount': 1,\n",
       " 'pendantData': None,\n",
       " 'time': 1525904882188,\n",
       " 'user': {'authStatus': 0,\n",
       "  'avatarUrl': 'http://p1.music.126.net/Eklu6D8QoR1Hb5UhLhCzPw==/109951163288324813.jpg',\n",
       "  'expertTags': None,\n",
       "  'experts': None,\n",
       "  'locationInfo': None,\n",
       "  'nickname': '狂妄嘻嘻',\n",
       "  'remarkName': None,\n",
       "  'userId': 1451756393,\n",
       "  'userType': 0,\n",
       "  'vipType': 0}}"
      ]
     },
     "execution_count": 140,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json_dict['comments'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-10T04:42:02.080722Z",
     "start_time": "2018-05-10T04:42:02.074947Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'beReplied': [{'content': '我们历史老师是一个年轻的小伙子。那是个阳光明媚的中午，他拖堂拖了很久，喇叭里响起了学校广播“校园之声”的开场白，接着就是这首歌。老师听到这首歌前奏后，自以为是地说一定是播音员自己唱的。我们都在下面反驳他，说人家歌就是这样的。。\\n而现在，距中考只有58天了，毕业后，就回不去了。',\n",
       "   'status': 0,\n",
       "   'user': {'authStatus': 0,\n",
       "    'avatarUrl': 'http://p1.music.126.net/gm976KYbWTvYvExzjBNeaw==/109951163217371336.jpg',\n",
       "    'expertTags': None,\n",
       "    'experts': None,\n",
       "    'locationInfo': None,\n",
       "    'nickname': '惴洛',\n",
       "    'remarkName': None,\n",
       "    'userId': 1325932231,\n",
       "    'userType': 0,\n",
       "    'vipType': 0}}],\n",
       " 'commentId': 1112261542,\n",
       " 'content': '还有不到一个月了高一学姐祝你考试加油哦',\n",
       " 'isRemoveHotComment': False,\n",
       " 'liked': False,\n",
       " 'likedCount': 0,\n",
       " 'pendantData': None,\n",
       " 'time': 1525876023865,\n",
       " 'user': {'authStatus': 0,\n",
       "  'avatarUrl': 'http://p1.music.126.net/kAuCCkW-fcC7yu4wix9z5Q==/109951163144186242.jpg',\n",
       "  'expertTags': None,\n",
       "  'experts': None,\n",
       "  'locationInfo': None,\n",
       "  'nickname': '土园yy',\n",
       "  'remarkName': None,\n",
       "  'userId': 275653796,\n",
       "  'userType': 0,\n",
       "  'vipType': 0}}"
      ]
     },
     "execution_count": 141,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json_dict['comments'][4]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-10T04:40:42.433766Z",
     "start_time": "2018-05-10T04:40:42.284057Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'beReplied': [],\n",
       " 'commentId': 1107837178,\n",
       " 'content': '冥月声音好听好温柔[爱心]表白',\n",
       " 'isRemoveHotComment': False,\n",
       " 'liked': False,\n",
       " 'likedCount': 3,\n",
       " 'pendantData': None,\n",
       " 'time': 1525515089450,\n",
       " 'user': {'authStatus': 0,\n",
       "  'avatarUrl': 'http://p1.music.126.net/suhvzXk2pEUOaeHUPU0aQQ==/109951163173870029.jpg',\n",
       "  'expertTags': None,\n",
       "  'experts': None,\n",
       "  'locationInfo': None,\n",
       "  'nickname': '黴祇',\n",
       "  'remarkName': None,\n",
       "  'userId': 619018018,\n",
       "  'userType': 0,\n",
       "  'vipType': 0}}"
      ]
     },
     "execution_count": 135,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "offset = 20\n",
    "id = '516997458'\n",
    "url, data = crypt_api(id, offset)\n",
    "json_text = get_json(url, data)\n",
    "json_dict = json.loads(json_text.decode(\"utf-8\"))\n",
    "comments_sum = json_dict['total']\n",
    "json_dict['comments'][0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-05-10T04:41:02.854104Z",
     "start_time": "2018-05-10T04:41:02.771941Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'beReplied': [],\n",
       " 'commentId': 1102303635,\n",
       " 'content': '找这首歌找了好久了！！无厘头的找，今天无意居然听到了(*^▽^)/★*☆',\n",
       " 'isRemoveHotComment': False,\n",
       " 'liked': False,\n",
       " 'likedCount': 1,\n",
       " 'pendantData': None,\n",
       " 'time': 1525072647936,\n",
       " 'user': {'authStatus': 0,\n",
       "  'avatarUrl': 'http://p1.music.126.net/fU8tvMVN2f5WkSUZehQ21Q==/3274345636764863.jpg',\n",
       "  'expertTags': None,\n",
       "  'experts': None,\n",
       "  'locationInfo': None,\n",
       "  'nickname': '黎诺0',\n",
       "  'remarkName': None,\n",
       "  'userId': 129375977,\n",
       "  'userType': 0,\n",
       "  'vipType': 0}}"
      ]
     },
     "execution_count": 136,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "offset = 40\n",
    "id = '516997458'\n",
    "url, data = crypt_api(id, offset)\n",
    "json_text = get_json(url, data)\n",
    "json_dict = json.loads(json_text.decode(\"utf-8\"))\n",
    "comments_sum = json_dict['total']\n",
    "json_dict['comments'][0]"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": false,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}