ch16/ch16_part1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "U2ll54gJt3hR"
   },
   "source": [
    "# 머신 러닝 교과서 3판"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "f6CoM_jTt3hS"
   },
   "source": [
    "# 16장 - 순환 신경망으로 순차 데이터 모델링 (1/2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "FAhOxXi8t3hT"
   },
   "source": [
    "**아래 링크를 통해 이 노트북을 주피터 노트북 뷰어(nbviewer.jupyter.org)로 보거나 구글 코랩(colab.research.google.com)에서 실행할 수 있습니다.**\n",
    "\n",
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://nbviewer.jupyter.org/github/rickiepark/python-machine-learning-book-3rd-edition/blob/master/ch16/ch16_part1.ipynb\"><img src=\"https://jupyter.org/assets/main-logo.svg\" width=\"28\" />주피터 노트북 뷰어로 보기</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/rickiepark/python-machine-learning-book-3rd-edition/blob/master/ch16/ch16_part1.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />구글 코랩(Colab)에서 실행하기</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ivB0ZSFGt3hT"
   },
   "source": [
    "### 목차"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "57dQGeStt3hT"
   },
   "source": [
    "- 순차 데이터 소개\n",
    "    - 순차 데이터 모델링: 순서를 고려한다\n",
    "    - 시퀀스 표현\n",
    "    - 시퀀스 모델링의 종류\n",
    "- 시퀀스 모델링을 위한 RNN\n",
    "    - RNN 반복 구조 이해하기\n",
    "    - RNN의 활성화 출력 계산\n",
    "    - 은닉 순환과 출력 순환\n",
    "    - 긴 시퀀스 학습의 어려움\n",
    "    - LSTM 셀\n",
    "- 텐서플로로 시퀀스 모델링을 위한 RNN 구현하기\n",
    "    - 첫 번째 프로젝트: IMDb 영화 리뷰 감성 분석\n",
    "        - 영화 리뷰 데이터 준비\n",
    "        - 문장 인코딩을 위한 임베딩 층\n",
    "        - RNN 모델 만들기\n",
    "        - 감성 분석 작업을 위한 RNN 모델 만들기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-31T14:52:56.448296Z",
     "iopub.status.busy": "2020-12-31T14:52:56.447399Z",
     "iopub.status.idle": "2020-12-31T14:52:56.450102Z",
     "shell.execute_reply": "2020-12-31T14:52:56.450840Z"
    },
    "id": "SHqvUk9kt3hT"
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Wr5StFyQt3hT"
   },
   "source": [
    "# 순차 데이터 소개\n",
    "## 순차 데이터 모델링: 순서를 고려한다\n",
    "## 시퀀스 표현"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 282
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:52:56.456190Z",
     "iopub.status.busy": "2020-12-31T14:52:56.455553Z",
     "iopub.status.idle": "2020-12-31T14:52:56.463431Z",
     "shell.execute_reply": "2020-12-31T14:52:56.464269Z"
    },
    "id": "k1c-6p52t3hU",
    "outputId": "6748cf88-eaa8-43e7-adcd-8cd732fc6eb6"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdVm\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 18,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdVm', width=700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QL3g5J07t3hU"
   },
   "source": [
    "## 시퀀스 모델링의 종류"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 427
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:52:56.470205Z",
     "iopub.status.busy": "2020-12-31T14:52:56.469239Z",
     "iopub.status.idle": "2020-12-31T14:52:56.510740Z",
     "shell.execute_reply": "2020-12-31T14:52:56.510046Z"
    },
    "id": "Qe4WgH2Ot3hU",
    "outputId": "80f53cf1-95c9-471c-e500-bcc0ebd3a141"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdVO\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 19,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdVO', width=700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Z5XrGPuht3hU"
   },
   "source": [
    "# 시퀀스 모델링을 위한 RNN\n",
    "## RNN 반복 구조 이해하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 311
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:52:56.516239Z",
     "iopub.status.busy": "2020-12-31T14:52:56.515511Z",
     "iopub.status.idle": "2020-12-31T14:52:56.521165Z",
     "shell.execute_reply": "2020-12-31T14:52:56.520502Z"
    },
    "id": "YrVhGKS0t3hV",
    "outputId": "9243c116-7c97-422b-8377-c2059bae5b73"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdV3\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 20,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdV3', width=700)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 613
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:52:56.525160Z",
     "iopub.status.busy": "2020-12-31T14:52:56.524521Z",
     "iopub.status.idle": "2020-12-31T14:52:56.530736Z",
     "shell.execute_reply": "2020-12-31T14:52:56.531285Z"
    },
    "id": "_L2SXQ0Bt3hV",
    "outputId": "098979aa-46b9-4290-9ae7-28bf6519757c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdVs\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 21,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdVs', width=700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Dv07BsIEt3hV"
   },
   "source": [
    "## RNN의 활성화 출력 계산"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 295
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:52:56.535282Z",
     "iopub.status.busy": "2020-12-31T14:52:56.534647Z",
     "iopub.status.idle": "2020-12-31T14:52:56.540928Z",
     "shell.execute_reply": "2020-12-31T14:52:56.540327Z"
    },
    "id": "KxnXQiRSt3hV",
    "outputId": "ccb214b1-cb96-4b9b-a3db-f1a79f11d98c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdVC\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 22,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdVC', width=700)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 373
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:52:56.544851Z",
     "iopub.status.busy": "2020-12-31T14:52:56.544209Z",
     "iopub.status.idle": "2020-12-31T14:52:56.551185Z",
     "shell.execute_reply": "2020-12-31T14:52:56.551727Z"
    },
    "id": "c7_mK8qVt3hV",
    "outputId": "595f9c85-9dd0-46a7-c0af-b7c6dbd62751"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdVW\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 23,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdVW', width=700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u8Jumkd0t3hW"
   },
   "source": [
    "## 은닉 순환과 출력 순환"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 602
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:52:56.555805Z",
     "iopub.status.busy": "2020-12-31T14:52:56.555162Z",
     "iopub.status.idle": "2020-12-31T14:52:56.561794Z",
     "shell.execute_reply": "2020-12-31T14:52:56.562334Z"
    },
    "id": "iuGEUtnYt3hW",
    "outputId": "eba01507-eee9-4d74-e6a9-de7ee2754457"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdV8\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 24,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdV8', width=700)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:52:56.568669Z",
     "iopub.status.busy": "2020-12-31T14:52:56.568012Z",
     "iopub.status.idle": "2020-12-31T14:52:58.384001Z",
     "shell.execute_reply": "2020-12-31T14:52:58.383269Z"
    },
    "id": "Wvo3vxr8t3hW",
    "outputId": "bf38fd9d-3b90-42cc-87d4-69188efc0bad"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "W_xh 크기: (5, 2)\n",
      "W_oo 크기: (2, 2)\n",
      "b_h 크기: (2,)\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "tf.random.set_seed(1)\n",
    "\n",
    "rnn_layer = tf.keras.layers.SimpleRNN(\n",
    "    units=2, use_bias=True, \n",
    "    return_sequences=True)\n",
    "rnn_layer.build(input_shape=(None, None, 5))\n",
    "\n",
    "w_xh, w_oo, b_h = rnn_layer.weights\n",
    "\n",
    "print('W_xh 크기:', w_xh.shape)\n",
    "print('W_oo 크기:', w_oo.shape)\n",
    "print('b_h 크기:', b_h.shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:52:58.395104Z",
     "iopub.status.busy": "2020-12-31T14:52:58.394214Z",
     "iopub.status.idle": "2020-12-31T14:52:58.414734Z",
     "shell.execute_reply": "2020-12-31T14:52:58.414063Z"
    },
    "id": "THMnNQoEt3hW",
    "outputId": "ad0bbada-3796-43dd-a98f-50ea0a76ebcd",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "타임 스텝 0 =>\n",
      "   입력           : [[1. 1. 1. 1. 1.]]\n",
      "   은닉           : [[0.41464037 0.96012145]]\n",
      "   출력 (수동)     : [[0.39240566 0.7443311 ]]\n",
      "   SimpleRNN 출력 : [0.39240566 0.7443311 ]\n",
      "\n",
      "타임 스텝 1 =>\n",
      "   입력           : [[2. 2. 2. 2. 2.]]\n",
      "   은닉           : [[0.82928073 1.9202429 ]]\n",
      "   출력 (수동)     : [[0.80116504 0.99129474]]\n",
      "   SimpleRNN 출력 : [0.80116504 0.99129474]\n",
      "\n",
      "타임 스텝 2 =>\n",
      "   입력           : [[3. 3. 3. 3. 3.]]\n",
      "   은닉           : [[1.243921  2.8803642]]\n",
      "   출력 (수동)     : [[0.9546826 0.999307 ]]\n",
      "   SimpleRNN 출력 : [0.9546826 0.999307 ]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "x_seq = tf.convert_to_tensor(\n",
    "    [[1.0]*5, [2.0]*5, [3.0]*5],\n",
    "    dtype=tf.float32)\n",
    "\n",
    "\n",
    "## SimepleRNN의 출력:\n",
    "output = rnn_layer(tf.reshape(x_seq, shape=(1, 3, 5)))\n",
    "\n",
    "## 수동으로 출력 계산하기:\n",
    "out_man = []\n",
    "for t in range(len(x_seq)):\n",
    "    xt = tf.reshape(x_seq[t], (1, 5))\n",
    "    print('타임 스텝 {} =>'.format(t))\n",
    "    print('   입력           :', xt.numpy())\n",
    "    \n",
    "    ht = tf.matmul(xt, w_xh) + b_h    \n",
    "    print('   은닉           :', ht.numpy())\n",
    "    \n",
    "    if t>0:\n",
    "        prev_o = out_man[t-1]\n",
    "    else:\n",
    "        prev_o = tf.zeros(shape=(ht.shape))\n",
    "        \n",
    "    ot = ht + tf.matmul(prev_o, w_oo)\n",
    "    ot = tf.math.tanh(ot)\n",
    "    out_man.append(ot)\n",
    "    print('   출력 (수동)     :', ot.numpy())\n",
    "    print('   SimpleRNN 출력 :'.format(t), output[0][t].numpy())\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oTJJZpjUt3hX"
   },
   "source": [
    "## 긴 시퀀스 학습의 어려움"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 352
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:52:58.419575Z",
     "iopub.status.busy": "2020-12-31T14:52:58.418902Z",
     "iopub.status.idle": "2020-12-31T14:52:58.424237Z",
     "shell.execute_reply": "2020-12-31T14:52:58.423554Z"
    },
    "id": "BrsADvYht3hX",
    "outputId": "82d014fd-5dd2-42be-a40e-98a487719cf6"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdV4\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 27,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdV4', width=700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RtDltN0gt3hX"
   },
   "source": [
    "## LSTM 셀"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 381
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:52:58.429035Z",
     "iopub.status.busy": "2020-12-31T14:52:58.428310Z",
     "iopub.status.idle": "2020-12-31T14:52:58.438957Z",
     "shell.execute_reply": "2020-12-31T14:52:58.439507Z"
    },
    "id": "6YpVdbeTt3hX",
    "outputId": "7d781324-a8b7-4105-d82d-4239ca947710"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdVR\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 28,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdVR', width=700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qNOM8cBit3hX"
   },
   "source": [
    "# 텐서플로로 시퀀스 모델링을 위한 RNN 구현하기\n",
    "## 첫 번째 프로젝트: IMDb 영화 리뷰 감성 분석\n",
    "### 영화 리뷰 데이터 준비"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-31T14:52:58.443781Z",
     "iopub.status.busy": "2020-12-31T14:52:58.443146Z",
     "iopub.status.idle": "2020-12-31T14:52:58.776050Z",
     "shell.execute_reply": "2020-12-31T14:52:58.775357Z"
    },
    "id": "EDdVCNv5t3hY"
   },
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import tensorflow_datasets as tfds\n",
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gUgiOmnUv7DQ",
    "outputId": "4b62c20c-c556-4f83-af5b-561ab958179a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory ‘../ch08’: File exists\n",
      "--2021-01-03 15:00:12--  https://github.com/rickiepark/python-machine-learning-book-3rd-edition/raw/master/ch08/movie_data.csv.gz\n",
      "Resolving github.com (github.com)... 192.30.255.113\n",
      "Connecting to github.com (github.com)|192.30.255.113|:443... connected.\n",
      "HTTP request sent, awaiting response... 302 Found\n",
      "Location: https://raw.githubusercontent.com/rickiepark/python-machine-learning-book-3rd-edition/master/ch08/movie_data.csv.gz [following]\n",
      "--2021-01-03 15:00:12--  https://raw.githubusercontent.com/rickiepark/python-machine-learning-book-3rd-edition/master/ch08/movie_data.csv.gz\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 26521894 (25M) [application/octet-stream]\n",
      "Saving to: ‘../ch08/movie_data.csv.gz’\n",
      "\n",
      "../ch08/movie_data. 100%[===================>]  25.29M   129MB/s    in 0.2s    \n",
      "\n",
      "2021-01-03 15:00:12 (129 MB/s) - ‘../ch08/movie_data.csv.gz’ saved [26521894/26521894]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 코랩에서 실행하는 경우 다음 코드를 실행하세요.\n",
    "!mkdir ../ch08\n",
    "!wget https://github.com/rickiepark/python-machine-learning-book-3rd-edition/raw/master/ch08/movie_data.csv.gz -O ../ch08/movie_data.csv.gz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-31T14:52:58.786349Z",
     "iopub.status.busy": "2020-12-31T14:52:58.785414Z",
     "iopub.status.idle": "2020-12-31T14:53:00.001551Z",
     "shell.execute_reply": "2020-12-31T14:53:00.002307Z"
    },
    "id": "9HcKmMPmt3hY"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import gzip\n",
    "import shutil\n",
    "\n",
    "\n",
    "with gzip.open('../ch08/movie_data.csv.gz', 'rb') as f_in, open('movie_data.csv', 'wb') as f_out:\n",
    "    shutil.copyfileobj(f_in, f_out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 204
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:00.009102Z",
     "iopub.status.busy": "2020-12-31T14:53:00.008204Z",
     "iopub.status.idle": "2020-12-31T14:53:00.445362Z",
     "shell.execute_reply": "2020-12-31T14:53:00.444609Z"
    },
    "id": "I-9BuHr_t3hY",
    "outputId": "3855da57-abf1-43c3-bd05-8ada297359c9"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>review</th>\n",
       "      <th>sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>49995</th>\n",
       "      <td>OK, lets start with the best. the building. al...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49996</th>\n",
       "      <td>The British 'heritage film' industry is out of...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49997</th>\n",
       "      <td>I don't even know where to begin on this one. ...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49998</th>\n",
       "      <td>Richard Tyler is a little boy who is scared of...</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49999</th>\n",
       "      <td>I waited long to watch this movie. Also becaus...</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                  review  sentiment\n",
       "49995  OK, lets start with the best. the building. al...          0\n",
       "49996  The British 'heritage film' industry is out of...          0\n",
       "49997  I don't even know where to begin on this one. ...          0\n",
       "49998  Richard Tyler is a little boy who is scared of...          0\n",
       "49999  I waited long to watch this movie. Also becaus...          1"
      ]
     },
     "execution_count": 32,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('movie_data.csv', encoding='utf-8')\n",
    "\n",
    "df.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:00.520021Z",
     "iopub.status.busy": "2020-12-31T14:53:00.519118Z",
     "iopub.status.idle": "2020-12-31T14:53:00.531067Z",
     "shell.execute_reply": "2020-12-31T14:53:00.530180Z"
    },
    "id": "kpF_RvlVt3hY",
    "outputId": "da6b7a27-16ce-49f5-c471-21785d790f46"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "b'In 1974, the teenager Martha Moxley (Maggie Grace)' 1\n",
      "b'OK... so... I really like Kris Kristofferson and h' 0\n",
      "b'***SPOILER*** Do not read this, if you think about' 0\n"
     ]
    }
   ],
   "source": [
    "# 단계 1: 데이터셋 만들기\n",
    "target = df.pop('sentiment')\n",
    "\n",
    "ds_raw = tf.data.Dataset.from_tensor_slices(\n",
    "    (df.values, target.values))\n",
    "\n",
    "## 확인:\n",
    "for ex in ds_raw.take(3):\n",
    "    tf.print(ex[0].numpy()[0][:50], ex[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8IBmW08tt3hY"
   },
   "source": [
    " * **훈련/검증/테스트 분할**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:00.537026Z",
     "iopub.status.busy": "2020-12-31T14:53:00.536266Z",
     "iopub.status.idle": "2020-12-31T14:53:00.540439Z",
     "shell.execute_reply": "2020-12-31T14:53:00.539876Z"
    },
    "id": "89xqs9SXt3hZ"
   },
   "outputs": [],
   "source": [
    "tf.random.set_seed(1)\n",
    "\n",
    "ds_raw = ds_raw.shuffle(\n",
    "    50000, reshuffle_each_iteration=False)\n",
    "\n",
    "ds_raw_test = ds_raw.take(25000)\n",
    "ds_raw_train_valid = ds_raw.skip(25000)\n",
    "ds_raw_train = ds_raw_train_valid.take(20000)\n",
    "ds_raw_valid = ds_raw_train_valid.skip(20000)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8eB8MAjOt3hZ"
   },
   "source": [
    " * **Tokenizer와 Encoder**\n",
    "   * `tfds.deprecated.text.Tokenizer`: https://www.tensorflow.org/datasets/api_docs/python/tfds/deprecated/text/Tokenizer\n",
    "   * `tfds.deprecated.text.TokenTextEncoder`: https://www.tensorflow.org/datasets/api_docs/python/tfds/deprecated/text/TokenTextEncoder"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hzP-YuF3t3hZ"
   },
   "source": [
    " * **시퀀스 인코딩: 각 시퀀스에서 마지막 100개 원소만 유지하기**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:00.547157Z",
     "iopub.status.busy": "2020-12-31T14:53:00.546406Z",
     "iopub.status.idle": "2020-12-31T14:53:04.613801Z",
     "shell.execute_reply": "2020-12-31T14:53:04.612898Z"
    },
    "id": "zH3IFCust3hZ",
    "outputId": "8f658291-2943-45df-a4fe-c50a9fed5172",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "어휘 사전 크기: 87007\n"
     ]
    }
   ],
   "source": [
    "## 단계 2: 고유 토큰(단어) 찾기\n",
    "from collections import Counter\n",
    "\n",
    "tokenizer = tfds.deprecated.text.Tokenizer()\n",
    "token_counts = Counter()\n",
    "\n",
    "for example in ds_raw_train:\n",
    "    tokens = tokenizer.tokenize(example[0].numpy()[0])\n",
    "    token_counts.update(tokens)\n",
    "    \n",
    "print('어휘 사전 크기:', len(token_counts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:04.657429Z",
     "iopub.status.busy": "2020-12-31T14:53:04.637210Z",
     "iopub.status.idle": "2020-12-31T14:53:04.747652Z",
     "shell.execute_reply": "2020-12-31T14:53:04.746754Z"
    },
    "id": "yzd68Ikut3hZ",
    "outputId": "508ab350-5cee-4620-def8-3acd81306f31"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[232, 9, 270, 1123]"
      ]
     },
     "execution_count": 36,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## 단계 3: 고유 토큰을 정수로 인코딩하기\n",
    "encoder = tfds.deprecated.text.TokenTextEncoder(token_counts)\n",
    "\n",
    "example_str = 'This is an example!'\n",
    "encoder.encode(example_str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:04.753242Z",
     "iopub.status.busy": "2020-12-31T14:53:04.752412Z",
     "iopub.status.idle": "2020-12-31T14:53:04.755655Z",
     "shell.execute_reply": "2020-12-31T14:53:04.754832Z"
    },
    "id": "o2_YKm8bt3ha"
   },
   "outputs": [],
   "source": [
    "## 단계 3-A: 변환 함수 정의하기\n",
    "def encode(text_tensor, label):\n",
    "    text = text_tensor.numpy()[0]\n",
    "    encoded_text = encoder.encode(text)\n",
    "    return encoded_text, label\n",
    "\n",
    "## 단계 3-B: 인코딩 함수를 텐서플로 연산으로 감싸기\n",
    "def encode_map_fn(text, label):\n",
    "    return tf.py_function(encode, inp=[text, label], \n",
    "                          Tout=(tf.int64, tf.int64))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:04.766645Z",
     "iopub.status.busy": "2020-12-31T14:53:04.766238Z",
     "iopub.status.idle": "2020-12-31T14:53:05.315769Z",
     "shell.execute_reply": "2020-12-31T14:53:05.316485Z"
    },
    "id": "frJTGpBxt3ha",
    "outputId": "6f8df563-771d-4786-f283-01d90f0547e1"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "시퀀스 길이: (24,)\n",
      "시퀀스 길이: (179,)\n",
      "시퀀스 길이: (262,)\n",
      "시퀀스 길이: (535,)\n",
      "시퀀스 길이: (130,)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(<tf.Tensor: shape=(130,), dtype=int64, numpy=\n",
       " array([  579,  1296,    32,   425,    40,   763,  9267,    65,   280,\n",
       "          308,     6,   481,   155,   473,     2,     3,   684,     9,\n",
       "          781,   176,   959,   730,  3917,    67,  9905,    13,   277,\n",
       "           24,    35,   371, 16368,     6,    14, 17231,    29,   187,\n",
       "         1651,   489,   503,   480,   143,    32,   270,  5851,  2402,\n",
       "           13,  3592,  3443,   425,  3313,   256,   257,  1577,   117,\n",
       "            8,   698,   270,   564,    56,     8,    42,  7517,  2629,\n",
       "          820,    25,    60,    79,   343,    32,   645,    14,   528,\n",
       "          241,    32,  1980,     8,    56,     8,    42,  1364,   573,\n",
       "         5183,    43,    12,  3870,    32,   312,   642,   251,  1401,\n",
       "        17232,     8,   698,   257,   750,     2,     9,    76,   235,\n",
       "            8,    42,   235,   840,   666,   258, 17233,   419,    32,\n",
       "        17234,   585,   420,   840,    25,    40,    13,    14,   198,\n",
       "          266,   623,   173,   179,  4103,   216,    25,   616, 14185,\n",
       "          186,    35, 16250,   120])>,\n",
       " <tf.Tensor: shape=(), dtype=int64, numpy=0>)"
      ]
     },
     "execution_count": 38,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ds_train = ds_raw_train.map(encode_map_fn)\n",
    "ds_valid = ds_raw_valid.map(encode_map_fn)\n",
    "ds_test = ds_raw_test.map(encode_map_fn)\n",
    "\n",
    "tf.random.set_seed(1)\n",
    "for example in ds_train.shuffle(1000).take(5):\n",
    "    print('시퀀스 길이:', example[0].shape)\n",
    "    \n",
    "example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JoiIum3jt3ha"
   },
   "source": [
    " * **batch() 대 padded_batch()**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YdUNeW8rt3ha"
   },
   "source": [
    "```python\n",
    "\n",
    "# 에러 발생\n",
    "BATCH_SIZE = 32\n",
    "train_data = all_encoded_data.batch(BATCH_SIZE)\n",
    "\n",
    "next(iter(train_data))\n",
    "\n",
    "# 이 코드는 에러를 발생시킵니다\n",
    "# 이 데이터셋에는 .batch()를 적용할 수 없습니다\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:05.323061Z",
     "iopub.status.busy": "2020-12-31T14:53:05.322063Z",
     "iopub.status.idle": "2020-12-31T14:53:05.568975Z",
     "shell.execute_reply": "2020-12-31T14:53:05.568087Z"
    },
    "id": "tXrml4TEt3ha",
    "outputId": "218c0401-661c-435d-9f8e-e02a4a26ce55"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "개별 샘플 크기: (119,)\n",
      "개별 샘플 크기: (688,)\n",
      "개별 샘플 크기: (308,)\n",
      "개별 샘플 크기: (204,)\n",
      "개별 샘플 크기: (326,)\n",
      "개별 샘플 크기: (240,)\n",
      "개별 샘플 크기: (127,)\n",
      "개별 샘플 크기: (453,)\n",
      "배치 차원: (4, 688)\n",
      "배치 차원: (4, 453)\n"
     ]
    }
   ],
   "source": [
    "## 일부 데이터 추출하기\n",
    "ds_subset = ds_train.take(8)\n",
    "for example in ds_subset:\n",
    "    print('개별 샘플 크기:', example[0].shape)\n",
    "\n",
    "## 배치 데이터 만들기\n",
    "ds_batched = ds_subset.padded_batch(\n",
    "    4, padded_shapes=([-1], []))\n",
    "\n",
    "for batch in ds_batched:\n",
    "    print('배치 차원:', batch[0].shape)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:05.575398Z",
     "iopub.status.busy": "2020-12-31T14:53:05.574415Z",
     "iopub.status.idle": "2020-12-31T14:53:05.579834Z",
     "shell.execute_reply": "2020-12-31T14:53:05.579134Z"
    },
    "id": "a2A9ol3Jt3hb"
   },
   "outputs": [],
   "source": [
    "## 배치 데이터 만들기\n",
    "train_data = ds_train.padded_batch(\n",
    "    32, padded_shapes=([-1],[]))\n",
    "\n",
    "valid_data = ds_valid.padded_batch(\n",
    "    32, padded_shapes=([-1],[]))\n",
    "\n",
    "test_data = ds_test.padded_batch(\n",
    "    32, padded_shapes=([-1],[]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qUybBb3zt3hb"
   },
   "source": [
    "### 문장 인코딩을 위한 임베딩 층\n",
    "\n",
    " * `input_dim`: 단어 개수, 즉 최대 정수 인덱스 + 1.\n",
    " * `output_dim`: \n",
    " * `input_length`: (패딩된) 시퀀스 길이\n",
    "    * 예를 들어, `'This is an example' -> [0, 0, 0, 0, 0, 0, 3, 1, 8, 9]`   \n",
    "    => input_lenght는 10\n",
    "\n",
    " * 이 층을 호출할 때 입력으로 정수값을 받습니다. 임베딩 층이 정수를 `[output_dim]` 크기의 실수 벡터로 변환합니다\n",
    "   * 입력 크기가 `[BATCH_SIZE]`이면 출력 크기는 `[BATCH_SIZE, output_dim]`가 됩니다\n",
    "   * 입력 크기가 `[BATCH_SIZE, 10]`이면 출력 크기는 `[BATCH_SIZE, 10, output_dim]`가 됩니다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 598
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:05.584719Z",
     "iopub.status.busy": "2020-12-31T14:53:05.584041Z",
     "iopub.status.idle": "2020-12-31T14:53:05.593015Z",
     "shell.execute_reply": "2020-12-31T14:53:05.592348Z"
    },
    "id": "8i2mrfBWt3hb",
    "outputId": "5325f610-5938-4441-dd2c-f18e98f7717a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdV0\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 41,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdV0', width=700)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:05.601333Z",
     "iopub.status.busy": "2020-12-31T14:53:05.600692Z",
     "iopub.status.idle": "2020-12-31T14:53:05.618008Z",
     "shell.execute_reply": "2020-12-31T14:53:05.617346Z"
    },
    "id": "esPfmEMMt3hb",
    "outputId": "d9069425-83fc-45ab-e71f-7c0126fe5020"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embed-layer (Embedding)      (None, 20, 6)             600       \n",
      "=================================================================\n",
      "Total params: 600\n",
      "Trainable params: 600\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.layers import Embedding\n",
    "\n",
    "\n",
    "model = tf.keras.Sequential()\n",
    "\n",
    "model.add(Embedding(input_dim=100,\n",
    "                    output_dim=6,\n",
    "                    input_length=20,\n",
    "                    name='embed-layer'))\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rqwKaKkqt3hb"
   },
   "source": [
    "### RNN 모델 만들기\n",
    "\n",
    "* **케라스 RNN 층:**\n",
    "  * `tf.keras.layers.SimpleRNN(units, return_sequences=False)`\n",
    "  * `tf.keras.layers.LSTM(..)`\n",
    "  * `tf.keras.layers.GRU(..)`\n",
    "  * `tf.keras.layers.Bidirectional()`\n",
    " \n",
    "* **`return_sequenes=?` 결정하기**\n",
    "  * 다층 RNN이면 마지막 층을 제외하고 모든 RNN 층을 `return_sequenes=True`로 지정합니다\n",
    "  * 마지막 RNN 층은 문제의 종류에 따라 결정됩니다:\n",
    "     * 다대다: -> `return_sequences=True`\n",
    "     * 다대일: -> `return_sequenes=False`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:05.626167Z",
     "iopub.status.busy": "2020-12-31T14:53:05.625414Z",
     "iopub.status.idle": "2020-12-31T14:53:05.706279Z",
     "shell.execute_reply": "2020-12-31T14:53:05.707065Z"
    },
    "id": "oy4iAXCHt3hc",
    "outputId": "81fa735e-18aa-4b51-d012-b86ab96b481d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, None, 32)          32000     \n",
      "_________________________________________________________________\n",
      "simple_rnn_2 (SimpleRNN)     (None, None, 32)          2080      \n",
      "_________________________________________________________________\n",
      "simple_rnn_3 (SimpleRNN)     (None, 32)                2080      \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 36,193\n",
      "Trainable params: 36,193\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "## SimpleRNN 층으로 RNN 모델 만들기\n",
    "from tensorflow.keras import Sequential\n",
    "from tensorflow.keras.layers import Embedding\n",
    "from tensorflow.keras.layers import SimpleRNN\n",
    "from tensorflow.keras.layers import Dense\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(1000, 32))\n",
    "model.add(SimpleRNN(32, return_sequences=True))\n",
    "model.add(SimpleRNN(32))\n",
    "model.add(Dense(1))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:05.715529Z",
     "iopub.status.busy": "2020-12-31T14:53:05.714882Z",
     "iopub.status.idle": "2020-12-31T14:53:06.010097Z",
     "shell.execute_reply": "2020-12-31T14:53:06.009262Z"
    },
    "id": "hP_dd6GUt3hc",
    "outputId": "d01f6655-49f7-4076-a9e7-4c768874e3a3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_2\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (None, None, 32)          320000    \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, None, 32)          8320      \n",
      "_________________________________________________________________\n",
      "lstm_1 (LSTM)                (None, 32)                8320      \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 336,673\n",
      "Trainable params: 336,673\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "## LSTM 층으로 RNN 모델 만들기\n",
    "from tensorflow.keras.layers import LSTM\n",
    "\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(10000, 32))\n",
    "model.add(LSTM(32, return_sequences=True))\n",
    "model.add(LSTM(32))\n",
    "model.add(Dense(1))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:06.020415Z",
     "iopub.status.busy": "2020-12-31T14:53:06.018360Z",
     "iopub.status.idle": "2020-12-31T14:53:06.272493Z",
     "shell.execute_reply": "2020-12-31T14:53:06.271598Z"
    },
    "id": "6UQfBf4yt3hc",
    "outputId": "6d5cfc18-064d-44fc-f3fb-3df08141ea4e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_3\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_2 (Embedding)      (None, None, 32)          320000    \n",
      "_________________________________________________________________\n",
      "gru (GRU)                    (None, None, 32)          6336      \n",
      "_________________________________________________________________\n",
      "gru_1 (GRU)                  (None, 32)                6336      \n",
      "_________________________________________________________________\n",
      "dense_2 (Dense)              (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 332,705\n",
      "Trainable params: 332,705\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "## GRU 층으로 RNN 모델 만들기\n",
    "from tensorflow.keras.layers import GRU\n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(10000, 32))\n",
    "model.add(GRU(32, return_sequences=True))\n",
    "model.add(GRU(32))\n",
    "model.add(Dense(1))\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TFSW6uDst3hc"
   },
   "source": [
    "### 감성 분석 작업을 위한 RNN 모델 만들기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T14:53:06.298278Z",
     "iopub.status.busy": "2020-12-31T14:53:06.297321Z",
     "iopub.status.idle": "2020-12-31T15:31:18.726835Z",
     "shell.execute_reply": "2020-12-31T15:31:18.727434Z"
    },
    "id": "FszX7_Qft3hc",
    "outputId": "49eba62b-3326-4727-89fe-1d45f3133ac9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_4\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embed-layer (Embedding)      (None, None, 20)          1740180   \n",
      "_________________________________________________________________\n",
      "bidir-lstm (Bidirectional)   (None, 128)               43520     \n",
      "_________________________________________________________________\n",
      "dense_3 (Dense)              (None, 64)                8256      \n",
      "_________________________________________________________________\n",
      "dense_4 (Dense)              (None, 1)                 65        \n",
      "=================================================================\n",
      "Total params: 1,792,021\n",
      "Trainable params: 1,792,021\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n",
      "Epoch 1/10\n",
      "625/625 [==============================] - 65s 92ms/step - loss: 0.6097 - accuracy: 0.6398 - val_loss: 0.4254 - val_accuracy: 0.8094\n",
      "Epoch 2/10\n",
      "625/625 [==============================] - 57s 91ms/step - loss: 0.3008 - accuracy: 0.8795 - val_loss: 0.4553 - val_accuracy: 0.8122\n",
      "Epoch 3/10\n",
      "625/625 [==============================] - 57s 91ms/step - loss: 0.2152 - accuracy: 0.9166 - val_loss: 0.4290 - val_accuracy: 0.8446\n",
      "Epoch 4/10\n",
      "625/625 [==============================] - 57s 91ms/step - loss: 0.0925 - accuracy: 0.9691 - val_loss: 0.5609 - val_accuracy: 0.7938\n",
      "Epoch 5/10\n",
      "625/625 [==============================] - 57s 91ms/step - loss: 0.0603 - accuracy: 0.9803 - val_loss: 0.6942 - val_accuracy: 0.8286\n",
      "Epoch 6/10\n",
      "625/625 [==============================] - 58s 92ms/step - loss: 0.0430 - accuracy: 0.9859 - val_loss: 0.6372 - val_accuracy: 0.8214\n",
      "Epoch 7/10\n",
      "625/625 [==============================] - 57s 91ms/step - loss: 0.0349 - accuracy: 0.9897 - val_loss: 0.7225 - val_accuracy: 0.8032\n",
      "Epoch 8/10\n",
      "625/625 [==============================] - 57s 91ms/step - loss: 0.0470 - accuracy: 0.9845 - val_loss: 0.6835 - val_accuracy: 0.8258\n",
      "Epoch 9/10\n",
      "625/625 [==============================] - 57s 91ms/step - loss: 0.0213 - accuracy: 0.9931 - val_loss: 0.8062 - val_accuracy: 0.8446\n",
      "Epoch 10/10\n",
      "625/625 [==============================] - 56s 90ms/step - loss: 0.0181 - accuracy: 0.9948 - val_loss: 0.8472 - val_accuracy: 0.8232\n",
      "782/782 [==============================] - 32s 40ms/step - loss: 0.8447 - accuracy: 0.8226\n",
      "테스트 정확도: 82.26%\n"
     ]
    }
   ],
   "source": [
    "embedding_dim = 20\n",
    "vocab_size = len(token_counts) + 2\n",
    "\n",
    "tf.random.set_seed(1)\n",
    "\n",
    "## 모델 생성\n",
    "bi_lstm_model = tf.keras.Sequential([\n",
    "    tf.keras.layers.Embedding(\n",
    "        input_dim=vocab_size,\n",
    "        output_dim=embedding_dim,\n",
    "        name='embed-layer'),\n",
    "    \n",
    "    tf.keras.layers.Bidirectional(\n",
    "        tf.keras.layers.LSTM(64, name='lstm-layer'),\n",
    "        name='bidir-lstm'), \n",
    "\n",
    "    tf.keras.layers.Dense(64, activation='relu'),\n",
    "    \n",
    "    tf.keras.layers.Dense(1, activation='sigmoid')\n",
    "])\n",
    "\n",
    "bi_lstm_model.summary()\n",
    "\n",
    "## 컴파일과 훈련:\n",
    "bi_lstm_model.compile(\n",
    "    optimizer=tf.keras.optimizers.Adam(1e-3),\n",
    "    loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),\n",
    "    metrics=['accuracy'])\n",
    "\n",
    "history = bi_lstm_model.fit(\n",
    "    train_data, \n",
    "    validation_data=valid_data, \n",
    "    epochs=10)\n",
    "\n",
    "## 테스트 데이터에서 평가\n",
    "test_results= bi_lstm_model.evaluate(test_data)\n",
    "print('테스트 정확도: {:.2f}%'.format(test_results[1]*100))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-31T15:31:18.733388Z",
     "iopub.status.busy": "2020-12-31T15:31:18.732717Z",
     "iopub.status.idle": "2020-12-31T15:31:19.095760Z",
     "shell.execute_reply": "2020-12-31T15:31:19.096539Z"
    },
    "id": "yAYzzBm2t3hd"
   },
   "outputs": [],
   "source": [
    "if not os.path.exists('models'):\n",
    "    os.mkdir('models')\n",
    "\n",
    "\n",
    "bi_lstm_model.save('models/Bidir-LSTM-full-length-seq.h5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bOGhajp5t3hd"
   },
   "source": [
    " * **짧은 시퀀스에 SimpleRNN 적용하기**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-31T15:31:19.114411Z",
     "iopub.status.busy": "2020-12-31T15:31:19.113501Z",
     "iopub.status.idle": "2020-12-31T15:31:19.116108Z",
     "shell.execute_reply": "2020-12-31T15:31:19.115368Z"
    },
    "id": "TDbCSxpIt3hd"
   },
   "outputs": [],
   "source": [
    "def preprocess_datasets(\n",
    "    ds_raw_train, \n",
    "    ds_raw_valid, \n",
    "    ds_raw_test,\n",
    "    max_seq_length=None,\n",
    "    batch_size=32):\n",
    "    \n",
    "    ## 단계 1: (데이터셋 만들기 이미 완료)\n",
    "    ## 단계 2: 고유 토큰 찾기\n",
    "    tokenizer = tfds.deprecated.text.Tokenizer()\n",
    "    token_counts = Counter()\n",
    "\n",
    "    for example in ds_raw_train:\n",
    "        tokens = tokenizer.tokenize(example[0].numpy()[0])\n",
    "        if max_seq_length is not None:\n",
    "            tokens = tokens[-max_seq_length:]\n",
    "        token_counts.update(tokens)\n",
    "\n",
    "    print('어휘 사전 크기:', len(token_counts))\n",
    "\n",
    "\n",
    "    ## 단계 3: 텍스트 인코딩하기\n",
    "    encoder = tfds.deprecated.text.TokenTextEncoder(token_counts)\n",
    "    def encode(text_tensor, label):\n",
    "        text = text_tensor.numpy()[0]\n",
    "        encoded_text = encoder.encode(text)\n",
    "        if max_seq_length is not None:\n",
    "            encoded_text = encoded_text[-max_seq_length:]\n",
    "        return encoded_text, label\n",
    "\n",
    "    def encode_map_fn(text, label):\n",
    "        return tf.py_function(encode, inp=[text, label], \n",
    "                              Tout=(tf.int64, tf.int64))\n",
    "\n",
    "    ds_train = ds_raw_train.map(encode_map_fn)\n",
    "    ds_valid = ds_raw_valid.map(encode_map_fn)\n",
    "    ds_test = ds_raw_test.map(encode_map_fn)\n",
    "\n",
    "    ## 단계 4: 배치 데이터 만들기\n",
    "    train_data = ds_train.padded_batch(\n",
    "        batch_size, padded_shapes=([-1],[]))\n",
    "\n",
    "    valid_data = ds_valid.padded_batch(\n",
    "        batch_size, padded_shapes=([-1],[]))\n",
    "\n",
    "    test_data = ds_test.padded_batch(\n",
    "        batch_size, padded_shapes=([-1],[]))\n",
    "\n",
    "    return (train_data, valid_data, \n",
    "            test_data, len(token_counts))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-31T15:31:19.128046Z",
     "iopub.status.busy": "2020-12-31T15:31:19.127068Z",
     "iopub.status.idle": "2020-12-31T15:31:19.129259Z",
     "shell.execute_reply": "2020-12-31T15:31:19.129831Z"
    },
    "id": "JVuxsNTRt3hd"
   },
   "outputs": [],
   "source": [
    "def build_rnn_model(embedding_dim, vocab_size,\n",
    "                    recurrent_type='SimpleRNN',\n",
    "                    n_recurrent_units=64,\n",
    "                    n_recurrent_layers=1,\n",
    "                    bidirectional=True):\n",
    "\n",
    "    tf.random.set_seed(1)\n",
    "\n",
    "    # 모델 생성\n",
    "    model = tf.keras.Sequential()\n",
    "    \n",
    "    model.add(\n",
    "        Embedding(\n",
    "            input_dim=vocab_size,\n",
    "            output_dim=embedding_dim,\n",
    "            name='embed-layer')\n",
    "    )\n",
    "    \n",
    "    for i in range(n_recurrent_layers):\n",
    "        return_sequences = (i < n_recurrent_layers-1)\n",
    "            \n",
    "        if recurrent_type == 'SimpleRNN':\n",
    "            recurrent_layer = SimpleRNN(\n",
    "                units=n_recurrent_units, \n",
    "                return_sequences=return_sequences,\n",
    "                name='simprnn-layer-{}'.format(i))\n",
    "        elif recurrent_type == 'LSTM':\n",
    "            recurrent_layer = LSTM(\n",
    "                units=n_recurrent_units, \n",
    "                return_sequences=return_sequences,\n",
    "                name='lstm-layer-{}'.format(i))\n",
    "        elif recurrent_type == 'GRU':\n",
    "            recurrent_layer = GRU(\n",
    "                units=n_recurrent_units, \n",
    "                return_sequences=return_sequences,\n",
    "                name='gru-layer-{}'.format(i))\n",
    "        \n",
    "        if bidirectional:\n",
    "            recurrent_layer = Bidirectional(\n",
    "                recurrent_layer, name='bidir-'+recurrent_layer.name)\n",
    "            \n",
    "        model.add(recurrent_layer)\n",
    "\n",
    "    model.add(tf.keras.layers.Dense(64, activation='relu'))\n",
    "    model.add(tf.keras.layers.Dense(1, activation='sigmoid'))\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T15:31:19.135494Z",
     "iopub.status.busy": "2020-12-31T15:31:19.134630Z",
     "iopub.status.idle": "2020-12-31T15:31:23.127554Z",
     "shell.execute_reply": "2020-12-31T15:31:23.127922Z"
    },
    "id": "WaDu41Pht3hd",
    "outputId": "ec91b61a-adc8-4e5e-c8d5-caf9ee14d45b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "어휘 사전 크기: 58063\n",
      "Model: \"sequential_5\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embed-layer (Embedding)      (None, None, 20)          1161300   \n",
      "_________________________________________________________________\n",
      "bidir-simprnn-layer-0 (Bidir (None, 128)               10880     \n",
      "_________________________________________________________________\n",
      "dense_5 (Dense)              (None, 64)                8256      \n",
      "_________________________________________________________________\n",
      "dense_6 (Dense)              (None, 1)                 65        \n",
      "=================================================================\n",
      "Total params: 1,180,501\n",
      "Trainable params: 1,180,501\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.layers import Bidirectional\n",
    "\n",
    "\n",
    "batch_size = 32\n",
    "embedding_dim = 20\n",
    "max_seq_length = 100\n",
    "\n",
    "train_data, valid_data, test_data, n = preprocess_datasets(\n",
    "    ds_raw_train, ds_raw_valid, ds_raw_test, \n",
    "    max_seq_length=max_seq_length, \n",
    "    batch_size=batch_size\n",
    ")\n",
    "\n",
    "\n",
    "vocab_size = n + 2\n",
    "\n",
    "rnn_model = build_rnn_model(\n",
    "    embedding_dim, vocab_size,\n",
    "    recurrent_type='SimpleRNN', \n",
    "    n_recurrent_units=64,\n",
    "    n_recurrent_layers=1,\n",
    "    bidirectional=True)\n",
    "\n",
    "rnn_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T15:31:23.140595Z",
     "iopub.status.busy": "2020-12-31T15:31:23.140192Z",
     "iopub.status.idle": "2020-12-31T15:37:30.416144Z",
     "shell.execute_reply": "2020-12-31T15:37:30.416906Z"
    },
    "id": "Q43zdoSVt3hd",
    "outputId": "498c01b7-07f9-43f9-f358-2f831a2a1cec"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "625/625 [==============================] - 109s 172ms/step - loss: 0.7017 - accuracy: 0.5042 - val_loss: 0.7046 - val_accuracy: 0.4940\n",
      "Epoch 2/10\n",
      "625/625 [==============================] - 110s 175ms/step - loss: 0.6998 - accuracy: 0.5164 - val_loss: 0.7093 - val_accuracy: 0.5026\n",
      "Epoch 3/10\n",
      "625/625 [==============================] - 109s 174ms/step - loss: 0.6869 - accuracy: 0.5491 - val_loss: 0.6050 - val_accuracy: 0.6702\n",
      "Epoch 4/10\n",
      "625/625 [==============================] - 112s 179ms/step - loss: 0.5571 - accuracy: 0.7173 - val_loss: 0.6158 - val_accuracy: 0.6678\n",
      "Epoch 5/10\n",
      "625/625 [==============================] - 110s 176ms/step - loss: 0.4779 - accuracy: 0.7926 - val_loss: 0.5643 - val_accuracy: 0.7386\n",
      "Epoch 6/10\n",
      "625/625 [==============================] - 110s 176ms/step - loss: 0.3756 - accuracy: 0.8505 - val_loss: 0.5608 - val_accuracy: 0.7732\n",
      "Epoch 7/10\n",
      "625/625 [==============================] - 109s 174ms/step - loss: 0.2714 - accuracy: 0.9015 - val_loss: 0.6027 - val_accuracy: 0.7792\n",
      "Epoch 8/10\n",
      "625/625 [==============================] - 109s 175ms/step - loss: 0.2267 - accuracy: 0.9208 - val_loss: 0.6233 - val_accuracy: 0.7790\n",
      "Epoch 9/10\n",
      "625/625 [==============================] - 111s 177ms/step - loss: 0.2443 - accuracy: 0.8974 - val_loss: 0.6566 - val_accuracy: 0.6240\n",
      "Epoch 10/10\n",
      "625/625 [==============================] - 110s 175ms/step - loss: 0.4436 - accuracy: 0.7828 - val_loss: 0.6292 - val_accuracy: 0.7580\n"
     ]
    }
   ],
   "source": [
    "rnn_model.compile(optimizer=tf.keras.optimizers.Adam(1e-3),\n",
    "                  loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),\n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "\n",
    "history = rnn_model.fit(\n",
    "    train_data, \n",
    "    validation_data=valid_data, \n",
    "    epochs=10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T15:37:30.422212Z",
     "iopub.status.busy": "2020-12-31T15:37:30.421332Z",
     "iopub.status.idle": "2020-12-31T15:37:51.102354Z",
     "shell.execute_reply": "2020-12-31T15:37:51.102953Z"
    },
    "id": "WaiatV_It3he",
    "outputId": "ec96d574-7033-484a-8193-4e8fe76e37ae"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "782/782 [==============================] - 25s 31ms/step - loss: 0.6021 - accuracy: 0.7717\n"
     ]
    }
   ],
   "source": [
    "results = rnn_model.evaluate(test_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T15:37:51.108896Z",
     "iopub.status.busy": "2020-12-31T15:37:51.108181Z",
     "iopub.status.idle": "2020-12-31T15:37:51.112037Z",
     "shell.execute_reply": "2020-12-31T15:37:51.111337Z"
    },
    "id": "x6QyEwQSt3he",
    "outputId": "10a9b944-6169-461e-e315-28f092173ecd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "테스트 정확도: 77.17%\n"
     ]
    }
   ],
   "source": [
    "print('테스트 정확도: {:.2f}%'.format(results[1]*100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "KjPp6Koit3he"
   },
   "source": [
    "## 연습문제: \n",
    "\n",
    "### 전체 길이를 사용한 시퀀스에 단방향 SimpleRNN 적용하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T15:37:51.118653Z",
     "iopub.status.busy": "2020-12-31T15:37:51.118013Z",
     "iopub.status.idle": "2020-12-31T15:37:55.390222Z",
     "shell.execute_reply": "2020-12-31T15:37:55.389585Z"
    },
    "id": "VDMMjV1xt3he",
    "outputId": "771a4e5b-3f47-4e52-eca1-3b5a73baf2c3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "어휘 사전 크기: 87007\n",
      "Model: \"sequential_6\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embed-layer (Embedding)      (None, None, 20)          1740180   \n",
      "_________________________________________________________________\n",
      "simprnn-layer-0 (SimpleRNN)  (None, 64)                5440      \n",
      "_________________________________________________________________\n",
      "dense_7 (Dense)              (None, 64)                4160      \n",
      "_________________________________________________________________\n",
      "dense_8 (Dense)              (None, 1)                 65        \n",
      "=================================================================\n",
      "Total params: 1,749,845\n",
      "Trainable params: 1,749,845\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "batch_size = 32\n",
    "embedding_dim = 20\n",
    "max_seq_length = None\n",
    "\n",
    "train_data, valid_data, test_data, n = preprocess_datasets(\n",
    "    ds_raw_train, ds_raw_valid, ds_raw_test, \n",
    "    max_seq_length=max_seq_length, \n",
    "    batch_size=batch_size\n",
    ")\n",
    "\n",
    "\n",
    "vocab_size = n + 2\n",
    "\n",
    "rnn_model = build_rnn_model(\n",
    "    embedding_dim, vocab_size,\n",
    "    recurrent_type='SimpleRNN', \n",
    "    n_recurrent_units=64,\n",
    "    n_recurrent_layers=1,\n",
    "    bidirectional=False)\n",
    "\n",
    "rnn_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T15:37:55.402376Z",
     "iopub.status.busy": "2020-12-31T15:37:55.401627Z",
     "iopub.status.idle": "2020-12-31T15:59:01.339172Z",
     "shell.execute_reply": "2020-12-31T15:59:01.338258Z"
    },
    "id": "eR8OJjstt3he",
    "outputId": "a6cd4a48-d887-4660-c854-03b1fec26339"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n",
      "625/625 [==============================] - 428s 684ms/step - loss: 0.6970 - accuracy: 0.5043 - val_loss: 0.6963 - val_accuracy: 0.4854\n",
      "Epoch 2/10\n",
      "625/625 [==============================] - 432s 692ms/step - loss: 0.6980 - accuracy: 0.5049 - val_loss: 0.6996 - val_accuracy: 0.4914\n",
      "Epoch 3/10\n",
      "625/625 [==============================] - 425s 680ms/step - loss: 0.6956 - accuracy: 0.5088 - val_loss: 0.6994 - val_accuracy: 0.4862\n",
      "Epoch 4/10\n",
      "625/625 [==============================] - 425s 681ms/step - loss: 0.6953 - accuracy: 0.5080 - val_loss: 0.6980 - val_accuracy: 0.4874\n",
      "Epoch 5/10\n",
      "625/625 [==============================] - 427s 683ms/step - loss: 0.6952 - accuracy: 0.5089 - val_loss: 0.6949 - val_accuracy: 0.4942\n",
      "Epoch 6/10\n",
      "625/625 [==============================] - 428s 684ms/step - loss: 0.6971 - accuracy: 0.4977 - val_loss: 0.6930 - val_accuracy: 0.5076\n",
      "Epoch 7/10\n",
      "625/625 [==============================] - 427s 683ms/step - loss: 0.6943 - accuracy: 0.5083 - val_loss: 0.6939 - val_accuracy: 0.5054\n",
      "Epoch 8/10\n",
      "625/625 [==============================] - 428s 685ms/step - loss: 0.6947 - accuracy: 0.5027 - val_loss: 0.6936 - val_accuracy: 0.5054\n",
      "Epoch 9/10\n",
      "625/625 [==============================] - 428s 685ms/step - loss: 0.6942 - accuracy: 0.4989 - val_loss: 0.6936 - val_accuracy: 0.4982\n",
      "Epoch 10/10\n",
      "625/625 [==============================] - 428s 685ms/step - loss: 0.6942 - accuracy: 0.4998 - val_loss: 0.6933 - val_accuracy: 0.5026\n"
     ]
    }
   ],
   "source": [
    "rnn_model.compile(optimizer=tf.keras.optimizers.Adam(1e-3),\n",
    "                  loss=tf.keras.losses.BinaryCrossentropy(from_logits=False),\n",
    "                  metrics=['accuracy'])\n",
    "\n",
    "history = rnn_model.fit(\n",
    "    train_data, \n",
    "    validation_data=valid_data, \n",
    "    epochs=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-P9l6HEHt3he"
   },
   "source": [
    "# 부록"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bAw6wJ0jt3hf"
   },
   "source": [
    "### A -- 데이터셋을 만드는 다른 방법: tensorflow_datasets 사용하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 911,
     "referenced_widgets": [
      "f08ee16054ec46fba5064c14c438a51c",
      "9b0bdb7a023840a69e539969ab7f07d7",
      "6c5517aff64a4d839c9e2efc7645abf0",
      "72dc62f6d65946fe9607ab37c2c6cf40",
      "512d3acff09647908dc79444d40a564b",
      "80b1b8f2c15440ffa5430944617397c7",
      "e689b12faf1c440cb2bb70a9ac609c5a",
      "beeeb5c05c2349d6a9845366b60c9493",
      "4d3ceab8e96e42039c4e0c12e6ae9045",
      "6c4b20badd9448d8a312514883e3cb29",
      "0980cba2ba8542d18c1ca41d76dcbcdd",
      "4bedcef7f0b24bc195b4c5d8a1d26172",
      "1ebec932aa9549f1927e7bde613d5d17",
      "520850433e134e4d9403ad17af1766f7",
      "594afdb2b1ca4b3b8bfbd00d8592eb36",
      "acd07d65e57e4e28be7a585681d1d958",
      "a5771cbffe85487a9fe272a7f6401735",
      "168be80911d942bcbcccc1cfda377cd3",
      "15e18fddc84b422884aea8b21e942def",
      "313c6e4551b2472daaf7c68185876feb",
      "bf78a1d017534cf196e6f7748be5f47a",
      "57ccd22d51414d6181344da8cb84452e",
      "dc610cf6b261491e8f63a3b037e4b126",
      "8958f7b609a4493a9b5906e3a27fa7af",
      "ca29598dfbac4ad8802bcb2814357f43",
      "71c6503f98ee4be88f47b2b457948bf2",
      "50a258e87c92409b9bf1d21e69c11717",
      "10a121953a744904b19dddde5629c7dd",
      "213eaeb737bf400b93094b33c97371dc",
      "73d0548d70ca4f1eb26890d51c7fe80b",
      "af00a33e81dd48348f7a2423be32bdf0",
      "f9bbe11c4c6a4660961358790979e0ba",
      "de5b4cb2ae8a4abb88fc92093fc9dc1b",
      "993632b08a1348d18a344d2b49cf139b",
      "6183e4ee40be44a0b25c66c2d53aab66",
      "3bb517721c5f4896a9bb1442be9fa89b",
      "d41f4d7e440d42cabe50d4339f94bcfe",
      "c296617f7a714dee94f6b7eb66b69dc4",
      "689238aab66f40dfb342331918418365",
      "cc195f3a7ada4a29b7bad0f984aa4302",
      "aff3787ee42d4084a885ad0aeae71dbf",
      "92b4ef5132f14f13bb559c9af677acea",
      "1ae12e24c0a64ab8ada1355621eaeef4",
      "16f57aeb74b647a09747a4e044f35b4c",
      "49d42e046b49420c8f76387811e3b8b7",
      "6e6d64fb323848dc9ae3e7e4ab77f6e1",
      "34bfc01be0b84b5eadcb60127ccaae09",
      "994a3b603dba4acd8bb7d48b2ca6a8d6",
      "6ffa7aaa4a334f48ac67524c23c087b9",
      "b7aefc95a9fd4458b4b59959bd400ead",
      "518c46f206e149f58a188865381c6e2d",
      "867091c35f704e2b9c325caee127bd00",
      "682fa39dc6f64d14939aa04ff9561f47",
      "544e9f94664a491587c5b9f9dc32cc0e",
      "8d58c1972add4e83a9d40d9a59d57abe",
      "571695d8350c41c78ba6a63ce80937bb",
      "c1a6749d349a411eb6bb4d685a0db845",
      "e50d6068762542ea8875b5aa0c6d23ac",
      "dd34b53e07b840b68d229b67d170e4c0",
      "58f5e22b735949bc8ad55f894f25f99f",
      "56a85060d5434952bdd9552c9defed45",
      "edf9c4d05c2a448a8cbcefe5c2683662",
      "eab5917ccbbb409d9c473d74132073c4",
      "9f6fff8bf8374f8c820233a83f620640"
     ]
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T15:59:01.345251Z",
     "iopub.status.busy": "2020-12-31T15:59:01.344304Z",
     "iopub.status.idle": "2020-12-31T16:00:03.995238Z",
     "shell.execute_reply": "2020-12-31T16:00:03.994443Z"
    },
    "id": "Bm4eEIM7t3hf",
    "outputId": "1a530661-af02-474d-b95d-7c10c56391fa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tfds.core.DatasetInfo(\n",
      "    name='imdb_reviews',\n",
      "    version=1.0.0,\n",
      "    description='Large Movie Review Dataset.\n",
      "This is a dataset for binary sentiment classification containing substantially more data than previous benchmark datasets. We provide a set of 25,000 highly polar movie reviews for training, and 25,000 for testing. There is additional unlabeled data for use as well.',\n",
      "    homepage='http://ai.stanford.edu/~amaas/data/sentiment/',\n",
      "    features=FeaturesDict({\n",
      "        'label': ClassLabel(shape=(), dtype=tf.int64, num_classes=2),\n",
      "        'text': Text(shape=(), dtype=tf.string),\n",
      "    }),\n",
      "    total_num_examples=100000,\n",
      "    splits={\n",
      "        'test': 25000,\n",
      "        'train': 25000,\n",
      "        'unsupervised': 50000,\n",
      "    },\n",
      "    supervised_keys=('text', 'label'),\n",
      "    citation=\"\"\"@InProceedings{maas-EtAl:2011:ACL-HLT2011,\n",
      "      author    = {Maas, Andrew L.  and  Daly, Raymond E.  and  Pham, Peter T.  and  Huang, Dan  and  Ng, Andrew Y.  and  Potts, Christopher},\n",
      "      title     = {Learning Word Vectors for Sentiment Analysis},\n",
      "      booktitle = {Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies},\n",
      "      month     = {June},\n",
      "      year      = {2011},\n",
      "      address   = {Portland, Oregon, USA},\n",
      "      publisher = {Association for Computational Linguistics},\n",
      "      pages     = {142--150},\n",
      "      url       = {http://www.aclweb.org/anthology/P11-1015}\n",
      "    }\"\"\",\n",
      "    redistribution_info=,\n",
      ")\n",
      "\n",
      "\u001b[1mDownloading and preparing dataset imdb_reviews/plain_text/1.0.0 (download: 80.23 MiB, generated: Unknown size, total: 80.23 MiB) to /root/tensorflow_datasets/imdb_reviews/plain_text/1.0.0...\u001b[0m\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "f08ee16054ec46fba5064c14c438a51c",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Dl Completed...', max=1.0, style=Progre…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "4d3ceab8e96e42039c4e0c12e6ae9045",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', description='Dl Size...', max=1.0, style=ProgressSty…"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "a5771cbffe85487a9fe272a7f6401735",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Shuffling and writing examples to /root/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteDHK5EH/imdb_reviews-train.tfrecord\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "ca29598dfbac4ad8802bcb2814357f43",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=25000.0), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "de5b4cb2ae8a4abb88fc92093fc9dc1b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Shuffling and writing examples to /root/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteDHK5EH/imdb_reviews-test.tfrecord\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "aff3787ee42d4084a885ad0aeae71dbf",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=25000.0), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6ffa7aaa4a334f48ac67524c23c087b9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=1.0, bar_style='info', max=1.0), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\r",
      "Shuffling and writing examples to /root/tensorflow_datasets/imdb_reviews/plain_text/1.0.0.incompleteDHK5EH/imdb_reviews-unsupervised.tfrecord\n"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "c1a6749d349a411eb6bb4d685a0db845",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "HBox(children=(FloatProgress(value=0.0, max=50000.0), HTML(value='')))"
      ]
     },
     "metadata": {
      "tags": []
     },
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING:absl:Dataset is using deprecated text encoder API which will be removed soon. Please use the plain_text version of the dataset and migrate to `tensorflow_text`.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[1mDataset imdb_reviews downloaded and prepared to /root/tensorflow_datasets/imdb_reviews/plain_text/1.0.0. Subsequent calls will reuse this data.\u001b[0m\n",
      "\r"
     ]
    },
    {
     "data": {
      "text/plain": [
       "dict_keys(['test', 'train', 'unsupervised'])"
      ]
     },
     "execution_count": 56,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "imdb_bldr = tfds.builder('imdb_reviews')\n",
    "print(imdb_bldr.info)\n",
    "\n",
    "imdb_bldr.download_and_prepare()\n",
    "\n",
    "datasets = imdb_bldr.as_dataset(shuffle_files=False)\n",
    "\n",
    "datasets.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2020-12-31T16:00:04.000279Z",
     "iopub.status.busy": "2020-12-31T16:00:03.999377Z",
     "iopub.status.idle": "2020-12-31T16:00:04.002868Z",
     "shell.execute_reply": "2020-12-31T16:00:04.001894Z"
    },
    "id": "YHn1LXgkt3hf"
   },
   "outputs": [],
   "source": [
    "imdb_train = datasets['train']\n",
    "imdb_train = datasets['test']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2THlQCVWt3hf"
   },
   "source": [
    "### B -- Tokenizer와 Encoder\n",
    "\n",
    " * `tfds.deprecated.text.Tokenizer`: https://www.tensorflow.org/datasets/api_docs/python/tfds/deprecated/text/Tokenizer\n",
    " * `tfds.deprecated.text.TokenTextEncoder`: https://www.tensorflow.org/datasets/api_docs/python/tfds/deprecated/text/TokenTextEncoder"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T16:00:04.012615Z",
     "iopub.status.busy": "2020-12-31T16:00:04.011509Z",
     "iopub.status.idle": "2020-12-31T16:00:04.015817Z",
     "shell.execute_reply": "2020-12-31T16:00:04.015226Z"
    },
    "id": "tZrY4Fcnt3hf",
    "outputId": "5ebb836f-7af3-47ff-feb5-e1779e2ad26a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<TokenTextEncoder vocab_size=6>\n",
      "[2, 4, 3, 1]\n",
      "[2, 4, 3, 1, 5, 5, 5, 5, 5, 5]\n"
     ]
    }
   ],
   "source": [
    "vocab_set = {'a', 'b', 'c', 'd'}\n",
    "encoder = tfds.deprecated.text.TokenTextEncoder(vocab_set)\n",
    "print(encoder)\n",
    "\n",
    "print(encoder.encode(b'a b c d, , : .'))\n",
    "\n",
    "print(encoder.encode(b'a b c d e f g h i z'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NY6z1aTit3hg"
   },
   "source": [
    "### C -- 케라스로 텍스트 전처리하기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T16:00:04.023781Z",
     "iopub.status.busy": "2020-12-31T16:00:04.022913Z",
     "iopub.status.idle": "2020-12-31T16:00:04.028002Z",
     "shell.execute_reply": "2020-12-31T16:00:04.027228Z"
    },
    "id": "LRO2xhDKt3hg",
    "outputId": "70d79e58-60f4-4157-c11a-dc07dacd0b3a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[1, 2, 3, 4], [5, 6, 7, 8]]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "array([[0, 0, 0, 0, 0, 0, 1, 2, 3, 4],\n",
       "       [0, 0, 0, 0, 0, 0, 5, 6, 7, 8]], dtype=int32)"
      ]
     },
     "execution_count": 59,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TOP_K = 200\n",
    "MAX_LEN = 10\n",
    "\n",
    "tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=TOP_K)\n",
    "\n",
    "tokenizer.fit_on_texts(['this is an example', 'je suis en forme '])\n",
    "sequences = tokenizer.texts_to_sequences(['this is an example', 'je suis en forme '])\n",
    "print(sequences)\n",
    "\n",
    "tf.keras.preprocessing.sequence.pad_sequences(sequences, maxlen=MAX_LEN)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T16:00:04.036271Z",
     "iopub.status.busy": "2020-12-31T16:00:04.035409Z",
     "iopub.status.idle": "2020-12-31T16:00:15.234946Z",
     "shell.execute_reply": "2020-12-31T16:00:15.234078Z"
    },
    "id": "vePZ3UTot3hg",
    "outputId": "3bba9a52-57b4-49d4-9888-3074a986e10c",
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000\n",
      "(25000, 500)\n"
     ]
    }
   ],
   "source": [
    "TOP_K = 20000\n",
    "MAX_LEN = 500\n",
    "\n",
    "tokenizer = tf.keras.preprocessing.text.Tokenizer(num_words=TOP_K)\n",
    "\n",
    "tokenizer.fit_on_texts(\n",
    "    [example['text'].numpy().decode('utf-8') \n",
    "     for example in imdb_train])\n",
    "\n",
    "x_train = tokenizer.texts_to_sequences(\n",
    "    [example['text'].numpy().decode('utf-8')\n",
    "     for example in imdb_train])\n",
    "\n",
    "print(len(x_train))\n",
    "\n",
    "\n",
    "x_train_padded = tf.keras.preprocessing.sequence.pad_sequences(\n",
    "    x_train, maxlen=MAX_LEN)\n",
    "\n",
    "print(x_train_padded.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9FYRIc-St3hg"
   },
   "source": [
    "### D -- 임베딩"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2020-12-31T16:00:15.250708Z",
     "iopub.status.busy": "2020-12-31T16:00:15.249808Z",
     "iopub.status.idle": "2020-12-31T16:00:15.258749Z",
     "shell.execute_reply": "2020-12-31T16:00:15.257884Z"
    },
    "id": "fEuT0TEut3hh",
    "outputId": "873c850c-a3b2-46eb-cf9f-e2e33c750414"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.0208060984 0.0142502077 0.0475785471 -0.00649005175]\n",
      " [-0.00420691818 -0.0375086069 -0.00477621704 0.00311584398]\n",
      " [0.028728161 -0.0440448038 -0.0428906195 -0.019158531]\n",
      " [-0.0248817336 0.0408470519 -0.00285203382 -0.0257614851]\n",
      " [0.0443614833 0.00331580639 0.043055404 -0.011118304]\n",
      " [-0.0281324144 0.00720113516 0.0192188732 -0.0186921246]]\n",
      "TensorShape([6, 4])\n",
      "[[-0.0208060984 0.0142502077 0.0475785471 -0.00649005175]]\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.layers import Embedding\n",
    "\n",
    "\n",
    "tf.random.set_seed(1)\n",
    "embed = Embedding(input_dim=100, output_dim=4)\n",
    "\n",
    "inp_arr = np.array([1, 98, 5, 6, 67, 45])\n",
    "tf.print(embed(inp_arr))\n",
    "tf.print(embed(inp_arr).shape)\n",
    "\n",
    "tf.print(embed(np.array([1])))"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "ch16_part1.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "0980cba2ba8542d18c1ca41d76dcbcdd": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Dl Size...: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_520850433e134e4d9403ad17af1766f7",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_1ebec932aa9549f1927e7bde613d5d17",
      "value": 1
     }
    },
    "10a121953a744904b19dddde5629c7dd": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_f9bbe11c4c6a4660961358790979e0ba",
      "placeholder": "​",
      "style": "IPY_MODEL_af00a33e81dd48348f7a2423be32bdf0",
      "value": " 15661/25000 [00:00&lt;00:00, 156608.66 examples/s]"
     }
    },
    "15e18fddc84b422884aea8b21e942def": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "info",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_57ccd22d51414d6181344da8cb84452e",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_bf78a1d017534cf196e6f7748be5f47a",
      "value": 1
     }
    },
    "168be80911d942bcbcccc1cfda377cd3": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "16f57aeb74b647a09747a4e044f35b4c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_994a3b603dba4acd8bb7d48b2ca6a8d6",
      "placeholder": "​",
      "style": "IPY_MODEL_34bfc01be0b84b5eadcb60127ccaae09",
      "value": " 15873/25000 [00:00&lt;00:00, 158723.34 examples/s]"
     }
    },
    "1ae12e24c0a64ab8ada1355621eaeef4": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "danger",
      "description": " 63%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_6e6d64fb323848dc9ae3e7e4ab77f6e1",
      "max": 25000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_49d42e046b49420c8f76387811e3b8b7",
      "value": 15873
     }
    },
    "1ebec932aa9549f1927e7bde613d5d17": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "213eaeb737bf400b93094b33c97371dc": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "313c6e4551b2472daaf7c68185876feb": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_8958f7b609a4493a9b5906e3a27fa7af",
      "placeholder": "​",
      "style": "IPY_MODEL_dc610cf6b261491e8f63a3b037e4b126",
      "value": " 25000/0 [00:11&lt;00:00, 3403.46 examples/s]"
     }
    },
    "34bfc01be0b84b5eadcb60127ccaae09": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "3bb517721c5f4896a9bb1442be9fa89b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_cc195f3a7ada4a29b7bad0f984aa4302",
      "placeholder": "​",
      "style": "IPY_MODEL_689238aab66f40dfb342331918418365",
      "value": " 25000/0 [00:11&lt;00:00, 3755.03 examples/s]"
     }
    },
    "49d42e046b49420c8f76387811e3b8b7": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "4bedcef7f0b24bc195b4c5d8a1d26172": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_acd07d65e57e4e28be7a585681d1d958",
      "placeholder": "​",
      "style": "IPY_MODEL_594afdb2b1ca4b3b8bfbd00d8592eb36",
      "value": " 80/80 [00:01&lt;00:00, 46.77 MiB/s]"
     }
    },
    "4d3ceab8e96e42039c4e0c12e6ae9045": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_0980cba2ba8542d18c1ca41d76dcbcdd",
       "IPY_MODEL_4bedcef7f0b24bc195b4c5d8a1d26172"
      ],
      "layout": "IPY_MODEL_6c4b20badd9448d8a312514883e3cb29"
     }
    },
    "50a258e87c92409b9bf1d21e69c11717": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "danger",
      "description": " 63%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_73d0548d70ca4f1eb26890d51c7fe80b",
      "max": 25000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_213eaeb737bf400b93094b33c97371dc",
      "value": 15661
     }
    },
    "512d3acff09647908dc79444d40a564b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "518c46f206e149f58a188865381c6e2d": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "info",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_544e9f94664a491587c5b9f9dc32cc0e",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_682fa39dc6f64d14939aa04ff9561f47",
      "value": 1
     }
    },
    "520850433e134e4d9403ad17af1766f7": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "544e9f94664a491587c5b9f9dc32cc0e": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "56a85060d5434952bdd9552c9defed45": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "571695d8350c41c78ba6a63ce80937bb": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "57ccd22d51414d6181344da8cb84452e": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "58f5e22b735949bc8ad55f894f25f99f": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_9f6fff8bf8374f8c820233a83f620640",
      "placeholder": "​",
      "style": "IPY_MODEL_eab5917ccbbb409d9c473d74132073c4",
      "value": " 47382/50000 [00:00&lt;00:00, 92909.48 examples/s]"
     }
    },
    "594afdb2b1ca4b3b8bfbd00d8592eb36": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6183e4ee40be44a0b25c66c2d53aab66": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "info",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_c296617f7a714dee94f6b7eb66b69dc4",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_d41f4d7e440d42cabe50d4339f94bcfe",
      "value": 1
     }
    },
    "682fa39dc6f64d14939aa04ff9561f47": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "689238aab66f40dfb342331918418365": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "6c4b20badd9448d8a312514883e3cb29": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6c5517aff64a4d839c9e2efc7645abf0": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "Dl Completed...: 100%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_80b1b8f2c15440ffa5430944617397c7",
      "max": 1,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_512d3acff09647908dc79444d40a564b",
      "value": 1
     }
    },
    "6e6d64fb323848dc9ae3e7e4ab77f6e1": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "6ffa7aaa4a334f48ac67524c23c087b9": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_518c46f206e149f58a188865381c6e2d",
       "IPY_MODEL_867091c35f704e2b9c325caee127bd00"
      ],
      "layout": "IPY_MODEL_b7aefc95a9fd4458b4b59959bd400ead"
     }
    },
    "71c6503f98ee4be88f47b2b457948bf2": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "72dc62f6d65946fe9607ab37c2c6cf40": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_beeeb5c05c2349d6a9845366b60c9493",
      "placeholder": "​",
      "style": "IPY_MODEL_e689b12faf1c440cb2bb70a9ac609c5a",
      "value": " 1/1 [00:01&lt;00:00,  1.74s/ url]"
     }
    },
    "73d0548d70ca4f1eb26890d51c7fe80b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "80b1b8f2c15440ffa5430944617397c7": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "867091c35f704e2b9c325caee127bd00": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_571695d8350c41c78ba6a63ce80937bb",
      "placeholder": "​",
      "style": "IPY_MODEL_8d58c1972add4e83a9d40d9a59d57abe",
      "value": " 50000/0 [00:16&lt;00:00, 3345.44 examples/s]"
     }
    },
    "8958f7b609a4493a9b5906e3a27fa7af": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "8d58c1972add4e83a9d40d9a59d57abe": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "92b4ef5132f14f13bb559c9af677acea": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "993632b08a1348d18a344d2b49cf139b": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "994a3b603dba4acd8bb7d48b2ca6a8d6": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9b0bdb7a023840a69e539969ab7f07d7": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "9f6fff8bf8374f8c820233a83f620640": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "a5771cbffe85487a9fe272a7f6401735": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_15e18fddc84b422884aea8b21e942def",
       "IPY_MODEL_313c6e4551b2472daaf7c68185876feb"
      ],
      "layout": "IPY_MODEL_168be80911d942bcbcccc1cfda377cd3"
     }
    },
    "acd07d65e57e4e28be7a585681d1d958": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "af00a33e81dd48348f7a2423be32bdf0": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "aff3787ee42d4084a885ad0aeae71dbf": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_1ae12e24c0a64ab8ada1355621eaeef4",
       "IPY_MODEL_16f57aeb74b647a09747a4e044f35b4c"
      ],
      "layout": "IPY_MODEL_92b4ef5132f14f13bb559c9af677acea"
     }
    },
    "b7aefc95a9fd4458b4b59959bd400ead": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "beeeb5c05c2349d6a9845366b60c9493": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "bf78a1d017534cf196e6f7748be5f47a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "c1a6749d349a411eb6bb4d685a0db845": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_dd34b53e07b840b68d229b67d170e4c0",
       "IPY_MODEL_58f5e22b735949bc8ad55f894f25f99f"
      ],
      "layout": "IPY_MODEL_e50d6068762542ea8875b5aa0c6d23ac"
     }
    },
    "c296617f7a714dee94f6b7eb66b69dc4": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "ca29598dfbac4ad8802bcb2814357f43": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_50a258e87c92409b9bf1d21e69c11717",
       "IPY_MODEL_10a121953a744904b19dddde5629c7dd"
      ],
      "layout": "IPY_MODEL_71c6503f98ee4be88f47b2b457948bf2"
     }
    },
    "cc195f3a7ada4a29b7bad0f984aa4302": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "d41f4d7e440d42cabe50d4339f94bcfe": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": "initial"
     }
    },
    "dc610cf6b261491e8f63a3b037e4b126": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "dd34b53e07b840b68d229b67d170e4c0": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "danger",
      "description": " 95%",
      "description_tooltip": null,
      "layout": "IPY_MODEL_edf9c4d05c2a448a8cbcefe5c2683662",
      "max": 50000,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_56a85060d5434952bdd9552c9defed45",
      "value": 47382
     }
    },
    "de5b4cb2ae8a4abb88fc92093fc9dc1b": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_6183e4ee40be44a0b25c66c2d53aab66",
       "IPY_MODEL_3bb517721c5f4896a9bb1442be9fa89b"
      ],
      "layout": "IPY_MODEL_993632b08a1348d18a344d2b49cf139b"
     }
    },
    "e50d6068762542ea8875b5aa0c6d23ac": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "e689b12faf1c440cb2bb70a9ac609c5a": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "eab5917ccbbb409d9c473d74132073c4": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "edf9c4d05c2a448a8cbcefe5c2683662": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "f08ee16054ec46fba5064c14c438a51c": {
     "model_module": "@jupyter-widgets/controls",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_6c5517aff64a4d839c9e2efc7645abf0",
       "IPY_MODEL_72dc62f6d65946fe9607ab37c2c6cf40"
      ],
      "layout": "IPY_MODEL_9b0bdb7a023840a69e539969ab7f07d7"
     }
    },
    "f9bbe11c4c6a4660961358790979e0ba": {
     "model_module": "@jupyter-widgets/base",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}