ch16/ch16_part2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mrDzC9FUyWJk"
   },
   "source": [
    "# 머신 러닝 교과서 3판"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "90ZMzS7wyWJm"
   },
   "source": [
    "# 16장 - 순환 신경망으로 순차 데이터 모델링 (2/2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i-YXfR8UyWJm"
   },
   "source": [
    "**아래 링크를 통해 이 노트북을 주피터 노트북 뷰어(nbviewer.jupyter.org)로 보거나 구글 코랩(colab.research.google.com)에서 실행할 수 있습니다.**\n",
    "\n",
    "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://nbviewer.jupyter.org/github/rickiepark/python-machine-learning-book-3rd-edition/blob/master/ch16/ch16_part2.ipynb\"><img src=\"https://jupyter.org/assets/main-logo.svg\" width=\"28\" />주피터 노트북 뷰어로 보기</a>\n",
    "  </td>\n",
    "  <td>\n",
    "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/rickiepark/python-machine-learning-book-3rd-edition/blob/master/ch16/ch16_part2.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />구글 코랩(Colab)에서 실행하기</a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5_u_b0r4yWJn"
   },
   "source": [
    "### 목차"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G-xguWppyWJn"
   },
   "source": [
    "- 텐서플로로 시퀀스 모델링을 위한 RNN 구현하기\n",
    "    - 두 번째 프로젝트: 텐서플로로 글자 단위 언어 모델 구현\n",
    "        - 데이터셋 전처리\n",
    "        - 문자 수준의 RNN 모델 만들기\n",
    "        - 평가 단계 - 새로운 텍스트 생성\n",
    "- 트랜스포머 모델을 사용한 언어 이해\n",
    "    - 셀프 어텐션 메카니즘 이해하기\n",
    "        - 셀프 어텐션 기본 구조\n",
    "        - 쿼리, 키, 값 가중치를 가진 셀프 어텐션 메카니즘\n",
    "    - 멀티-헤드 어텐션과 트랜스포머 블록\n",
    "- 요약"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "execution": {
     "iopub.execute_input": "2021-01-01T08:20:46.163997Z",
     "iopub.status.busy": "2021-01-01T08:20:46.163019Z",
     "iopub.status.idle": "2021-01-01T08:20:46.165178Z",
     "shell.execute_reply": "2021-01-01T08:20:46.165803Z"
    },
    "id": "RLnk-L-ZyWJn"
   },
   "outputs": [],
   "source": [
    "from IPython.display import Image"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "k8wSLT9byWJn"
   },
   "source": [
    "## 두 번째 프로젝트: 텐서플로로 글자 단위 언어 모델 구현"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 239
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:20:46.170192Z",
     "iopub.status.busy": "2021-01-01T08:20:46.169457Z",
     "iopub.status.idle": "2021-01-01T08:20:46.205811Z",
     "shell.execute_reply": "2021-01-01T08:20:46.206653Z"
    },
    "id": "-tysyJWSyWJn",
    "outputId": "40ad7e89-afe4-496b-98a8-34696bbc53c9"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdVE\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 2,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdVE', width=700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xrqTH2wvyWJo"
   },
   "source": [
    "### 데이터셋 전처리"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:20:46.213066Z",
     "iopub.status.busy": "2021-01-01T08:20:46.212126Z",
     "iopub.status.idle": "2021-01-01T08:20:50.103519Z",
     "shell.execute_reply": "2021-01-01T08:20:50.102717Z"
    },
    "id": "GtxzLN-RyWJo",
    "outputId": "d1c38c71-51b2-47cc-9927-96bd593c44d4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 1144k  100 1144k    0     0  4594k      0 --:--:-- --:--:-- --:--:-- 4594k\n"
     ]
    }
   ],
   "source": [
    "! curl -O http://www.gutenberg.org/files/1268/1268-0.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:20:50.111825Z",
     "iopub.status.busy": "2021-01-01T08:20:50.110901Z",
     "iopub.status.idle": "2021-01-01T08:20:50.138395Z",
     "shell.execute_reply": "2021-01-01T08:20:50.137468Z"
    },
    "id": "mRwzImO7yWJo",
    "outputId": "69562f67-c609-4f6b-88fb-7bf17b2b6dea"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "567 1112917\n",
      "전체 길이: 1112350\n",
      "고유한 문자: 80\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "\n",
    "## 텍스트 읽고 전처리하기\n",
    "with open('1268-0.txt', 'r', encoding='UTF8') as fp:\n",
    "    text=fp.read()\n",
    "    \n",
    "start_indx = text.find('THE MYSTERIOUS ISLAND')\n",
    "end_indx = text.find('End of the Project Gutenberg')\n",
    "print(start_indx, end_indx)\n",
    "\n",
    "text = text[start_indx:end_indx]\n",
    "char_set = set(text)\n",
    "print('전체 길이:', len(text))\n",
    "print('고유한 문자:', len(char_set))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 288
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:20:50.143620Z",
     "iopub.status.busy": "2021-01-01T08:20:50.142625Z",
     "iopub.status.idle": "2021-01-01T08:20:50.148075Z",
     "shell.execute_reply": "2021-01-01T08:20:50.148851Z"
    },
    "id": "HjzozZLuyWJp",
    "outputId": "355c0206-31bb-440c-defa-e0644749e0f1"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdVz\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 5,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdVz', width=700)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:20:50.198178Z",
     "iopub.status.busy": "2021-01-01T08:20:50.167394Z",
     "iopub.status.idle": "2021-01-01T08:20:50.298755Z",
     "shell.execute_reply": "2021-01-01T08:20:50.297847Z"
    },
    "id": "X3zvJimcyWJp",
    "outputId": "4d545558-291a-4bf2-a931-7511e0045de0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "인코딩된 텍스트 크기:  (1112350,)\n",
      "THE MYSTERIOUS       == 인코딩 ==>  [44 32 29  1 37 48 43 44 29 42 33 39 45 43  1]\n",
      "[33 43 36 25 38 28]  == 디코딩 ==>  ISLAND\n"
     ]
    }
   ],
   "source": [
    "chars_sorted = sorted(char_set)\n",
    "char2int = {ch:i for i,ch in enumerate(chars_sorted)}\n",
    "char_array = np.array(chars_sorted)\n",
    "\n",
    "text_encoded = np.array(\n",
    "    [char2int[ch] for ch in text],\n",
    "    dtype=np.int32)\n",
    "\n",
    "print('인코딩된 텍스트 크기: ', text_encoded.shape)\n",
    "\n",
    "print(text[:15], '     == 인코딩 ==> ', text_encoded[:15])\n",
    "print(text_encoded[15:21], ' == 디코딩 ==> ', ''.join(char_array[text_encoded[15:21]]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 418
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:20:50.303833Z",
     "iopub.status.busy": "2021-01-01T08:20:50.302867Z",
     "iopub.status.idle": "2021-01-01T08:20:50.309281Z",
     "shell.execute_reply": "2021-01-01T08:20:50.310125Z"
    },
    "id": "c9JdWEpwyWJp",
    "outputId": "fd69d106-1e7f-4d72-f9a0-2ffb373a268a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdVV\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 7,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdVV', width=700)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:20:50.316611Z",
     "iopub.status.busy": "2021-01-01T08:20:50.315609Z",
     "iopub.status.idle": "2021-01-01T08:20:52.091027Z",
     "shell.execute_reply": "2021-01-01T08:20:52.091768Z"
    },
    "id": "_E5CojZ5yWJp",
    "outputId": "ea69c7f9-8561-4aab-fdee-a765142ee20e"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "44 -> T\n",
      "32 -> H\n",
      "29 -> E\n",
      "1 ->  \n",
      "37 -> M\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "\n",
    "\n",
    "ds_text_encoded = tf.data.Dataset.from_tensor_slices(text_encoded)\n",
    "\n",
    "for ex in ds_text_encoded.take(5):\n",
    "    print('{} -> {}'.format(ex.numpy(), char_array[ex.numpy()]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:20:52.100011Z",
     "iopub.status.busy": "2021-01-01T08:20:52.099319Z",
     "iopub.status.idle": "2021-01-01T08:20:52.107438Z",
     "shell.execute_reply": "2021-01-01T08:20:52.108132Z"
    },
    "id": "Ilw6v7iKyWJq",
    "outputId": "f16a330e-7817-4aac-fcd8-1fd00c60794a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[44 32 29  1 37 48 43 44 29 42 33 39 45 43  1 33 43 36 25 38 28  1  6  6\n",
      "  6  0  0  0  0  0 40 67 64 53 70 52 54 53  1 51]  ->  74\n",
      "'THE MYSTERIOUS ISLAND ***\\n\\n\\n\\n\\nProduced b'  ->  'y'\n"
     ]
    }
   ],
   "source": [
    "seq_length = 40\n",
    "chunk_size = seq_length + 1\n",
    "\n",
    "ds_chunks = ds_text_encoded.batch(chunk_size, drop_remainder=True)\n",
    "\n",
    "## inspection:\n",
    "for seq in ds_chunks.take(1):\n",
    "    input_seq = seq[:seq_length].numpy()\n",
    "    target = seq[seq_length].numpy()\n",
    "    print(input_seq, ' -> ', target)\n",
    "    print(repr(''.join(char_array[input_seq])), \n",
    "          ' -> ', repr(''.join(char_array[target])))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 514
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:20:52.113143Z",
     "iopub.status.busy": "2021-01-01T08:20:52.112271Z",
     "iopub.status.idle": "2021-01-01T08:20:52.120737Z",
     "shell.execute_reply": "2021-01-01T08:20:52.121466Z"
    },
    "id": "9Ot6_7ptyWJq",
    "outputId": "246e2d3f-6956-4511-937e-b07f2185da4a"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdVr\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 10,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdVr', width=700)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:20:52.131350Z",
     "iopub.status.busy": "2021-01-01T08:20:52.130702Z",
     "iopub.status.idle": "2021-01-01T08:20:52.199641Z",
     "shell.execute_reply": "2021-01-01T08:20:52.198873Z"
    },
    "id": "oDIGchMbyWJq",
    "outputId": "85d71662-1cae-415c-d045-307352477855"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "입력 (x): 'THE MYSTERIOUS ISLAND ***\\n\\n\\n\\n\\nProduced b'\n",
      "타깃 (y): 'HE MYSTERIOUS ISLAND ***\\n\\n\\n\\n\\nProduced by'\n",
      "\n",
      "입력 (x): ' Anthony Matonak, and Trevor Carlson\\n\\n\\n\\n'\n",
      "타깃 (y): 'Anthony Matonak, and Trevor Carlson\\n\\n\\n\\n\\n'\n",
      "\n"
     ]
    }
   ],
   "source": [
    "## x & y를 나누기 위한 함수를 정의합니다\n",
    "def split_input_target(chunk):\n",
    "    input_seq = chunk[:-1]\n",
    "    target_seq = chunk[1:]\n",
    "    return input_seq, target_seq\n",
    "\n",
    "ds_sequences = ds_chunks.map(split_input_target)\n",
    "\n",
    "## 확인:\n",
    "for example in ds_sequences.take(2):\n",
    "    print('입력 (x):', repr(''.join(char_array[example[0].numpy()])))\n",
    "    print('타깃 (y):', repr(''.join(char_array[example[1].numpy()])))\n",
    "    print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:20:52.206427Z",
     "iopub.status.busy": "2021-01-01T08:20:52.205493Z",
     "iopub.status.idle": "2021-01-01T08:20:52.214632Z",
     "shell.execute_reply": "2021-01-01T08:20:52.214055Z"
    },
    "id": "EwXM1x7MyWJr",
    "outputId": "60915d1d-f923-4d24-b7de-1dab9a790047"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<BatchDataset shapes: ((None, 40), (None, 40)), types: (tf.int32, tf.int32)>"
      ]
     },
     "execution_count": 12,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 배치 크기\n",
    "BATCH_SIZE = 64\n",
    "BUFFER_SIZE = 10000\n",
    "\n",
    "tf.random.set_seed(1)\n",
    "ds = ds_sequences.shuffle(BUFFER_SIZE).batch(BATCH_SIZE)# drop_remainder=True)\n",
    "\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ALd_KVXbyWJr"
   },
   "source": [
    "### 문자 수준의 RNN 모델 만들기"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:20:52.222512Z",
     "iopub.status.busy": "2021-01-01T08:20:52.221844Z",
     "iopub.status.idle": "2021-01-01T08:20:52.507739Z",
     "shell.execute_reply": "2021-01-01T08:20:52.508469Z"
    },
    "id": "2DQqLCqAyWJr",
    "outputId": "dc6d7239-47a9-45fe-dc53-fc7ece709f36"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (None, None, 256)         20480     \n",
      "_________________________________________________________________\n",
      "lstm (LSTM)                  (None, None, 512)         1574912   \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, None, 80)          41040     \n",
      "=================================================================\n",
      "Total params: 1,636,432\n",
      "Trainable params: 1,636,432\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "def build_model(vocab_size, embedding_dim, rnn_units):\n",
    "    model = tf.keras.Sequential([\n",
    "        tf.keras.layers.Embedding(vocab_size, embedding_dim),\n",
    "        tf.keras.layers.LSTM(\n",
    "            rnn_units, return_sequences=True),\n",
    "        tf.keras.layers.Dense(vocab_size)\n",
    "    ])\n",
    "    return model\n",
    "\n",
    "\n",
    "charset_size = len(char_array)\n",
    "embedding_dim = 256\n",
    "rnn_units = 512\n",
    "\n",
    "tf.random.set_seed(1)\n",
    "\n",
    "model = build_model(\n",
    "    vocab_size = charset_size,\n",
    "    embedding_dim=embedding_dim,\n",
    "    rnn_units=rnn_units)\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:20:52.523264Z",
     "iopub.status.busy": "2021-01-01T08:20:52.522357Z",
     "iopub.status.idle": "2021-01-01T08:52:55.869819Z",
     "shell.execute_reply": "2021-01-01T08:52:55.870445Z"
    },
    "id": "6oDKVX0JyWJr",
    "outputId": "a07c56cf-7a11-4cd0-cc87-487860b2a555"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/20\n",
      "424/424 [==============================] - 13s 14ms/step - loss: 2.7300\n",
      "Epoch 2/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.8338\n",
      "Epoch 3/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.5942\n",
      "Epoch 4/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.4573\n",
      "Epoch 5/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.3741\n",
      "Epoch 6/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.3196\n",
      "Epoch 7/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.2760\n",
      "Epoch 8/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.2429\n",
      "Epoch 9/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.2142\n",
      "Epoch 10/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.1912\n",
      "Epoch 11/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.1675\n",
      "Epoch 12/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.1531\n",
      "Epoch 13/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.1370\n",
      "Epoch 14/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.1200\n",
      "Epoch 15/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.1057\n",
      "Epoch 16/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.0913\n",
      "Epoch 17/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.0771\n",
      "Epoch 18/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.0633\n",
      "Epoch 19/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.0496\n",
      "Epoch 20/20\n",
      "424/424 [==============================] - 6s 14ms/step - loss: 1.0370\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7fd52bdfc940>"
      ]
     },
     "execution_count": 14,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.compile(\n",
    "    optimizer='adam', \n",
    "    loss=tf.keras.losses.SparseCategoricalCrossentropy(\n",
    "        from_logits=True\n",
    "    ))\n",
    "\n",
    "model.fit(ds, epochs=20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HPxMUjzzyWJr"
   },
   "source": [
    "### 평가 단계 - 새로운 텍스트 생성"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:52:55.887487Z",
     "iopub.status.busy": "2021-01-01T08:52:55.886806Z",
     "iopub.status.idle": "2021-01-01T08:52:55.892794Z",
     "shell.execute_reply": "2021-01-01T08:52:55.892440Z"
    },
    "id": "qJ8Wf-ofyWJs",
    "outputId": "e0b112fa-be26-4887-8f4c-b67e126c2345"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "확률: [0.33333334 0.33333334 0.33333334]\n",
      "array([[1, 2, 0, 1, 0, 1, 1, 2, 1, 1]])\n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(1)\n",
    "\n",
    "logits = [[1.0, 1.0, 1.0]]\n",
    "print('확률:', tf.math.softmax(logits).numpy()[0])\n",
    "\n",
    "samples = tf.random.categorical(\n",
    "    logits=logits, num_samples=10)\n",
    "tf.print(samples.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:52:55.899437Z",
     "iopub.status.busy": "2021-01-01T08:52:55.898456Z",
     "iopub.status.idle": "2021-01-01T08:52:55.903367Z",
     "shell.execute_reply": "2021-01-01T08:52:55.904194Z"
    },
    "id": "PZd-fbXByWJs",
    "outputId": "bae14124-d351-4d81-ce71-3629b8f073a5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "확률: [0.10650698 0.10650698 0.78698605]\n",
      "array([[2, 2, 0, 2, 2, 2, 2, 2, 1, 2]])\n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(1)\n",
    "\n",
    "logits = [[1.0, 1.0, 3.0]]\n",
    "print('확률:', tf.math.softmax(logits).numpy()[0])\n",
    "\n",
    "samples = tf.random.categorical(\n",
    "    logits=logits, num_samples=10)\n",
    "tf.print(samples.numpy())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:52:55.914798Z",
     "iopub.status.busy": "2021-01-01T08:52:55.914137Z",
     "iopub.status.idle": "2021-01-01T08:53:06.162148Z",
     "shell.execute_reply": "2021-01-01T08:53:06.162896Z"
    },
    "id": "R_b2-tLDyWJs",
    "outputId": "15cec843-e9bf-4dda-daa4-f9930bab4a52"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The island was eggs in his most ary where a portion of this vessel nor\n",
      "me, true to admire\n",
      "ascending beyond ouced. They would have been able to work.\n",
      "Try which each event which devotion them to be only forward and to speed again\n",
      "by Pencroft.\n",
      "\n",
      "“It is delia. In the perimet was pass that, my friends, without deniun excetive years. listened stone\n",
      "of seeing some enight where\n",
      "were a fare of the pirates have been accomplised by Granite House and extremity of center;\n",
      "“but containing the corral.\n",
      "\n",
      "The sailor\n",
      "to turn\n"
     ]
    }
   ],
   "source": [
    "def sample(model, starting_str, \n",
    "           len_generated_text=500, \n",
    "           max_input_length=40,\n",
    "           scale_factor=1.0):\n",
    "    encoded_input = [char2int[s] for s in starting_str]\n",
    "    encoded_input = tf.reshape(encoded_input, (1, -1))\n",
    "\n",
    "    generated_str = starting_str\n",
    "\n",
    "    model.reset_states()\n",
    "    for i in range(len_generated_text):\n",
    "        logits = model(encoded_input)\n",
    "        logits = tf.squeeze(logits, 0)\n",
    "\n",
    "        scaled_logits = logits * scale_factor\n",
    "        new_char_indx = tf.random.categorical(\n",
    "            scaled_logits, num_samples=1)\n",
    "        \n",
    "        new_char_indx = tf.squeeze(new_char_indx)[-1].numpy()    \n",
    "\n",
    "        generated_str += str(char_array[new_char_indx])\n",
    "        \n",
    "        new_char_indx = tf.expand_dims([new_char_indx], 0)\n",
    "        encoded_input = tf.concat(\n",
    "            [encoded_input, new_char_indx],\n",
    "            axis=1)\n",
    "        encoded_input = encoded_input[:, -max_input_length:]\n",
    "\n",
    "    return generated_str\n",
    "\n",
    "tf.random.set_seed(1)\n",
    "print(sample(model, starting_str='The island'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Stm4dLH2yWJs"
   },
   "source": [
    "* **예측 가능성 대 무작위성**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:53:06.171621Z",
     "iopub.status.busy": "2021-01-01T08:53:06.170721Z",
     "iopub.status.idle": "2021-01-01T08:53:06.175211Z",
     "shell.execute_reply": "2021-01-01T08:53:06.175546Z"
    },
    "id": "-UKHgrqTyWJs",
    "outputId": "921af54f-53e0-4af1-9113-1b652c029db0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "스케일 조정 전의 확률:  [0.10650698 0.10650698 0.78698604]\n",
      "0.5배 조정 후 확률:   [0.21194156 0.21194156 0.57611688]\n",
      "0.1배 조정 후 확률:   [0.31042377 0.31042377 0.37915245]\n"
     ]
    }
   ],
   "source": [
    "logits = np.array([[1.0, 1.0, 3.0]])\n",
    "\n",
    "print('스케일 조정 전의 확률: ', tf.math.softmax(logits).numpy()[0])\n",
    "\n",
    "print('0.5배 조정 후 확률:  ', tf.math.softmax(0.5*logits).numpy()[0])\n",
    "\n",
    "print('0.1배 조정 후 확률:  ', tf.math.softmax(0.1*logits).numpy()[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:53:06.182417Z",
     "iopub.status.busy": "2021-01-01T08:53:06.182031Z",
     "iopub.status.idle": "2021-01-01T08:53:16.415192Z",
     "shell.execute_reply": "2021-01-01T08:53:16.415932Z"
    },
    "id": "ickgML_8yWJt",
    "outputId": "953cf5cb-dfd5-409c-bc90-b4bb688b5a55"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The island was extremely still trees, and a few months this very stream which they had once extremely profitical. The engineer was heard, and the colonists were allowed them to be surprised to the corral, and a few minutes he expected to the ocean, and as the reporter and Neb and Pencroft also accompanied to ascertain if the explosion of the vessel was as position on the island, or rather the colonists had continued to proceed to the angle of the colony, there was not without any attempt to say, there is \n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(1)\n",
    "print(sample(model, starting_str='The island', \n",
    "             scale_factor=2.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:53:16.424579Z",
     "iopub.status.busy": "2021-01-01T08:53:16.424183Z",
     "iopub.status.idle": "2021-01-01T08:53:26.644678Z",
     "shell.execute_reply": "2021-01-01T08:53:26.643763Z"
    },
    "id": "UF4sqqpcyWJt",
    "outputId": "c77a78c3-8a51-48e0-ea35-44ded7d686b8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The island had egepoes how Cyrus 90 whole floorely, the ciffes. He never, whree to\n",
      "netto “and that is go favoce,” depterois, abodpoh Pac; the mosn, upryinsty head, which went worg-siepplementod--ionly!\n",
      "\n",
      "“We can, at lew var.\n",
      "He\n",
      "\n",
      "Who is he speelclosk delkes none to estage on. I’nd from,! Thinking you?” ascenends? To! thelfes! we likels,\n",
      "200\n",
      "throm feel?)\n",
      "save Penholagons Africatibley.\n",
      "\n",
      "You hust my len. He had calked;\n",
      "yee we\n",
      "gatly yem zegans exist in; vercefe tarden-fall, with the rooms, dlengris\n",
      "numberf our \n"
     ]
    }
   ],
   "source": [
    "tf.random.set_seed(1)\n",
    "print(sample(model, starting_str='The island', \n",
    "             scale_factor=0.5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_V30iCf5yWJt"
   },
   "source": [
    "# 트랜스포머 모델을 사용한 언어 이해\n",
    "\n",
    "## 셀프 어텐션 메카니즘 이해하기\n",
    "\n",
    "### 셀프 어텐션 기본 구조"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 349
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:53:26.650577Z",
     "iopub.status.busy": "2021-01-01T08:53:26.649535Z",
     "iopub.status.idle": "2021-01-01T08:53:26.663374Z",
     "shell.execute_reply": "2021-01-01T08:53:26.664093Z"
    },
    "id": "aWx-IXRtyWJt",
    "outputId": "3150262d-5623-41b1-87ed-4d60357acb3c"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdVo\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 21,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdVo', width=700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EXNdJdxnyWJt"
   },
   "source": [
    "### 쿼리, 키, 값 가중치를 가진 셀프 어텐션 메카니즘"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o99EBntFyWJt"
   },
   "source": [
    "## 멀티-헤드 어텐션과 트랜스포머 블록"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 278
    },
    "execution": {
     "iopub.execute_input": "2021-01-01T08:53:26.668048Z",
     "iopub.status.busy": "2021-01-01T08:53:26.667215Z",
     "iopub.status.idle": "2021-01-01T08:53:26.672956Z",
     "shell.execute_reply": "2021-01-01T08:53:26.673524Z"
    },
    "id": "Klo3F7nvyWJu",
    "outputId": "fe751ccf-11cb-4884-8d3f-82790f116c75"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<img src=\"https://git.io/JLdV6\" width=\"700\"/>"
      ],
      "text/plain": [
       "<IPython.core.display.Image object>"
      ]
     },
     "execution_count": 22,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Image(url='https://git.io/JLdV6', width=700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VXU_Pv1zyWJu"
   },
   "source": [
    "\n",
    "...\n",
    "\n",
    "\n",
    "# 요약\n",
    "\n",
    "...\n"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "ch16_part2.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}