chap6.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c4968ddf",
   "metadata": {},
   "source": [
    "# 第6章: 機械学習"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c93e60a6",
   "metadata": {},
   "source": [
    "## 50. データの入手・整形"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e0cd719",
   "metadata": {},
   "source": [
    "[News Aggregator Data Set](https://archive.ics.uci.edu/ml/datasets/News+Aggregator)をダウンロードし、以下の要領で学習データ（train.txt），検証データ（valid.txt），評価データ（test.txt）を作成せよ．\n",
    "\n",
    "1. ダウンロードしたzipファイルを解凍し，readme.txtの説明を読む．\n",
    "2. 情報源（publisher）が”Reuters”, “Huffington Post”, “Businessweek”, “Contactmusic.com”, “Daily Mail”の事例（記事）のみを抽出する．\n",
    "3. 抽出された事例をランダムに並び替える．\n",
    "4. 抽出された事例の80%を学習データ，残りの10%ずつを検証データと評価データに分割し，それぞれtrain.txt，valid.txt，test.txtというファイル名で保存する．ファイルには，１行に１事例を書き出すこととし，カテゴリ名と記事見出しのタブ区切り形式とせよ（このファイルは後に問題70で再利用する）．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "1d2d02df",
   "metadata": {},
   "outputs": [],
   "source": [
    "import csv\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3328137a",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"NewsAggregatorDataset/newsCorpora.csv\", header=None, sep=\"\\t\", names=[\"ID\", \"TITLE\", \"URL\", \"PUBLISHER\", \"CATEGORY\", \"STORY\", \"HOSTNAME\", \"TIMESTAMP\"], quoting=csv.QUOTE_NONE)\n",
    "\n",
    "a = [\"Reuters\", \"Huffington Post\", \"Businessweek\", \"Contactmusic.com\", \"Daily Mail\"]\n",
    "b = data[data[\"PUBLISHER\"].isin(a)]\n",
    "c = b.loc[:, [\"CATEGORY\", \"TITLE\"]]\n",
    "\n",
    "train, valid_test = train_test_split(c, train_size=0.8, shuffle=True)\n",
    "valid, test = train_test_split(valid_test, train_size=0.5)\n",
    "\n",
    "train.to_csv(\"NewsAggregatorDataset/train.txt\", sep=\"\\t\", index=False)\n",
    "valid.to_csv(\"NewsAggregatorDataset/valid.txt\", sep=\"\\t\", index=False)\n",
    "test.to_csv(\"NewsAggregatorDataset/test.txt\", sep=\"\\t\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e39bb549",
   "metadata": {},
   "source": [
    "学習データと評価データを作成したら，各カテゴリの事例数を確認せよ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a73bd7e1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[train]\n",
      " b    4505\n",
      "e    4218\n",
      "t    1238\n",
      "m     723\n",
      "Name: CATEGORY, dtype: int64\n",
      "[test]\n",
      " e    575\n",
      "b    537\n",
      "t    139\n",
      "m     85\n",
      "Name: CATEGORY, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "print(\"[train]\\n\", train[\"CATEGORY\"].value_counts())\n",
    "print(\"[test]\\n\", test[\"CATEGORY\"].value_counts())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3646782b",
   "metadata": {},
   "source": [
    "## 51. 特徴量抽出"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1446275c",
   "metadata": {},
   "source": [
    "学習データ，検証データ，評価データから特徴量を抽出し，それぞれtrain.feature.txt，valid.feature.txt，test.feature.txtというファイル名で保存せよ． なお，カテゴリ分類に有用そうな特徴量は各自で自由に設計せよ．記事の見出しを単語列に変換したものが最低限のベースラインとなるであろう．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0ad3ce7e",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a187e1c0",
   "metadata": {},
   "outputs": [],
   "source": [
    "#単語の出現頻度を特徴量とする。\n",
    "\n",
    "txt_vec = CountVectorizer()\n",
    "\n",
    "#訓練データのみで計算したものをvalid, testに適応。単語は全て小文字に変換。\n",
    "train_feature = txt_vec.fit_transform(train[\"TITLE\"].str.lower())\n",
    "valid_feature = txt_vec.transform(valid[\"TITLE\"].str.lower())\n",
    "test_feature = txt_vec.transform(test[\"TITLE\"].str.lower())\n",
    "\n",
    "with open(\"NewsAggregatorDataset/train.feature.txt\", \"w\") as f:\n",
    "    for d in train_feature.toarray():\n",
    "        f.write(\"%s\\n\" % d)\n",
    "with open(\"NewsAggregatorDataset/valid.feature.txt\", \"w\") as f:\n",
    "    for d in valid_feature.toarray()[0]:\n",
    "        f.write(\"%s\\n\" % d)\n",
    "with open(\"NewsAggregatorDataset/test.feature.txt\", \"w\") as f:\n",
    "    for d in test_feature.toarray()[0]:\n",
    "        f.write(\"%s\\n\" % d)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11caba6f",
   "metadata": {},
   "source": [
    "## 52.学習\n",
    "\n",
    "51で構築した学習データを用いて，ロジスティック回帰モデルを学習せよ．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "2dabe58f",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import confusion_matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "bc8892bf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "LogisticRegression(max_iter=200, random_state=42)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#デフォルトの最大反復回数100だと収束しなかったため200に指定しました。-> 100と精度は変わらない…？\n",
    "lr = LogisticRegression(random_state=42, max_iter=200) \n",
    "lr.fit(train_feature, train[\"CATEGORY\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c0cd0df1",
   "metadata": {},
   "source": [
    "## 53. 予測\n",
    "\n",
    "52で学習したロジスティック回帰モデルを用い，与えられた記事見出しからカテゴリとその予測確率を計算するプログラムを実装せよ．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "7cecb58f",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predictor(model, test_data):\n",
    "    #return カテゴリ予測, 予測確率\n",
    "    return model.predict(test_data), np.max(model.predict_proba(test_data), axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8552982f",
   "metadata": {},
   "source": [
    "## 54. 正解率の計測\n",
    "\n",
    "52で学習したロジスティック回帰モデルの正解率を，学習データおよび評価データ上で計測せよ"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "fbfa8d0d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy(train_data) :  0.9965368775739424\n",
      "Accuracy(test_data) :  0.9161676646706587\n"
     ]
    }
   ],
   "source": [
    "train_pred, train_probs = predictor(lr, train_feature)\n",
    "test_pred, test_probs = predictor(lr, test_feature)\n",
    "\n",
    "def calc_accu(original, predict):\n",
    "    cm = confusion_matrix(original, predict)\n",
    "    accuracy = sum([cm[i][i] for i in range(len(cm))]) / sum(map(sum, cm))\n",
    "    return accuracy\n",
    "\n",
    "print(\"Accuracy(train_data) : \", calc_accu(train[\"CATEGORY\"], train_pred))\n",
    "print(\"Accuracy(test_data) : \", calc_accu(test[\"CATEGORY\"], test_pred))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df452511",
   "metadata": {},
   "source": [
    "## 55. 混同行列の作成\n",
    "\n",
    "52で学習したロジスティック回帰モデルの混同行列（confusion matrix）を，学習データおよび評価データ上で作成せよ．"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd6429fd",
   "metadata": {},
   "source": [
    "学習データ(train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "0b4b67dc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pred b</th>\n",
       "      <th>Pred e</th>\n",
       "      <th>Pred t</th>\n",
       "      <th>Pred m</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>4495</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>7</td>\n",
       "      <td>4210</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>t</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>717</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>m</th>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>1225</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pred b  Pred e  Pred t  Pred m\n",
       "b    4495       3       0       7\n",
       "e       7    4210       0       1\n",
       "t       3       3     717       0\n",
       "m      10       3       0    1225"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "category = [\"b\", \"e\", \"t\", \"m\"]\n",
    "kind = [\"train\", \"test\", \"valid\"]\n",
    "\n",
    "category_pre = [\"Pred b\", \"Pred e\", \"Pred t\", \"Pred m\"]\n",
    "cm_train = confusion_matrix(train[\"CATEGORY\"], train_pred)\n",
    "pd.DataFrame(cm_train, columns=category_pre, index=category)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e035fdc",
   "metadata": {},
   "source": [
    "評価データ(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2f7b6726",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pred b</th>\n",
       "      <th>Pred e</th>\n",
       "      <th>Pred t</th>\n",
       "      <th>Pred m</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>513</td>\n",
       "      <td>10</td>\n",
       "      <td>5</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>12</td>\n",
       "      <td>555</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>t</th>\n",
       "      <td>9</td>\n",
       "      <td>16</td>\n",
       "      <td>55</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>m</th>\n",
       "      <td>21</td>\n",
       "      <td>14</td>\n",
       "      <td>3</td>\n",
       "      <td>101</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pred b  Pred e  Pred t  Pred m\n",
       "b     513      10       5       9\n",
       "e      12     555       3       5\n",
       "t       9      16      55       5\n",
       "m      21      14       3     101"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cm_test = confusion_matrix(test[\"CATEGORY\"], test_pred)\n",
    "pd.DataFrame(cm_test, columns=category_pre, index=category)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8c166ac",
   "metadata": {},
   "source": [
    "## 56. 適合率，再現率，F1スコアの計測\n",
    "\n",
    "52で学習したロジスティック回帰モデルの適合率，再現率，F1スコアを，評価データ上で計測せよ．カテゴリごとに適合率，再現率，F1スコアを求め，カテゴリごとの性能をマイクロ平均（micro-average）とマクロ平均（macro-average）で統合せよ．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "f24588f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_score, recall_score, f1_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "224dffc2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>precision</th>\n",
       "      <th>recall</th>\n",
       "      <th>f1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>0.924324</td>\n",
       "      <td>0.955307</td>\n",
       "      <td>0.939560</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>0.932773</td>\n",
       "      <td>0.965217</td>\n",
       "      <td>0.948718</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>t</th>\n",
       "      <td>0.833333</td>\n",
       "      <td>0.647059</td>\n",
       "      <td>0.728477</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>m</th>\n",
       "      <td>0.841667</td>\n",
       "      <td>0.726619</td>\n",
       "      <td>0.779923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>micro</th>\n",
       "      <td>0.916168</td>\n",
       "      <td>0.916168</td>\n",
       "      <td>0.916168</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>macro</th>\n",
       "      <td>0.883024</td>\n",
       "      <td>0.823551</td>\n",
       "      <td>0.849169</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       precision    recall        f1\n",
       "b       0.924324  0.955307  0.939560\n",
       "e       0.932773  0.965217  0.948718\n",
       "t       0.833333  0.647059  0.728477\n",
       "m       0.841667  0.726619  0.779923\n",
       "micro   0.916168  0.916168  0.916168\n",
       "macro   0.883024  0.823551  0.849169"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "precision = precision_score(test[\"CATEGORY\"], test_pred, average=None)\n",
    "recall = recall_score(test[\"CATEGORY\"], test_pred, average=None)\n",
    "f1 = f1_score(test[\"CATEGORY\"], test_pred, average=None)\n",
    "\n",
    "precision = np.append(precision, precision_score(test[\"CATEGORY\"], test_pred, average=\"micro\"))\n",
    "recall = np.append(recall, recall_score(test[\"CATEGORY\"], test_pred, average=\"micro\"))\n",
    "f1 = np.append(f1, f1_score(test[\"CATEGORY\"], test_pred, average=\"micro\"))\n",
    "\n",
    "precision = np.append(precision, precision_score(test[\"CATEGORY\"], test_pred, average=\"macro\"))\n",
    "recall = np.append(recall, recall_score(test[\"CATEGORY\"], test_pred, average=\"macro\"))\n",
    "f1 = np.append(f1, f1_score(test[\"CATEGORY\"], test_pred, average=\"macro\"))\n",
    "\n",
    "pd.DataFrame(list(zip(precision, recall, f1)), index=category+[\"micro\", \"macro\"], columns=[\"precision\", \"recall\", \"f1\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc8beabc",
   "metadata": {},
   "source": [
    "## 57. 特徴量の重みの確認\n",
    "\n",
    "52で学習したロジスティック回帰モデルの中で，重みの高い特徴量トップ10と，重みの低い特徴量トップ10を確認せよ．"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd2e3f22",
   "metadata": {},
   "source": [
    "BEST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "da9faebe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1位</th>\n",
       "      <th>2位</th>\n",
       "      <th>3位</th>\n",
       "      <th>4位</th>\n",
       "      <th>5位</th>\n",
       "      <th>6位</th>\n",
       "      <th>7位</th>\n",
       "      <th>8位</th>\n",
       "      <th>9位</th>\n",
       "      <th>10位</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>bank</td>\n",
       "      <td>fed</td>\n",
       "      <td>ecb</td>\n",
       "      <td>ukraine</td>\n",
       "      <td>obamacare</td>\n",
       "      <td>china</td>\n",
       "      <td>argentina</td>\n",
       "      <td>euro</td>\n",
       "      <td>yellen</td>\n",
       "      <td>buy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>kardashian</td>\n",
       "      <td>chris</td>\n",
       "      <td>paul</td>\n",
       "      <td>miley</td>\n",
       "      <td>movie</td>\n",
       "      <td>cyrus</td>\n",
       "      <td>transformers</td>\n",
       "      <td>thrones</td>\n",
       "      <td>wedding</td>\n",
       "      <td>hbo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>t</th>\n",
       "      <td>ebola</td>\n",
       "      <td>fda</td>\n",
       "      <td>cancer</td>\n",
       "      <td>study</td>\n",
       "      <td>mers</td>\n",
       "      <td>drug</td>\n",
       "      <td>cdc</td>\n",
       "      <td>cases</td>\n",
       "      <td>medical</td>\n",
       "      <td>cigarettes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>m</th>\n",
       "      <td>google</td>\n",
       "      <td>facebook</td>\n",
       "      <td>climate</td>\n",
       "      <td>apple</td>\n",
       "      <td>microsoft</td>\n",
       "      <td>heartbleed</td>\n",
       "      <td>activision</td>\n",
       "      <td>tesla</td>\n",
       "      <td>nasa</td>\n",
       "      <td>fcc</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           1位        2位       3位       4位         5位          6位  \\\n",
       "b        bank       fed      ecb  ukraine  obamacare       china   \n",
       "e  kardashian     chris     paul    miley      movie       cyrus   \n",
       "t       ebola       fda   cancer    study       mers        drug   \n",
       "m      google  facebook  climate    apple  microsoft  heartbleed   \n",
       "\n",
       "             7位       8位       9位         10位  \n",
       "b     argentina     euro   yellen         buy  \n",
       "e  transformers  thrones  wedding         hbo  \n",
       "t           cdc    cases  medical  cigarettes  \n",
       "m    activision    tesla     nasa         fcc  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ranking = [str(i)+\"位\" for i in range(1,11)]\n",
    "best = []\n",
    "\n",
    "for i in range(len(category)):\n",
    "    lst = sorted(list(zip(lr.coef_[i], txt_vec.get_feature_names_out())))\n",
    "    best.append([lst[:10:-1][j][1] for j in range(10)])\n",
    "pd.DataFrame(best, index=category, columns=ranking)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "645d8ecd",
   "metadata": {},
   "source": [
    "WORST"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "db82c771",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>1位</th>\n",
       "      <th>2位</th>\n",
       "      <th>3位</th>\n",
       "      <th>4位</th>\n",
       "      <th>5位</th>\n",
       "      <th>6位</th>\n",
       "      <th>7位</th>\n",
       "      <th>8位</th>\n",
       "      <th>9位</th>\n",
       "      <th>10位</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>b</th>\n",
       "      <td>activision</td>\n",
       "      <td>aereo</td>\n",
       "      <td>ebola</td>\n",
       "      <td>twitch</td>\n",
       "      <td>cap</td>\n",
       "      <td>she</td>\n",
       "      <td>heartbleed</td>\n",
       "      <td>gay</td>\n",
       "      <td>subscription</td>\n",
       "      <td>nintendo</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>e</th>\n",
       "      <td>google</td>\n",
       "      <td>gm</td>\n",
       "      <td>china</td>\n",
       "      <td>microsoft</td>\n",
       "      <td>billion</td>\n",
       "      <td>facebook</td>\n",
       "      <td>risk</td>\n",
       "      <td>climate</td>\n",
       "      <td>study</td>\n",
       "      <td>obamacare</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>t</th>\n",
       "      <td>gm</td>\n",
       "      <td>google</td>\n",
       "      <td>dimon</td>\n",
       "      <td>facebook</td>\n",
       "      <td>apple</td>\n",
       "      <td>climate</td>\n",
       "      <td>ceo</td>\n",
       "      <td>twitter</td>\n",
       "      <td>bank</td>\n",
       "      <td>netflix</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>m</th>\n",
       "      <td>stocks</td>\n",
       "      <td>percent</td>\n",
       "      <td>move</td>\n",
       "      <td>fed</td>\n",
       "      <td>american</td>\n",
       "      <td>men</td>\n",
       "      <td>grows</td>\n",
       "      <td>valued</td>\n",
       "      <td>thrones</td>\n",
       "      <td>being</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           1位       2位     3位         4位        5位        6位          7位  \\\n",
       "b  activision    aereo  ebola     twitch       cap       she  heartbleed   \n",
       "e      google       gm  china  microsoft   billion  facebook        risk   \n",
       "t          gm   google  dimon   facebook     apple   climate         ceo   \n",
       "m      stocks  percent   move        fed  american       men       grows   \n",
       "\n",
       "        8位            9位        10位  \n",
       "b      gay  subscription   nintendo  \n",
       "e  climate         study  obamacare  \n",
       "t  twitter          bank    netflix  \n",
       "m   valued       thrones      being  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "worst = []\n",
    "for i in range(len(category)):\n",
    "    lst = sorted(list(zip(lr.coef_[i], txt_vec.get_feature_names_out())))\n",
    "    worst.append([lst[:10][j][1] for j in range(10)])\n",
    "pd.DataFrame(worst, index=category, columns=ranking)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1548743d",
   "metadata": {},
   "source": [
    "## 58. 正則化パラメータの変更\n",
    "\n",
    "ロジスティック回帰モデルを学習するとき，正則化パラメータを調整することで，学習時の過学習（overfitting）の度合いを制御できる．異なる正則化パラメータでロジスティック回帰モデルを学習し，学習データ，検証データ，および評価データ上の正解率を求めよ．実験の結果は，正則化パラメータを横軸，正解率を縦軸としたグラフにまとめよ．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "261f62fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "81a229e3",
   "metadata": {},
   "outputs": [],
   "source": [
    "def buildModel_and_calcScore(param, feature_data, original_data, solver=\"lbfgs\"):\n",
    "    lr = LogisticRegression(random_state=42, max_iter=20000, C=param, solver=solver) \n",
    "    lr.fit(train_feature, train[\"CATEGORY\"])\n",
    "    pred, probs = predictor(lr, feature_data)\n",
    "    return calc_accu(original_data, pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "a65bd531",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEKCAYAAAAfGVI8AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8qNh9FAAAACXBIWXMAAAsTAAALEwEAmpwYAAA5cUlEQVR4nO3deZxN5R/A8c/33tlnMIzdWLOvgwmlZA0lQkVp/SVJIrRok8qv9KOikqK0R2hTKYxIkez7kp2xzdiG2WfufX5/3MEYQ4M5c2bmft+v133NWZ5z7vce1/nec57zPI8YY1BKKeW9HHYHoJRSyl6aCJRSystpIlBKKS+niUAppbycJgKllPJymgiUUsrL+dgdwKUqWbKkqVKlit1hKKVUgbJy5cojxphS2a0rcImgSpUqrFixwu4wlFKqQBGRPRdap7eGlFLKy2kiUEopL6eJQCmlvFyBqyPITlpaGtHR0SQnJ9sdiuUCAgIIDw/H19fX7lCUUoWEZYlARKYAXYAYY0z9bNYLMB64CUgE7jfGrLqc94qOjqZIkSJUqVIFz24LJ2MMR48eJTo6mqpVq9odjlKqkLDy1tAnQKeLrO8M1Mh49QMmXu4bJScnExYWVqiTAICIEBYW5hVXPkqpvGPZFYExZpGIVLlIkW7AZ8bTD/ZSEQkVkXLGmIOX836FPQmc5i2fU1nD5TakprtJTXeT4nKdmU51uc9Op7tJc5/bPX3W7urP67w+m97sDVn3wcXnL+FzeKvaZYtQsURQru/XzjqCCsC+TPPRGcvOSwQi0g/PVQOVKlXKk+AuxYkTJ/jqq68YMGDAJW1300038dVXXxEaGmpNYKpASUhJZ8HWGBJTXKRkOTGnZjlpp5yzLvvplGzWudx6ui3IRt1an7tbVM71/RaIymJjzCRgEkBkZGS++yafOHGC995777xEkJ6ejo/PhQ/x7NmzrQ5NFRB/7zzKkzPXsfdYYrbrHQL+Pk78fByel9OB/+npjHk/HwchAT5npv18Mso4M5dz4u/r+JcyDnycDrK7+My6KLsr1OyuWbPfVzbb6gXvRZUrFmDJfu1MBPuBipnmwzOWFTjDhw9nx44dRERE4OvrS0BAAMWLF2fLli38888/3Hrrrezbt4/k5GQGDx5Mv379gLOtpOPj4+ncuTPXXXcdS5YsoUKFCvzwww8EBgba/MmU1ZLTXPzv1618vGQXFYsH8el/mlGtZPB5J3kfpz7praxjZyKYBQwUkWlAcyDucusHMnvpx41sOnDyioPLrG75orx4S70Lrh89ejQbNmxgzZo1LFy4kJtvvpkNGzacebJnypQplChRgqSkJK6++mp69uxJWFjYOfvYtm0bU6dOZfLkydxxxx1888033H333bn6OVT+smrvcZ6YvpadRxK4p0VlhneuTbB/gbhIV4WMlY+PTgVaAyVFJBp4EfAFMMa8D8zG8+jodjyPjz5gVSx5rVmzZuc83vn222/z3XffAbBv3z62bdt2XiKoWrUqERERADRt2pTdu3fnVbgqj6Wku3hr3jYmLdpBuWKBfNm3OS2rl7Q7LOXFrHxq6M5/WW+AR3P7fS/2yz2vBAcHn5leuHAhUVFR/PXXXwQFBdG6detsH//09/c/M+10OklKSsqTWFXeWh8dx7AZa/jncDy9IivyfJc6FAnQxoHKXnodmguKFCnCqVOnsl0XFxdH8eLFCQoKYsuWLSxdujSPo1P5QWq6m3cXbGfCgu2UDPHj4/uvpk3t0naHpRSgiSBXhIWF0bJlS+rXr09gYCBlypQ5s65Tp068//771KlTh1q1atGiRQsbI1V22HLoJEO/Xsumgyfp3rgCI2+pR7EgvQpQ+YdkbSiS30VGRpqs4xFs3ryZOnXq2BRR3vO2z1tQpbvcfLBoJ+Oi/qFYoC//7d6AjvXK2h2W8lIistIYE5ndOr0iUMoC22NOMWzGOtbuO8HNDcrxyq31KRHsZ3dYSmVLE4FSucjlNkz5cxdj5m4lyM/JO3c25pZG5e0OS6mL0kSgVC7ZfSSBJ2asZcWe47SvU4ZXe9SndBFrWoIqlZs0ESh1hdxuw+dL9zD6ly34OIU3bm9EjyYVtINAVWBoIlDqCuw7lsjT36xjyY6jtKpZitd7NqBcMe0aRBUsmgiUugzGGKYt38eonzYB8FqPBvS+uqJeBagCSXuyygWnex+9HOPGjSMxMfseJ1X+dDAuifs/Xs4z366nYXgovz7eijubVdIkoAosTQS5QBOBdzDG8M3KaG58axHLdh3jpa71+LJvc0sGClEqL+mtoVyQuRvqDh06ULp0aaZPn05KSgrdu3fnpZdeIiEhgTvuuIPo6GhcLhcvvPAChw8f5sCBA7Rp04aSJUuyYMECuz+KuoCYU8k8990G5m06TGTl4oy9vRFVSgb/+4ZKFQCFLxH8MhwOrc/dfZZtAJ1HX3B15m6o586dy8yZM1m2bBnGGLp27cqiRYuIjY2lfPny/Pzzz4CnD6JixYrx5ptvsmDBAkqW1N4n86sf1x5gxA8bSEh18dxNdfjPdVVxOvQ2kCo8Cl8isNncuXOZO3cujRs3BiA+Pp5t27Zx/fXXM2zYMJ5++mm6dOnC9ddfb3Ok6t8cS0jlhe838PP6gzQKL8YbdzSieukidoelVK4rfIngIr/c84IxhmeeeYaHH374vHWrVq1i9uzZPP/887Rr144RI0bYEKHKiTkbD/Hcd+uJS0rjyY61eLhVNR0lTBVahS8R2CBzN9QdO3bkhRdeoE+fPoSEhLB//358fX1JT0+nRIkS3H333YSGhvLhhx+es63eGsof4hLTGPnjRr5bvZ+65Yry+YPNqVOuqN1hKWUpTQS5IHM31J07d+auu+7immuuASAkJIQvvviC7du38+STT+JwOPD19WXixIkA9OvXj06dOlG+fHmtLLbZgq0xDP9mHUfiUxnUrgYD21THz0evAlThp91QF0De9nmtlpruZsQPG5i2fB81y4Twxu0RNAgvZndYSuUq7YZaqYt4Y+5Wpi3fx8M3VGNoh5r4+zjtDkmpPKWJQHm1Rf/E8sGindzVvBLPdNarLOWdNBEor3UkPoWh09dSo3QIL9xc1+5w8g9jMl7u819kXn6BMuctv4J9OXzALxh8g8EvCHyDPPNOHeozN2kiUF7JGMOTM9ZyMjmNzx9sRqCfjbeDkuNgzxLYtQgOrgN32r+fULnYSfhC6zLWX3DbTOvzO6ff2aTgF5z99HnLgsAv5Oy0b8a6M9MZfx3e94CAJgLllT5evJsFW2MZeUvdvH88NDUB9v7lOfHv+gMOrvGchH0CoGxD8A0EcVzgJRmvbNYhWcplt+1F5jO2dyOkicGF4BJwAekILjG4EdLhzHLPy+ASSAfcgMsYTxkMbsiYdp8tmzGfjsFtIB3jWW7cGeuNZx/GDcYFrjRPcnSlnTt9ZlkquNLBHQfJsZCQljGf6llvXJf27+PwAYevJ9k4fcHpi6/4EOLwIVh8CXL4Eez0JdgZQLDTnyCHPyE+gQT7BODnE+DZzuF7Ztsz+8myT085n4xlfuD0ybKtX/bbW5CoNBEor7PxQByjf9lCu9qlue/aKta/YVoyRC/znPR3LYL9Kz0nMYcvhEdCqyehyvUQfjX4Zj+iWbo7nRRXCsnpySS7kklJTyHZlZztfOZyZ+azWZeSlkKSK+m8bVNcKdYfkxxwiAPHpfSL6ch4+Qrgm/E6LetVjslmMrsroTQMabgyFzmdAbPhYwzBbjfBbkOQ8fwNdrsJPm95xjrjJshtCMlmmyBjzv/0N78BV/f9lwNx6SxNBCLSCRgPOIEPjTGjs6yvDEwBSgHHgLuNMdFWxpQfhISEEB8fz4EDBxg0aBAzZ848r0zr1q0ZO3YskZHZPu2lLlNiajqDpq4mNMiX/93W0Jquo11psH+V56S/exHs/RtcKZ5f3eUbwzWPQtVWUKkF+AXjNm7WH1lP1NoJrDy8ksS0RM+JOtNJO92dflmh+Dh8CHAG4O/0J8AnwDPt40+AM4AQvxBKOkuemc+83s/hh4/DB6c4cTqcZ/76iA9OhxOHOM5MO8WJj8MHhzjOTDslo0zWfWTaj0Mc5+wza7n81K13mjuNxLREEtMSSUhLICE9wfM30+vMurQEEtPiSUiNJyEtnlOpCRxKTyAhLZHE9EQS0pNw5/D2W6D4EuzwJcThS5D40NfH0MGCz2dZIhARJzAB6ABEA8tFZJYxZlOmYmOBz4wxn4pIW+A14B6rYspvypcvn20SUNZ55adN7DySwOf/aU5YiH/u7NTtgkPrMm71LII9f0FagmddmQaeX3BVW0HlayDA0z4h3Z3OqsOrmLdnHr/t/Y2YpBh8HD5ElIqgTFAZAnw8J+9An0D8nf74+/gT6Aw856Sd9eR+en3m7XwcetGfG3wdvhTzL0Yx/ytvX2KMIdmVfE7yiE+LPyfJZE4qp8vFp8XjX6pWLnya81n5LWkGbDfG7AQQkWlANyBzIqgLDM2YXgB8b2E8lhk+fDgVK1bk0UcfBWDkyJH4+PiwYMECjh8/TlpaGqNGjaJbt27nbLd79266dOnChg0bSEpK4oEHHmDt2rXUrl2bpKQkOz5KoTZ7/UGmLvO0F7iuxhV06eF2Q+zms/f49/zpqfAFKFkLIu6CqtdD5esgOOzMZqmuVJZGL2L+3vks2LuA4ynHCXAG0LJCS9pVascNFW+gqJ92Z1HYiQiBPoEE+gRCPhnV1MpEUAHYl2k+GmiepcxaoAee20fdgSIiEmaMOZq5kIj0A/oBVKpU6aJv+vqy19lybMuVRZ5F7RK1ebrZ0xdc36tXLx5//PEziWD69OnMmTOHQYMGUbRoUY4cOUKLFi3o2rXrBS93J06cSFBQEJs3b2bdunU0adIkVz+Dt9t/Ionh36yjYXgxhnW4xF9VxsDR7Wd/8e/+ExKPeNYVrwJ1u0HVG6DKdVCk7DmbJqYlsuTAEubtmcei6EXEp8UT7BtMq/BWdKjcgZblWxLkqwPbKHvZfd34BPCuiNwPLAL2k001jDFmEjAJPF1M5GWAOdG4cWNiYmI4cOAAsbGxFC9enLJlyzJkyBAWLVqEw+Fg//79HD58mLJly2a7j0WLFjFo0CAAGjZsSMOGDfPyIxRqLrdhyLQ1uNyGt3s3zln/Qcf3ZJz0Myp4Tx30LC9aAWp08FTuVr0eQs//YXIq9RS/R//O/D3z+XP/nyS7kgn1D6VD5Q60r9yeFuVa4Of0y+VPqdTlszIR7AcqZpoPz1h2hjHmAJ4rAkQkBOhpjDlxJW96sV/uVrr99tuZOXMmhw4dolevXnz55ZfExsaycuVKfH19qVKlCsnJybbE5u3e/W07y3Yf442LjSp26lDGL/7fPX9P7PUsDy6VcdJv5XmVqOZ57DKLY8nHWLB3AVF7o1h6cCnp7nRKBZbi1uq30r5ye5qWaar361W+ZeU3czlQQ0Sq4kkAvYG7MhcQkZLAMWOMG3gGzxNEBVKvXr146KGHOHLkCL///jvTp0+ndOnS+Pr6smDBAvbs2XPR7Vu1asVXX31F27Zt2bBhA+vWrcujyAu3FbuPMX7+P3SLKE+PJhWyL7T2a/j+Ec/z5gGhnls81wz0nPhL1c72xA9wOOEw8/fOJ2pvFCsPr8Rt3FQIqUCf2n1oX7k9DUs1xCHe1zhJFTyWJQJjTLqIDATm4Hl8dIoxZqOIvAysMMbMAloDr4mIwXNr6FGr4rFavXr1OHXqFBUqVKBcuXL06dOHW265hQYNGhAZGUnt2rUvuv0jjzzCAw88QJ06dahTpw5NmzbNo8gLr7ikNAZPW0OF4oGMurV+9vUzq7+AHwZ6Tv4d/wtl6oPjwq2M953aR9SeKKL2RrEu1pOsqxWrRt8GfWlfqT21S9TOV489KpUT2g11AeRtn/dyGGMYOHU1v244xIz+19CkUvHzC638BH4cDNXaQO+vPF0MZLOfHSd2ELU3iqg9UWw9vhWAOiXq0L5ye9pXak+10GoWfxqlrpx2Q628zowV0fy87iBPdqyVfRJY/iH8PAyqd4BeX5zTotcYw6Zjmzy//PdEsfvkbgQhonQET0Q+QbtK7QgvEp6Hn0Ypa2kiUIXOjth4Xpy1kRbVStD/hqvOL/D3B/DLU1CzE9zxGfj443K7WBu7lqi9UczfM58DCQdwipPIspH0qdOHdpXaUSqoVN5/GKXygCYCVaikpLsYNHU1/r4OxvVqjNOR5X79kndh7nNQuwvc9jH4+LHl2BYGzh/I4cTD+Dp8ubb8tfRv1J/WFVtTPCCbqwmlCplCkwiMMV5RSVfQ6nTy2phft7LxwEkm3dOUssWydOD251sQNdLTAKznR+D0JfpUNI9EPYJTnLx+/eu0Cm9FiF+ILbErZZdCkQgCAgI4evQoYWFhhToZGGM4evQoAQHZ91Dp7RZujeHDP3dxT4vK3FgvS8O938fAglFQvyd0nwROH44mHeXheQ+T6krls86fcVVoNreRlPIChSIRhIeHEx0dTWxsrN2hWC4gIIDwcK2ozCr2VApPzFhLrTJFeO7mTE9UGQMLR8Pvo6FhL+j2Hjh9SExL5NH5jxKTGMPkGydrElBerVAkAl9fX6pWrWp3GMombrfhiRlrOZWczpd9WxDgm9EOwBj47RX44w2I6ANd3wGHkzRXGkMWDmHLsS2MazOOiNIRtsavlN0KRSJQ3m3K4l38/k8sr3SrR62yRTwLjYF5I2DJ29DkPugyDhwO3MbN84ufZ8mBJbx87cu0rtjaztCVyhc0EagCbcP+OF7/dQsd6pbh7haVPQuNgTnPwtL3IPJBuGksOBwYYxi7Yiyzd81mUONBdK/R3d7glconNBGoAisxNZ1B01ZTItiP13tmjDZmjKeNwLJJ0Lw/dBp9pq+gTzZ+wuebPueu2nfRt0HuD/enVEGliUAVWC/N2sSuIwl82bc5JYL9PAPGzB4GK6Z4Oo27cdSZJDBrxyzeXPkmHat05OlmTxfqp8uUulSaCFSB9PO6g3y9Yh8DWl/FtVeV9CSBnwbDqs+g5ePQfuSZJPBH9B+MWDyC5mWb8+p1r2qPoEploYlAFTjRxxMZ/u06IiqGMqRDTc+YwT8MhLVfQasnoc1zZ5LAuth1DPt9GDWL12Rcm3E6IIxS2dBEoAqUdJebx6etwRh4u3djfHHDd4/A+unQ+llofXZgol1xu3h0/qOEBYTxXvv3tMWwUhegiUAVKO/8tp0Ve44zrlcElUJ94duHYOO30PYFaPXEmXKHEw7z8LyHcYiDDzp8QMnAKxisXqlCThOBKjCW7TrGO79to0fjCtzaoBTM/A9sngUdXoaWg8+UO5l6kv5R/YlLiWNKpylUKnr+uMJKqbM0EagCIS4xjcenraZiiSBe7lIDZtwPW3+Gjq/CNWcHtktOT+ax+Y+x++Ru3mv3HvXC6tkXtFIFhCYCle8ZY3jmu3XEnErh24eaEPL9A7BtDnQeA837nSnncrsY/sdwVsWs4n+t/sc15a+xMWqlCg5NBCrf+3r5PmavP8SzHarQ8M8BsGM+dHkLIv9zpowxhlF/j2L+3vkMbzaczlU72xixUgWLJgKVr22PieelHzfRplowD0U/C7t+93Qe1+Tec8pNXDuRmf/M5MH6D9KnTh+bolWqYNJEoPKt06ONhfqk8oHzbWTXErj1PYi465xy07dOZ+Laidxa/VYGNxl8gb0ppS5EE4HKt17/ZSt7Dh7mz/CJ+EWvgh6ToeHt55SJ2hPFqKWjaBXeiheveVG7jlDqMmgiUPnSgq0xTF+8kV/CxlP86Cbo+aFndLFMlh9aztOLnqZBqQaMvWEsPg79Oit1OfR/jsp3Yk4lM/LrJcwM/h8Vk3bC7R97xhnOZOuxrQz6bRDhRcKZ0HYCgT6BNkWrVMFnae9bItJJRLaKyHYRGZ7N+koiskBEVovIOhG5ycp4VP7ndhtGTP2DCa6XqGV2IXd8dl4S2B+/n0eiHiHIN4j3279PaECoPcEqVUhYlghExAlMADoDdYE7RaRulmLPA9ONMY2B3sB7VsWjCoYvflvFY9HDqOPYh/T6AmrffM76Y8nH6D+vP8muZD5o/wHlQsrZFKlShYeVVwTNgO3GmJ3GmFRgGtAtSxkDFM2YLgYcsDAelc9t2raDZn/cR03nARx3TYVanc5Zn5iWyMD5AzmYcJB3275L9eLVbYpUqcLFyjqCCsC+TPPRQPMsZUYCc0XkMSAYaJ/djkSkH9APoFIl7TemMEo8up+gqd0oK4dJuX0qvjU6nLM+zZ3G0IVD2Xh0I+Naj6NJmSY2RapU4WP3CB13Ap8YY8KBm4DPRc4fNcQYM8kYE2mMiSxVqlSeB6ksdvIg8R90orQrhp03fkJI3XOTgNu4GbF4BIsPLGZEixG0qdTGpkCVKpysTAT7gYqZ5sMzlmX2IDAdwBjzFxAAaH/B3iRuP/EfdCQoJYZZDd6m7rU3n1fkrZVv8dPOn3is8WP0rNkzm50opa6ElYlgOVBDRKqKiB+eyuBZWcrsBdoBiEgdPIkg1sKYVH7hdsGaqaRPbg/xMfy3xChu637HecU+3fgpn2z8hN61evNQg4dsCFSpws+yOgJjTLqIDATmAE5gijFmo4i8DKwwxswChgGTRWQInorj+40xxqqYVD7gdsOm72DhaDjyD3ud1XiGEYy9pw8+znN/l/y440fGrhjLjZVvZHiz4dpqWCmLWNqgzBgzG5idZdmITNObgJZWxqDyCWNgy0+w4DWI2ciJkKt4yQzl19RIxvVuSsUSQecUX7x/MSMWj6BZ2Wa8dv1rOB1OmwJXqvDTlsXKWsbAtnmwYBQcXEtqsapMLPYU4w835NrqpZnTvQGVws5NAhuObGDIwiFUL15dB5xXKg9oIlDWMAZ2LoQF/4Xo5ZjQykTVfJFBG2vi5+fH67fV4bam4efd7tkdt5sBUQMoEVCCie0nUsSviD3xK+VFNBGo3Ld7sScB7FkMRSuw99pXeWRDbTauS6ZLw3K8eEs9ShXxP2+z2MRY+kf1R0R0wHml8pAmApV7olfAb6Ng5wIIKUNKh9cYc6QFHy04SNmi8OG9kbSvWybbTU+lnqJ/VH+OJR/j444fU7lo5TwOXinvpYlAXbkDa2DBq55xhIPC4MZRLCrWlWd+3MGBuIPc06IyT3asRZEA32w3T3GlMOi3QeyM28mEdhOoV1IHnFcqL2kiUJfv8EZPAtjyEwSEQrsRHKv/AC/P2cP3azZQvXQIMx6+hsgqJS64C5fbxTN/PMOKwyt4/frXubb8tXkXv1IK0ESgLseRbbDwNdjwLfgXgRuGY1o8wg9bEnj53ZWcSk5jcLsaDGhzFf4+F37sMzk9mdHLRjNvzzyeuvopbqqmvZArZYd/TQQicgvwszHGnQfxqPzs2E74/X+w7mvwCYDrhsC1jxGdEsBzUzfw+z+xNK4Uyus9G1KzzIWf9nG5Xfyw4wcmrJlATGIMfRv05Z669+ThB1FKZZaTK4JewDgR+QZP6+AtFsek8psT+2DRGFjzJTh8oMUAaPk4rqCSfLpkN2PnLkOAl7rW4+4WlXE6sm8BbIzh9+jfGbdyHDvidtCwVEP+1+p/NC3TNG8/j1LqHP+aCIwxd4tIUTJ6ChURA3wMTDXGnLI6QGWjkwfhjzdg1aee+cj/wHVDoWg5thw6ydOfLmHtvhO0qVWKUd0bUCH0wsNFro1dy5sr3mRVzCqqFK3CW63fol2ldtpthFL5QI7qCIwxJ0VkJhAIPA50B54UkbeNMe9YGJ+yQ3wsLB4Hyz8EdzpE9IFWT0JoRZLTXEyYu5WJC3dQLNCX8b0j6Nqo/AVP6LvidvH2qreJ2htFWEAYL7R4ge41uuPryP4JIqVU3stJHUFX4AGgOvAZ0MwYEyMiQcAmQBNBYZF4DJa8DX9PgvQkaNgbbngKSlQFYNmuYwz/dh07YxPo0aQCL9xcl+LB2Xf/EJsYy8S1E/l227f4O/15NOJR7q17L0G+QdmWV0rZJydXBD2Bt4wxizIvNMYkisiD1oSl8lRyHPw1Af56D1LjoX5PaD0cStYA4GRyGq//soUv/95LePFAPvtPM1rVzH6AoPjUeD7e+DGfb/qcNFcavWr1ol/DfoQFhuXlJ1JKXYKcJIKRwMHTMyISCJQxxuw2xsy3KjCVB1Li4e/3Yck7kHwC6twCrZ+FMnXPFJm78RAv/LCB2FMp9L2uKkNvrEmQ3/lfmzRXGtP/mc6kdZM4lnyMTlU6MajxICoWrXheWaVU/pKTRDADyNzKx5Wx7GpLIlJ5Y81UmPscJB6Fmp2gzbNQrtGZ1TGnkhk5ayOz1x+idtkiTLonkkYVQ8/bjdu4mbN7Dm+vepvo+GialW3GkKZDqF+yfh5+GKXUlchJIvAxxqSenjHGpGaMOKYKquO7YdZjUD4C7poO4ZFnVhljmLEimlE/byI53c2THWvRr1U1fJ3nD2b398G/eXPlm2w6uomaxWsysf1EWpZvqU8CKVXA5CQRxIpI14wRxRCRbsARa8NSlvptlKc9wB2fQ9FyZxbvPpLAM9+u56+dR2lWtQSjezSgWqmQ8zbfemwrb618i8UHFlMuuByvXvcqN1W9SQePUaqAykki6A98KSLvAgLsA+61NCplnQNrYP0MuH7YmSSQ7nLz4Z+7eGveP/j5OHitRwN6RVbEkaVh2IH4A7y7+l1+2vkTRfyK8ETkE/Su3Rt/5/ldSiulCo6cNCjbAbQQkZCM+XjLo1LWiRoJgSWg5WAANuyP4+lv1rHxwEk61ivDy93qU6ZowDmbnEg+weT1k5m6ZSqCcH/9+3mw/oMU8y9mwwdQSuW2HDUoE5GbgXpAwOn7v8aYly2MS1lhx2+esQI6vkaSI4Rxszfz4Z+7CAv24/27m9CpfrlziienJ/Pl5i/5aP1HJKQn0PWqrjwa8Shlg8va9AGUUlbISYOy94EgoA3wIXAbsMziuFRuc7th3osQWglX0/9w5+SlrNl3gjubVWR45zoUCzzb0tfldjFrxyzeXfMuMYkx3BB+A4ObDKZG8Ro2fgCllFVyckVwrTGmoYisM8a8JCJvAL9YHZjKZRtmwqF10GMyX6+OYc2+E4y5rSG3R559zv90p3DjV41n+4ntNCjZgNHXj+bqsvqksFKFWU4SQXLG30QRKQ8cBcpdpLzKb9JT4LdXoGxD4q7qxtg3F9Gsagluaxp+pkjmTuEqF63MGze8QYfKHfRRUKW8QE4SwY8iEgqMAVYBBphsZVAqly3/CE7shXvG89b87ZxITGXkLfUQEXbF7eKd1e8wb888wgLCeL758/So2UM7hVPKi1w0EYiIA5hvjDkBfCMiPwEBxpi4nOxcRDoB4wEn8KExZnSW9W/hqXsATz1EaWNM6CV9AnVxyXGesQSqtWFr8NV8vvQP7mpeidLFU3nlr1f4Zts3+Dv9GRAxgPvq3qedwinlhS6aCIwxbhGZADTOmE8BUnKyYxFxAhOADkA0sFxEZhljNmXa/5BM5R87/T4qFy0eD0nHMO1f5KUfNxLi78PAtpW5Z3YvDiUc4vaat/Nwo4cpGVjS7kiVUjbJya2h+SLSE/jWGGMuYd/NgO3GmJ0AIjIN6Ian6+rs3Am8eAn7V//m5AFPj6INbmfOsbIs2bGKV7rV45udnxMdH83kGyfTolwLu6NUStns/A5kzvcwnk7mUkTkpIicEpGTOdiuAp5WyKdFZyw7j4hUBqoCv+VgvyqnFr4GxkVKq2d55afN1C5bhJZ1YMr6KdxU9SZNAkopIGctiy88Cnnu6Q3MNMa4slspIv2AfgCVKlXKg3AKgZgtsPoLaN6f99e62H8iiakPNWfMihfwdfoyLHKY3REqpfKJnDQoa5Xd8qwD1WRjP5C5M/rwjGXZ6Q08eqEdGWMmAZMAIiMjL+X2lPea/zL4hXCw0aNMfG89NzcsR5LvOv7c/ydPRj5J6aDSdkeolMonclJH8GSm6QA89/5XAm3/ZbvlQA0RqYonAfQG7spaSERqA8WBv3ISsMqBvUth68/Q9gVGLYgBYOiNVRmwsDfVQ6tzZ507bQ5QKZWf5OTW0C2Z50WkIjAuB9uli8hAYA6ex0enGGM2isjLwIrT3VrjSRDTLrEiWl2IMTD3BShSjr/L9Obn2WsZ2qEms/d9wYGEA3zc8WNtI6CUOkeOOp3LIhqok5OCxpjZwOwsy0ZkmR95GTGoC9nyE0Qvw9VlPC/+spPw4oF0inDSe/bHdKnWhciykf++D6WUV8lJHcE7eFoTg+cpowg8LYxVfuNKh6iXoGRNvkq5ji2HtjKxT2PeXPUy/k5/rSBWSmUrJ1cEKzJNpwNTjTGLLYpHXYnVn8PRbcTf+hljf9hJy+ph+BbdxOJVi3n66qe10ZhSKls5SQQzgeTTj3aKiFNEgowxidaGpi5JaoKn3UDFFozeVZX4lGie7lyNJ/66mxrFa9C7dm+7I1RK5VM5aVA2HwjMNB8IRFkTjrpsS9+D+MPsbPwUXy3bxz0tKrPw8DQOJhzkuebP4eO4nOogpZQ3yEkiCMg8PGXGtPZMlp8kHIE/x2Nq38zwZUGEBvnRs7k/H2/8mK5XdaVpmaZ2R6iUysdykggSRKTJ6RkRaQokWReSumSLxkBaIgvDB7Bs9zGeuLEm764bQ4AzgCFNh/z79kopr5aT+wWPAzNE5AAgQFmgl5VBqUtwbBcs/4i0Rnfz7B8p1K9QlLDSW1myZQnDmw3XCmKl1L/KSYOy5Rmtf2tlLNpqjEmzNiyVY7+NAocPH/ncwcG4OMbeUZsXV9xHreK16FVL87VS6t/9660hEXkUCDbGbDDGbABCRGSA9aGpf3VgNWyYSVzjfry59BS3RpRn+YkZHE48zHMttIJYKZUzOakjeChjhDIAjDHHgYcsi0jljDEw70UICmPEkXb4OIS7rgvk002f0u2qbjQurWP8KKVyJieJwCmZRjDPGHnMz7qQVI7smA+7fmdHnQH8sDmeAa2vYvKmNwh0BmoFsVLqkuQkEfwKfC0i7USkHTAV+MXasNRFud0wbyQmtDID/4mgclgQVSpvZ+nBpTzW5DHCAsPsjlApVYDkJBE8jWfksP4Zr/Wc28BM5bX1M+Dwen4P78/m2BSe6lSVt1aNpU6JOtxR8w67o1NKFTD/mgiMMW7gb2A3nrEI2gKbrQ1LXVBaMvw2ivQyDRm0oSo31CzF1pRviUmM4dnmz+J0OO2OUClVwFzwsRIRqYlnQPk7gSPA1wDGmDZ5E5rK1oqPIG4vn5QYQmKq4f7WQQz983O6V+9OROkIu6NTShVAF3u+cAvwB9DFGLMdQES0FtJOSSdg0RhOVWjFf7eU4cGWlfly21sE+gbyeNPH7Y5OKVVAXezWUA/gILBARCZnVBTLRcorqy0eh0k6wYuJtxMW7E+9mrv4+9DfDG48mBIBJeyOTilVQF0wERhjvjfG9AZqAwvwdDVRWkQmisiNeRSfOi1uPyydyL7wLnx7MIzHO1Tk3TVvUadEHW6reZvd0SmlCrCcVBYnGGO+yhi7OBxYjedJIpWXFr6GMW4eO3QTjSqGcsjxIzFJMTzf4nmtIFZKXZGcPD56hjHmuDFmkjGmnVUBqWzEbIE1X7K8VE/Wxhejb9tAvtz8JT1r9KRhqYZ2R6eUKuAuKREom0SNxOUbwsB9benZpALf7X2XIN8gBjcZbHdkSqlCQBNBfrdnCfzzC98F3U6iTzGurr+H5YeWM7jJYIoHFLc7OqVUIaCJID8zBuaNIDmwDM8fuo7+bcozacN46oXVo2eNnnZHp5QqJDQR5Gebf4To5bzjvp3yJUsQH/gLR5KOaAWxUipXWZoIRKSTiGwVke0iMvwCZe4QkU0islFEvrIyngLFlQbzX+JYcDXej2tO33aBTNv6FT1r9qR+yfp2R6eUKkQsG7kko7vqCUAHIBpYLiKzjDGbMpWpATwDtDTGHBeR0lbFU+Cs/hyObucF91O0rlOWuYffI8QvhMGNtYJYKZW7rLwiaAZsN8bsNMakAtOAblnKPARMyBjsBmNMjIXxFBypCbBwNDsDGzLP1ZjrI/ax8vBKHm/yOKEBoXZHp5QqZKxMBBWAfZnmozOWZVYTqCkii0VkqYh0sjCeguOvCRB/mCdO9OCelmX4ZMs7NCjZgB41etgdmVKqELK7stgHqAG0xtPL6WQRCc1aSET6icgKEVkRGxubtxHmtfhYzOLxLPG7lv1FGiDF53Is+RjPNX8Oh9j9z6WUKoysPLPsBypmmg/PWJZZNDDLGJNmjNkF/IMnMZwjozVzpDEmslSpUpYFnC8sGoNJS+L5Uz24v7U/M7ZN4/aat1OvZD27I1NKFVJWJoLlQA0RqSoifkBvYFaWMt/juRpAREriuVW008KY8rdjOzErpvAdbQmtVJe/TkymqF9RBjUZZHdkSqlCzLJEYIxJBwYCc/CMaDbdGLNRRF4Wka4ZxeYAR0VkE54eTp80xhy1KqZ8b/4rpOHk9eRbaRe5j9WxqxnSdAjF/IvZHZlSqhATY4zdMVySyMhIs2LFCrvDyH37V8HkNrzj6sGeiH4sTx9OeEg4n9/0udYNKKWumIisNMZEZrdOzzD5gTGYqBc56SjGl86uBJeZz/Hk4zzXQiuIlVLW07NMfrB9PrJrEW+k3Mqt1xfl+50zuKPWHdQNq2t3ZEopL2BZy2KVQ24X7nkjOCBl+btEF0qmfEIxv2I81vgxuyNTSnkJvSKw2/oZOGI2Mjrldtq1iGFt7BqtIFZK5SlNBHZKSyY96mU2mGok12rHT9GTaVSqEd2qZ+2JQymlrKOJwE7LJ+Nzaj//c99FqSqLOJFyQlsQK6XynJ5x7JJ0nPSFY1joakR4i/rM3vMtvWr1ok5YHbsjU0p5Ga0ston7j3E4Uk/yUcA9pJsvCPUPZWDjgXaHpZTyQnpFYIe4aNxLJ/KdqyXVrvFhw9F1DG06lKJ+Re2OTCnlhfSKwAapUf8Ft4tfy/Zha8z7NC7dmFuuusXusJRSXkqvCPLa4U34rJ/Gp+kdKVJ7C3GpcVpBrJSylZ598pLbTcIPw4g3AWxt0IWo6O+5s/ad1CpRy+7IlFJeTBNBHkpb/C7BB5bwhtzD3sBZlAgowYCIAXaHpZTycpoI8og5vBF+e5k5rkh8W9dm8/GNDIscphXESinbaSLIC+kpHP3sPk64A9nR8gXmHPqYJqWb0KVaF7sjU0opTQR5YdeMZymZsI1vKj7NP/7fcyr1FM82fxYRsTs0pZTSRGC13SvnUnnLR/wY0JH1VbYyf+98hjYdqhXESql8Q9sRWOjY0Vj8fxzADkcZvmkQzMr9C3mu+XP0rt3b7tCUUuoMvSKwSGq6m3Uf9ieIYzxftw6rj67i5Wtf1iSglMp3NBFYwBjD9M8n0DhlPg9Uq8fWpN2Mvn403Wt0tzs0pZQ6j94assCMhSu4du/r3FehMnslgTdueIN2ldrZHZZSSmVLE0EuW7wtloBFQxhSvgj7/X14p+3btKzQ0u6wlFLqgjQR5KJdRxL4adqLrKtwhFi/ICZ2eJ+ry15td1hKKXVRmghyycnkNIZ/NoVj5RcS7/RncqcpNCodYXdYSin1ryytLBaRTiKyVUS2i8jwbNbfLyKxIrIm49XXynis4nIb+n/1AweKTSJZhI/avqNJQClVYFh2RSAiTmAC0AGIBpaLyCxjzKYsRb82xhToobme+elXtpn/EoKLyY0ep3qlVnaHpJRSOWblFUEzYLsxZqcxJhWYBnSz8P1sMf6P+cw58jyhJPNJsaup3rSf3SEppdQlsTIRVAD2ZZqPzliWVU8RWSciM0WkooXx5Lqp6xby4banKeNO4ZN4J5W7vG13SEopdcnsblD2I1DFGNMQmAd8ml0hEeknIitEZEVsbGyeBnghP21byKsrh1LKbfjs4EEqdHsfAorZHZZSSl0yKxPBfiDzL/zwjGVnGGOOGmNSMmY/BJpmtyNjzCRjTKQxJrJUqVKWBHsp5u76jWcWDyE4LZgZB3ZStvkAqHKd3WEppdRlsTIRLAdqiEhVEfEDegOzMhcQkXKZZrsCmy2MJ1f8uutXnlg0FEkuxazjBwgrVRfaPm93WEopddkse2rIGJMuIgOBOYATmGKM2SgiLwMrjDGzgEEi0hVIB44B91sVT274YfsPvLB4BGmJlZjhEkqnnoQes8DH3+7QlFLqslnaoMwYMxuYnWXZiEzTzwDPWBlDbpm+dTqvLH2F9PjqDPdpTP3Db8CNo6BMPbtDU0qpK6Iti3Pgs42fMWbFGExCHVq6enBv3DCocj20eNTu0JRS6orZ/dRQvjdp3STGrBiDT1IEIcfv472AjxBxwK0TwaGHTylV8OmZ7AKMMYxfNZ53Vr9DsfQWJEb34ruItfgdWAY3jYXQAtXkQSmlLkhvDWXDGMPry1/ny81fUsm3LRs3t+eTTv6UW/Qm1OsODe+wO0SllMo1mgiycLldvLL0Fb7Z9g1Ni3Vj4dIWDGoVTuuND0NwSbj5TRCxO0yllMo1mggySXen8/zi5/l55890Cr+bb+fXp32dMgyRaRC7Be75DoJK2B2mUkrlKq0jyJDmSuOpRU/x886fubtWf+b8EUH1UkV4p/kJ5O+J0OxhuKqt3WEqpVSu0ysCIMWVwtCFQ1kUvYhBEcOYFlUFpyOVj3pVJ3BqOyhZE9qPtDtMpZSyhNcngsS0RAYtGMSyg8t4vvkIfvmrMnuOHuGLvs0JX/IkJMTAnV+BX5DdoSqllCW8+tbQqdRT9I/qz/JDy/nvdf9lx876LNway0vd6tEiYQFs+AZaD4fyje0OVSmlLOO1iSAuJY6H5j7E+tj1jGk1hpQTjZm0aCf3XlOZPrWd8PNQCG8GLYfYHapSSlnKK28NHUk6Qr95/dgTt4fxbccT7GrAwG+Xck21MF64uTZ82R1c6dDjA3B65SFSSnkRrzvLHUo4xENzH+Jw4mEmtJ9ApcBGdH13MeVCA3ivTxN8l0+CXYvglrehRDW7w1VKKct5VSLYH7+fB+c8yImUE7zf/n3qFG/E7R8sITnNxdSHmlM8YQdEjYRaN0GTe+0OVyml8oTXJILdcbvpO7cvSelJfHjjh9QLq8fAqavZeOAkH90XSY0wf/jwIfAv4rka0NbDSikv4TWJ4Pfo30lzpzGl4xRqlajFO/O38fO6gwzvXJu2tct4rgQOrYc7p0GI/cNhKqVUXvGaRHBv3XvpUq0LYYFh/LrhEG/M+4cejSvwcKtqsOcv+HMcNLkPanW2O1SllMpTXvP4qIgQFhjG5oMnGTp9DY0qhvJqjwZIyin4rh8UrwwdX7U7TKWUynNec0UAcCQ+hb6frqBIgA+T72lKgK8Tfn4G4qLhP3PAP8TuEJVSKs95zRVBarqbAV+s4kh8CpPvjaR00QDYNAvWfAHXD4OKzewOUSmlbOE1VwTv/raNZbuPMb53BA3DQ+HUYfhxMJSLgBuetjs8pZSyjdckggevq0alsGC6RVQAY2DWQEhLhB6TwOlrd3hKKWUbr0kExYJ8ua1puGdmxRTYNhc6j4FStewNTCmlbOY1dQRnHNkOc5/3DDJzdV+7o1FKKdtZmghEpJOIbBWR7SIy/CLleoqIEZFIK+PBle55VNTpB90mgMP78qBSSmVl2ZlQRJzABKAzUBe4U0TqZlOuCDAY+NuqWM74YyzsXwm3jIOi5S1/O6WUKgis/EncDNhujNlpjEkFpgHdsin3CvA6kGxhLBC9En7/HzTsBfW6W/pWSilVkFiZCCoA+zLNR2csO0NEmgAVjTE/WxiHx+H1EFoJbhpj+VsppVRBYttTQyLiAN4E7s9B2X5AP4BKlSpd3hs2vR8a3Qk+/pe3vVJKFVJWXhHsBypmmg/PWHZaEaA+sFBEdgMtgFnZVRgbYyYZYyKNMZGlSl1Bz6CaBJRS6jxWJoLlQA0RqSoifkBvYNbplcaYOGNMSWNMFWNMFWAp0NUYs8LCmJRSSmVhWSIwxqQDA4E5wGZgujFmo4i8LCJdrXpfpZRSl8bSOgJjzGxgdpZlIy5QtrWVsSillMqetqhSSikvp4lAKaW8nCYCpZTycpoIlFLKy4kxxu4YLomIxAJ77I7jCpUEjtgdRD6ix+MsPRbn0uNxris5HpWNMdk2xCpwiaAwEJEVxhhre1otQPR4nKXH4lx6PM5l1fHQW0NKKeXlNBEopZSX00Rgj0l2B5DP6PE4S4/FufR4nMuS46F1BEop5eX0ikAppbycJgKllPJymgiUUsrLaSLIh0QkWERWiEgXu2Oxm4jcKiKTReRrEbnR7njyWsZ34dOMY9DH7njs5u3fh6xy61yhiSAXicgUEYkRkQ1ZlncSka0isl1EhudgV08D062JMu/kxvEwxnxvjHkI6A/0sjLevHKJx6UHMDPjGBTKcTwu5XgUxu9DZpfxfyZXzhWaCHLXJ0CnzAtExAlMADoDdYE7RaSuiDQQkZ+yvEqLSAdgExCT18Fb4BOu8Hhk2vT5jO0Kg0/I4XHBM8TrvoxirjyMMS99Qs6Px2mF6fuQ2Sfk/P9Mrp0rbBu8vjAyxiwSkSpZFjcDthtjdgKIyDSgmzHmNeC8yzkRaQ0E4/kHTxKR2cYYt5VxWyWXjocAo4FfjDGrLA45T1zKcQGi8SSDNRTSH26XcjxEZDOF7PuQ2SV+N0LIpXOFJgLrVeDsLzrw/MdufqHCxpjnAETkfuBIQU0CF3FJxwN4DGgPFBOR6saY960MzkYXOi5vA++KyM3Aj3YEZpMLHQ9v+T5klu2xMMYMhNw5V2giyKeMMZ/YHUN+YIx5G8/J0CsZYxKAB+yOI7/w9u9DdnLjXFEoLzXzmf1AxUzz4RnLvJUej+zpcTmXHo+zLD8WmgistxyoISJVRcQP6A3MsjkmO+nxyJ4el3Pp8TjL8mOhiSAXichU4C+glohEi8iDxph0YCAwB9gMTDfGbLQzzryixyN7elzOpcfjLLuOhXY6p5RSXk6vCJRSystpIlBKKS+niUAppbycJgKllPJymgiUUsrLaSJQSikvp4lAeQ0RcYnIGhHZICIzRCQoH8TUWkSutTsO5d00EShvkmSMiTDG1AdS8fRp/69ExMo+uVoDl5QILI5HeSFtUKa8hojEG2NCMqb7Aw2BX/D0be8HHAX6GGMOi8hI4CqgGrAXeAb4HE+3vwADjTFLMroNfwk4ATTAM0jIemAwEAjcaozZISK3ZH2fjPVL8YwzEIunZ80twPtApYz3edwYszhrPMaYO3P36Chvpr8slNfJ+EXdGfgV+BNoYYwxItIXeAoYllG0LnCdMSYp4zZSB2NMsojUAKYCkRnlGgF1gGPATuBDY0wzERmM5+T+eHbvY4wZJiLvA/HGmLEZsX0FvGWM+VNEKuHpVqBO1nisOjbKO2kiUN4kUETWZEz/AXwE1AK+FpFyeH6t78pUflamk64vnnEBIvD8gq+ZqdxyY8xBABHZAczNWL4eaJMxHX6R98msPVDXMx4PAEVFJCSbeJTKNZoIlDdJMsZEZF4gIu8AbxpjZmXc5hmZaXVCpukhwGE8v/4dQHKmdSmZpt2Z5t2c/T92sffJzIHnyiHz/slIDAnZbqHUFdLKYuXtinG2b/f7/qXcwYxRoO4BnLn0PqeAIpnm5+K5nQRAxhWIUpbSRKC83UhghoisBI5cpNx7wH0ishaozaX/Or/Q+/wIdM94rPV6YBAQKSLrRGQTOXyySakroU8NKaWUl9MrAqWU8nKaCJRSystpIlBKKS+niUAppbycJgKllPJymgiUUsrLaSJQSikvp4lAKaW83P8BSTIRyM+7nzYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "feature = [train_feature, test_feature, valid_feature]\n",
    "original = [train[\"CATEGORY\"], test[\"CATEGORY\"], valid[\"CATEGORY\"]]\n",
    "\n",
    "params = np.logspace(-5, 4, 10, base=10)\n",
    "accu_list = []\n",
    "\n",
    "for i in range(3):\n",
    "    accu_list.append([buildModel_and_calcScore(p, feature[i], original[i]) for p in params])\n",
    "    plt.plot(params, accu_list[i], label=kind[i])\n",
    "\n",
    "plt.xscale(\"log\")\n",
    "plt.ylabel(\"Accuracy\")\n",
    "plt.xlabel(\"Paramater\")\n",
    "plt.legend()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ac490ad6",
   "metadata": {},
   "source": [
    "## 59. ハイパーパラメータの探索\n",
    "\n",
    "学習アルゴリズムや学習パラメータを変えながら，カテゴリ分類モデルを学習せよ．検証データ上の正解率が最も高くなる学習アルゴリズム・パラメータを求めよ．また，その学習アルゴリズム・パラメータを用いたときの評価データ上の正解率を求めよ．"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "36604706",
   "metadata": {},
   "outputs": [],
   "source": [
    "def for_allkind_buildModel_and_calcScore(param, solver=\"lbfgs\"):\n",
    "    score = []\n",
    "    \n",
    "    for i in range(3):\n",
    "        score.append(buildModel_and_calcScore(param, feature[i], original[i]))\n",
    "    \n",
    "    return score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "a9c7e175",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "solver :  newton-cg\n",
      "parameter :  100.0\n",
      "accuracy :  0.9124251497005988\n"
     ]
    }
   ],
   "source": [
    "solver = [\"newton-cg\", \"lbfgs\", \"liblinear\", \"sag\", \"saga\"]\n",
    "max_valid_acc = 0\n",
    "\n",
    "for s in solver:\n",
    "    accu_list = []\n",
    "    for p in params:\n",
    "        tmp = for_allkind_buildModel_and_calcScore(p, s)\n",
    "        if max_valid_acc < tmp[2]:\n",
    "            max_valid_acc = tmp[2]\n",
    "            max_solver = s\n",
    "            max_param = p\n",
    "\n",
    "print(\"solver : \", max_solver)\n",
    "print(\"parameter : \", max_param)\n",
    "print(\"accuracy : \", max_valid_acc)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}