dataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Dataset\n",
    "\n",
    "Download the dataset from InsideAirbnb for New York City and build an inverted index using Whoosh.\n",
    "The function `populate_index` takes care of all. It requires the directory name where the index will be stored, usually it's `index` and\n",
    "it's located in the current working directory.\n",
    "\n",
    "`.gitignore` tells git to ignore the top level subdirectory `index`, where the inverted index for this project should be stored.\n",
    "\n",
    "Once you download and generate the II, you don't need to rebuild it anymore and you can directly load it using Whoosh open_in."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from placerank.dataset import populate_index"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download dataset source and build an index. Currently `[\"id\", \"name\", \"description\", \"neighborhood_overview\"]` will be indexed, to add keys to the indexing edit the II schema and add keys to the function `placerank.dataset.DocumentLogicView`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "URL = \"http://data.insideairbnb.com/united-states/ma/cambridge/2023-12-26/data/listings.csv.gz\"\n",
    "LOCAL_COPY = \"datasets/cambridge_listings.csv\"\n",
    "INDEX_DIR = \"index\"\n",
    "\n",
    "populate_index(INDEX_DIR, LOCAL_COPY, URL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Search\n",
    "\n",
    "To search in the inverted index, we have to open it first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "from whoosh.index import open_dir\n",
    "\n",
    "ix = open_dir(\"index\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Two additional objects are required to perform queries: a `Searcher` and a `QueryParser`. Their names are pretty self-explanatory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from whoosh.qparser import QueryParser\n",
    "\n",
    "parser = QueryParser(\"neighborhood_overview\", ix.schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This query parser will search for terms in the \"neighborhood_overview\" field only. Then display the results of the query."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "('3678159', 'Rental unit in Cambridge · ★4.82 · 1 bedroom · 1 bed · 1 shared bath')\n",
      "('53574737', 'Rental unit in Cambridge · ★4.92 · 2 bedrooms · 2 beds · 1.5 baths')\n",
      "('918749466220965200', 'Rental unit in Cambridge · ★4.88 · 3 bedrooms · 4 beds · 2.5 baths')\n",
      "('39896724', 'Rental unit in Cambridge · ★4.87 · 3 bedrooms · 3 beds · 1 bath')\n",
      "('6363476', 'Rental unit in Cambridge · ★4.99 · 2 bedrooms · 3 beds · 1 bath')\n",
      "('30428472', 'Home in Cambridge · ★4.92 · 1 bedroom · 2 beds · 1 bath')\n"
     ]
    }
   ],
   "source": [
    "UIN = \"cozy\"\n",
    "\n",
    "query = parser.parse(UIN)\n",
    "with ix.searcher() as searcher:\n",
    "    results = searcher.search(query)\n",
    "    print(*[(hit.get(\"id\"), hit.get(\"name\")) for hit in results], sep='\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': '53574737',\n",
       " 'name': 'Rental unit in Cambridge · ★4.92 · 2 bedrooms · 2 beds · 1.5 baths',\n",
       " 'room_type': 'Entire home/apt',\n",
       " 'description': '',\n",
       " 'neighborhood_overview': 'Just around the corner from Harvard University, Mt Auburn Hospital, restaurants, bars, and coffee shops, with great access to public transportation; This cozy basement apartment is fully furnished with comfortable, modern accents. The Unit is located on Mt Auburn Street Between Longfellow & Channing. This is in a safe, friendly, clean, and vibrant neighborhood. Minutes walk from the Charles River, Harvard Sq., Redline, Harvard University, Mt Auburn Hospital, and Huron Village.'}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from placerank.dataset import load_page\n",
    "\n",
    "ID = \"53574737\"\n",
    "\n",
    "load_page(LOCAL_COPY, ID)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sentiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rental unit in New York · ★4.71 · 1 bedroom · 1 bed · 1 bath\n",
      "Rental unit in New York · 1 bedroom · 3 beds · 1 bath\n",
      "Home in Brooklyn · ★4.67 · 1 bedroom · 1 bed · 1 shared bath\n",
      "Serviced apartment in New York · ★4.81 · 3 bedrooms · 4 beds · 2 baths\n",
      "Rental unit in Staten Island  · ★4.88 · 1 bedroom · 1 bed · 1.5 shared baths\n",
      "[[{'label': 'admiration', 'score': 0.991719126701355}, {'label': 'neutral', 'score': 0.5229455828666687}]]\n"
     ]
    }
   ],
   "source": [
    "from sentimentModule.sentiment import GoEmotionsClassifier\n",
    "classifier = GoEmotionsClassifier()\n",
    "\n",
    "UIN = \"beautiful, quiet and peaceful apartment\"\n",
    "\n",
    "query = parser.parse(UIN)\n",
    "with ix.searcher() as searcher:\n",
    "    results = searcher.search(query)\n",
    "    sentiment = classifier.classify_texts(str(query))\n",
    "    print(*[hit.get(\"name\", sentiment) for hit in results], sentiment, sep='\\n')"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}