Data_Collection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Collection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I have used `pushshift` for data collecion as we can get 1000 articles in a single API call and we don't need any credentials to set up. Using this ~4 lakh articles were scraped, from Jan. 10, 2018 to April 10, 2020. It is to be noted that `pushshift` do not offer `upvotes`, and `downvotes` in their API. It only give `score` (which is upvotes-downvotes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Imports\n",
    "\n",
    "import pandas as pd\n",
    "import requests\n",
    "import json\n",
    "import csv\n",
    "import time\n",
    "import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `getPushshiftData` function is used to create the URL and get the 1000 submissions between the entered timestamps"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getPushshiftData(after, before, sub):\n",
    "    url = 'https://api.pushshift.io/reddit/search/submission/?size=1000&after=' + \\\n",
    "        str(after)+'&before='+str(before)+'&subreddit='+str(sub)\n",
    "    print (url)\n",
    "    r = requests.get(url)\n",
    "    data = json.loads(r.text)\n",
    "    return data['data']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `writeSubData` function is used to write the required details from the scraped submissions to a CSV file. We have scrpaed 16 fields from the articles even though we will be using only 'flair', 'title' and 'selftext' for classification. This is done keeping EDA step in mind. I believe we can get some good insights about data by looking at the other fields. For example we can do virality analysis of the subreddit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def writeSubData(subm):\n",
    "    #print(subm)\n",
    "    subData = []  # list to store data points\n",
    "    title = subm['title']\n",
    "    url = subm['url']\n",
    "    try:\n",
    "        flair = subm['link_flair_text']\n",
    "    except KeyError:\n",
    "        flair = \"NaN\"\n",
    "    author = subm['author']\n",
    "    stickied = subm['stickied']\n",
    "    pinned = subm['pinned']\n",
    "    over_18 = subm['over_18']\n",
    "    try:\n",
    "        selftext = subm['selftext']\n",
    "    except KeyError:\n",
    "        selftext = 'Nan'\n",
    "    \n",
    "    spoiler = subm['spoiler']\n",
    "    sub_id = subm['id']\n",
    "    score = subm['score']\n",
    "    num_crossposts = subm['num_crossposts']\n",
    "    is_video = subm['is_video']\n",
    "    created = datetime.datetime.fromtimestamp(subm['created_utc'])  # 1520561700.0\n",
    "    numComms = subm['num_comments']\n",
    "    permalink = subm['permalink']\n",
    "\n",
    "    #new_line= (sub_id,title,url,author,score,created,numComms,permalink,flair)\n",
    "\n",
    "    with open('data_final.csv', 'a+', newline='') as csvfile:\n",
    "        fieldnames = ['sub_id', 'title', 'url', 'author', 'score', 'created', 'numComms', 'permalink', 'flair',\n",
    "                      'stickied', 'pinned', 'over_18', 'selftext', 'spoiler', 'num_crossposts', 'is_video']\n",
    "        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)\n",
    "\n",
    "        writer.writeheader()\n",
    "        writer.writerow({'sub_id': sub_id, 'title': title, 'url': url, 'author': author, 'score': score, 'created': created, 'numComms': numComms, 'permalink': permalink, 'flair': flair, 'stickied': stickied,\n",
    "                         'pinned': pinned,'over_18': over_18, 'selftext': selftext, 'spoiler': spoiler, 'num_crossposts': num_crossposts, 'is_video': is_video})\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### This codeblock start scraping and writing to csv file, I will not run this here as I have already collected the data at this point"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "begin_time = datetime.datetime.now()\n",
    "sub = 'india'\n",
    "before = \"1586524763\"  # April 10, 2020 1:19:23 PM\n",
    "after = \"1515568144\"  # Wednesday, January 10, 2018 7:09:04 PM\n",
    "subCount = 0\n",
    "subStats = {}\n",
    "\n",
    "data = getPushshiftData(after, before, sub)\n",
    "\n",
    "while len(data) > 0:\n",
    "    for submission in data:\n",
    "        #print (submission)\n",
    "        writeSubData(submission)\n",
    "        subCount += 1\n",
    "\n",
    "    # print(len(data))\n",
    "    # print(str(datetime.datetime.fromtimestamp(data[-1]['created_utc'])))\n",
    "    after = data[-1]['created_utc']\n",
    "    data = getPushshiftData(after, before, sub)\n",
    "\n",
    "print('finished scraping, total time taken:')\n",
    "\n",
    "print(datetime.datetime.now() - begin_time)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}