week2_intro_to_pandas.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "week2_intro_to_pandas.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5eBw14EpD_ME",
        "colab_type": "text"
      },
      "source": [
        "# Big Data Colab Week 2: Intro to Pandas\n",
        "\n",
        "Welcome to week two of MAGIC's Big Data Colab. This module serves as an introduction to the powerful data science and manipulation library, pandas.\n",
        "\n",
        "Per their own documentation, pandas is a \"fast, powerful, flexible and easy to use open source data analysis and manipulation tool,\n",
        "built on top of the Python programming language.\"\n",
        "\n",
        "As mentioned last week, being an independent learner is a large part of being a success programmer. To solve problems, you have to be able to read documenation. The pandas library is a great place to learn that skill. The library is a massive open source project with lots of great [documentation](https://pandas.pydata.org/docs/). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "96hUEJtF8aDz",
        "colab_type": "text"
      },
      "source": [
        "## Numpy Package Overview\n",
        "\n",
        "Numpy provides a lot of the base funcitonality for packages you'll see later in this tutorial. In essence, it provides a lot of cool ways to manipulate lists and lists of lists. [Here](https://numpy.org/doc/stable/user/quickstart.html) is some documentation. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zepIZsxT9NCq",
        "colab_type": "text"
      },
      "source": [
        "## Data Structures in Numpy\n",
        "\n",
        "The base structure in the Numpy package in the numpy array. A numpy array is essentially a regular python list that is optimized and supports more complex operations."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "RMeoG6rO9izW",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "\n",
        "print(\"Array of nums\")\n",
        "a = np.arange(10)\n",
        "print(a)\n",
        "\n",
        "print(\"Array of arrays\")\n",
        "a = np.arange(15).reshape(3,5)\n",
        "print(a)\n",
        "\n",
        "print(\"Array from list\")\n",
        "a = np.array([2,3,4])\n",
        "\n",
        "print(a)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AVcMNYTt-EIC",
        "colab_type": "text"
      },
      "source": [
        "**2.1)** Create a numpy array that contains the numbers 0 to 99. Reshape the array so it contains 10 arrays of 10 numbers"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zlL3RWpJ-SCV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "##### YOUR CODE HERE ########"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WgONpD81-1eB",
        "colab_type": "text"
      },
      "source": [
        "## Operations on Numpy Arrays\n",
        "Numpy applies math operators on arrays elementwise. A new array is created and filled with the result. This makes our lives much easier because we don't have to use for loops to do math operations.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WxoQOKvr--3N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "A = np.array([1,2,3,4,5])\n",
        "B = np.array([6,7,8,9,10])\n",
        "\n",
        "print(A + B)\n",
        "print(A - B)\n",
        "print(A * B)\n",
        "print(A * 5)\n",
        "print(A < B)\n",
        "print(A == B)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GNevjP1KAM5P",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "A = [1,2,3,4,5]\n",
        "B = [6,7,8,9,10]\n",
        "sum_lst = []\n",
        "\n",
        "for i in range(len(A)):\n",
        "  sum = A[i] + B[i]\n",
        "  sum_lst.append(sum)\n",
        "\n",
        "print(sum_lst)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JYZh7QSx_mSH",
        "colab_type": "text"
      },
      "source": [
        "**2.2)** You're given a numpy array and want to multiply each element by 2 then subtract 5. Use numpy operations on the array to do this and print the new array. Can you do it in one line?"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bMxNnjJ5_3J4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "######## YOUR CODE HERE #########\n",
        "C = np.array([23,43,5,67,132])\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1dmBlKuTFVE5",
        "colab_type": "text"
      },
      "source": [
        "## Package Overview\n",
        "\n",
        "As mentioned above, the pandas documentation is really, really great. Because of that I'm not going to try to explain things better than the developers themselves. Any *italicized text* is a snippet taken directly from the documentation.\n",
        "\n",
        "*pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language.*\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mnY310iuGOwn",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd"
      ],
      "execution_count": 15,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YZKmxoi1GXOA",
        "colab_type": "text"
      },
      "source": [
        "## Data Structures in Pandas\n",
        "\n",
        "Last week you learned about simple data types like integers, strings, lists, and dictionaries. These data structures are provided to you by the python language. However when we're working with more complex datasets, we need more complex (they're not too complex don't worry :) ) data structures.\n",
        "\n",
        "Pandas provides you with two additional data structures. \n",
        "1.   [Series](https://pandas.pydata.org/pandas-docs/stable/reference/series.html)\n",
        "2.   [Dataframe](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.html)\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WF7zeUjqHokK",
        "colab_type": "text"
      },
      "source": [
        "**Series**: You can think of a series as the same as a list data structure where all elements of the list must be the same type (i.e. number, string, list, dict). A series is still indexed like a python list, starting at 0. At its core, a series is just a list optimized for pandas. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iIHcydHqE8Gk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        },
        "outputId": "69254ca2-a8f5-40c3-aa20-08dceb8a3063"
      },
      "source": [
        "s = pd.Series([1,2,4,5, np.nan , 6, 8]) # NaN?\n",
        "\n",
        "\n",
        "lst = ['hey', 'how', 'are', 'you']\n",
        "q = pd.Series(lst)"
      ],
      "execution_count": 17,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "0      1\n",
              "1      4\n",
              "2    hey\n",
              "3      6\n",
              "4      7\n",
              "dtype: object"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "gQJ7H1jgJA-Q",
        "colab_type": "text"
      },
      "source": [
        "**2.1)** Using your skills from last week and what you now know about series, concatenate lst with lst2 then make a series from the new longer list."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FIjifKarIz87",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "##### Your code here. #######\n",
        "lst = [1,2,3,4,5]\n",
        "lst2 = [6,7,8,9,10]\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "feGaC4qHJrix",
        "colab_type": "text"
      },
      "source": [
        "**Dataframe**: You can think of a dataframe as a group of series where each series has a label. It helps me to visualize a dataframe as an excel sheet where there are columns of data (i.e. series) with labels. The dataframe would be the entire block of information in the excel sheet. You'll notice below that like a list the rows of a dataframe are also indexed."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2D7L1-irKrg3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 235
        },
        "outputId": "a7e3db2f-f6b9-4ca3-983c-85a1822115bf"
      },
      "source": [
        "data_dict = {\n",
        "        'Names': ['Sam', 'Paige', 'Casey', 'Max', 'Alec', 'Collin'],\n",
        "        'Ages': [21, 22, 34, 45, 54, 66],\n",
        "        'Marital Status': ['Single', 'Married', 'Married', 'Single', 'Married', 'Married']\n",
        "        }\n",
        "\n",
        "df = pd.DataFrame(data_dict)\n",
        "\n",
        "df"
      ],
      "execution_count": 19,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>Names</th>\n",
              "      <th>Ages</th>\n",
              "      <th>Marital Status</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Sam</td>\n",
              "      <td>21</td>\n",
              "      <td>Single</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Paige</td>\n",
              "      <td>22</td>\n",
              "      <td>Married</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Casey</td>\n",
              "      <td>34</td>\n",
              "      <td>Married</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Max</td>\n",
              "      <td>45</td>\n",
              "      <td>Single</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Alec</td>\n",
              "      <td>54</td>\n",
              "      <td>Married</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>Collin</td>\n",
              "      <td>66</td>\n",
              "      <td>Married</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    Names  Ages Marital Status\n",
              "0     Sam    21         Single\n",
              "1   Paige    22        Married\n",
              "2   Casey    34        Married\n",
              "3     Max    45         Single\n",
              "4    Alec    54        Married\n",
              "5  Collin    66        Married"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q6qbupF0LccB",
        "colab_type": "text"
      },
      "source": [
        "**2.2)** In the space below, a dictionary is defined that maps a string to a list. Add three entries to the dictionary, a name label with a list of names, a favorite food label with a list of foods, and an animal label with a list of favorite animals. Then load the dictionary into a dataframe."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "s6gGSTrQL2qT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "favs_dict = {}"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "g8buh14JgSB5",
        "colab_type": "text"
      },
      "source": [
        "## Data Exploration in Pandas\n",
        "The first step in a data science workflow after loading your dataset is to get familiar with the data. Data exploration is important becuase it helps build intution for what the results of your queries and tests should be. Further down the line this could help you spot errors or bugs in your code. Pandas has lots of built in functionality to help you explore a dataset."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8SKV5S3OgMTT",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# sklearn is another data science library. Here I'm just using it to grab a dataset.\n",
        "from sklearn.datasets import load_iris\n",
        "iris = load_iris()\n",
        "iris_df = pd.DataFrame(data= np.c_[iris['data'], iris['target']],\n",
        "                     columns= iris['feature_names'] + ['target'])"
      ],
      "execution_count": 71,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rMssnIFIhwzy",
        "colab_type": "text"
      },
      "source": [
        "First lets consider three things:\n",
        "\n",
        "1. What does the data [look](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.head.html) like?\n",
        "2.   What are the [ranges](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.describe.html) of your data? Useful to check for outliers and get a feel for the dataset.\n",
        "3.   Are there any [missing or invalid values](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.isna.html) in the dataframe?\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WiJLrJ7hiOqC",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 669
        },
        "outputId": "9219ddbc-ac5c-4389-ebaf-74039fb05624"
      },
      "source": [
        "# 1) What does the data look like?\n",
        "iris_df.head(20)"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sepal length (cm)</th>\n",
              "      <th>sepal width (cm)</th>\n",
              "      <th>petal length (cm)</th>\n",
              "      <th>petal width (cm)</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>5.1</td>\n",
              "      <td>3.5</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4.9</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>4.7</td>\n",
              "      <td>3.2</td>\n",
              "      <td>1.3</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4.6</td>\n",
              "      <td>3.1</td>\n",
              "      <td>1.5</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5.0</td>\n",
              "      <td>3.6</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>5.4</td>\n",
              "      <td>3.9</td>\n",
              "      <td>1.7</td>\n",
              "      <td>0.4</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>4.6</td>\n",
              "      <td>3.4</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.3</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>5.0</td>\n",
              "      <td>3.4</td>\n",
              "      <td>1.5</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>4.4</td>\n",
              "      <td>2.9</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>4.9</td>\n",
              "      <td>3.1</td>\n",
              "      <td>1.5</td>\n",
              "      <td>0.1</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>5.4</td>\n",
              "      <td>3.7</td>\n",
              "      <td>1.5</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>4.8</td>\n",
              "      <td>3.4</td>\n",
              "      <td>1.6</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>4.8</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.1</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>4.3</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.1</td>\n",
              "      <td>0.1</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>5.8</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1.2</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>5.7</td>\n",
              "      <td>4.4</td>\n",
              "      <td>1.5</td>\n",
              "      <td>0.4</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>5.4</td>\n",
              "      <td>3.9</td>\n",
              "      <td>1.3</td>\n",
              "      <td>0.4</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>5.1</td>\n",
              "      <td>3.5</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.3</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>5.7</td>\n",
              "      <td>3.8</td>\n",
              "      <td>1.7</td>\n",
              "      <td>0.3</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>5.1</td>\n",
              "      <td>3.8</td>\n",
              "      <td>1.5</td>\n",
              "      <td>0.3</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    sepal length (cm)  sepal width (cm)  ...  petal width (cm)  target\n",
              "0                 5.1               3.5  ...               0.2     0.0\n",
              "1                 4.9               3.0  ...               0.2     0.0\n",
              "2                 4.7               3.2  ...               0.2     0.0\n",
              "3                 4.6               3.1  ...               0.2     0.0\n",
              "4                 5.0               3.6  ...               0.2     0.0\n",
              "5                 5.4               3.9  ...               0.4     0.0\n",
              "6                 4.6               3.4  ...               0.3     0.0\n",
              "7                 5.0               3.4  ...               0.2     0.0\n",
              "8                 4.4               2.9  ...               0.2     0.0\n",
              "9                 4.9               3.1  ...               0.1     0.0\n",
              "10                5.4               3.7  ...               0.2     0.0\n",
              "11                4.8               3.4  ...               0.2     0.0\n",
              "12                4.8               3.0  ...               0.1     0.0\n",
              "13                4.3               3.0  ...               0.1     0.0\n",
              "14                5.8               4.0  ...               0.2     0.0\n",
              "15                5.7               4.4  ...               0.4     0.0\n",
              "16                5.4               3.9  ...               0.4     0.0\n",
              "17                5.1               3.5  ...               0.3     0.0\n",
              "18                5.7               3.8  ...               0.3     0.0\n",
              "19                5.1               3.8  ...               0.3     0.0\n",
              "\n",
              "[20 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vfr0_BzFCZ8G",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "iris_df.tail(10)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rwPyq1ZDCbIw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "iris_df.size\n",
        "len(iris_df)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "QmvJI2P5kOog",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 2) What are the ranges of the data?\n",
        "iris_df.describe()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IOfDiLwropDl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# 3) Are there any missing or invalid values?\n",
        "iris_df.isna()\n",
        "iris_df.isna().any()"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kl4PIBYSpEeS",
        "colab_type": "text"
      },
      "source": [
        "**2.3)** In the space below, there is a dataframe that contains data regarding housing in California. Perform a exploratory analysis on the dataframe using the pandas methods shown above. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bQoywdQSd3-R",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "3a2687db-699b-427f-f1c7-a5457985977f"
      },
      "source": [
        "housing_df = pd.read_csv(\"https://storage.googleapis.com/mledu-datasets/california_housing_train.csv\", sep=\",\")\n",
        "housing_df.head()"
      ],
      "execution_count": 45,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>longitude</th>\n",
              "      <th>latitude</th>\n",
              "      <th>housing_median_age</th>\n",
              "      <th>total_rooms</th>\n",
              "      <th>total_bedrooms</th>\n",
              "      <th>population</th>\n",
              "      <th>households</th>\n",
              "      <th>median_income</th>\n",
              "      <th>median_house_value</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>-114.31</td>\n",
              "      <td>34.19</td>\n",
              "      <td>15.0</td>\n",
              "      <td>5612.0</td>\n",
              "      <td>1283.0</td>\n",
              "      <td>1015.0</td>\n",
              "      <td>472.0</td>\n",
              "      <td>1.4936</td>\n",
              "      <td>66900.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>-114.47</td>\n",
              "      <td>34.40</td>\n",
              "      <td>19.0</td>\n",
              "      <td>7650.0</td>\n",
              "      <td>1901.0</td>\n",
              "      <td>1129.0</td>\n",
              "      <td>463.0</td>\n",
              "      <td>1.8200</td>\n",
              "      <td>80100.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>-114.56</td>\n",
              "      <td>33.69</td>\n",
              "      <td>17.0</td>\n",
              "      <td>720.0</td>\n",
              "      <td>174.0</td>\n",
              "      <td>333.0</td>\n",
              "      <td>117.0</td>\n",
              "      <td>1.6509</td>\n",
              "      <td>85700.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>-114.57</td>\n",
              "      <td>33.64</td>\n",
              "      <td>14.0</td>\n",
              "      <td>1501.0</td>\n",
              "      <td>337.0</td>\n",
              "      <td>515.0</td>\n",
              "      <td>226.0</td>\n",
              "      <td>3.1917</td>\n",
              "      <td>73400.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>-114.57</td>\n",
              "      <td>33.57</td>\n",
              "      <td>20.0</td>\n",
              "      <td>1454.0</td>\n",
              "      <td>326.0</td>\n",
              "      <td>624.0</td>\n",
              "      <td>262.0</td>\n",
              "      <td>1.9250</td>\n",
              "      <td>65500.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   longitude  latitude  ...  median_income  median_house_value\n",
              "0    -114.31     34.19  ...         1.4936             66900.0\n",
              "1    -114.47     34.40  ...         1.8200             80100.0\n",
              "2    -114.56     33.69  ...         1.6509             85700.0\n",
              "3    -114.57     33.64  ...         3.1917             73400.0\n",
              "4    -114.57     33.57  ...         1.9250             65500.0\n",
              "\n",
              "[5 rows x 9 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 45
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q-ujCSCumSkl",
        "colab_type": "text"
      },
      "source": [
        "## Data selection in Pandas\n",
        "\n",
        "### Selecting Columns\n",
        "In a large dataframe, it may be useful to display only select columns. This can be done similarly to querying a dict for a certain value. To display certain columns, pass the names of the columns (as a list) to the dataframe.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ln-Ulwj4oH0c",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "iris_df[['sepal length (cm)', 'sepal width (cm)']]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AeZwVGYzoYKK",
        "colab_type": "text"
      },
      "source": [
        "**2.4)** Try out selecting certain columns from the housing dataframe."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rvXo4uxVogAN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "##### YOUR CODE HERE ########"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rHNeyQa-onto",
        "colab_type": "text"
      },
      "source": [
        "### Selecting Rows\n",
        "Selecting rows is slightly more complex than selecting columns. For example, you may want to select rows based on an index, a conditions, or even multiple conditions.\n",
        "\n",
        "To accomplish this, we rely on the `loc` and `iloc` functions. They are similar in name and easy to confuse but provide different functionality.\n",
        "\n",
        "`iloc`: Looks up rows by index (iloc...i-loc...i-locate...index-locate). Allows you to look up a specific row or a range of rows\n",
        "\n",
        "`loc`: Looks up rows based on condition(s).\n",
        "\n",
        "See examples below."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GTeVWD3HqLpj",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "####### iloc #########\n",
        "\n",
        "iris_df.iloc[45] # Select individual rows."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FXV28qTRDQSb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "iris_df.iloc[1:15] # Select range of rows.\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eOi9KY1KDQcL",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "iris_df.iloc[1:15, 1:3] # Select slice of rows and columns."
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pFKp8a0pq9QD",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "####### loc ###########\n",
        "iris_df.loc[iris_df['sepal length (cm)'] > 5.8]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4C3wdROZDVQu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "iris_df.loc[(iris_df['sepal length (cm)'] > 5.8) and (iris_df['petal length (cm)'] > 4.8)]"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cGsopatbBMin",
        "colab_type": "text"
      },
      "source": [
        "### Iterating Over a Dataframe\n",
        "Much like lists, sometimes its useful to iterate over a Dataframe (i.e. for x in df). However, as you'll see below there are a few tricks to speeding up dataframe iteration. [Here](https://towardsdatascience.com/how-to-make-your-pandas-loop-71-803-times-faster-805030df4f06) is an article with more information."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dsW5JwZhBawO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "3eea0422-cc0b-4eee-b54d-fbc55d640cf5"
      },
      "source": [
        "# Naive loop. Iterates over Dataframe object in python. \n",
        "# Doesnt take advantage of built in C code in pandas.\n",
        "# What is +=?\n",
        "\n",
        "%%timeit\n",
        "dummy = 0\n",
        "for row in range(0, len(iris_df)):\n",
        "  dummy+= iris_df.iloc[row]['petal width (cm)']"
      ],
      "execution_count": 41,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100 loops, best of 3: 15.2 ms per loop\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Azuso_RbCV63",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "93b19315-8fe4-4493-bdad-5b3b596f59fe"
      },
      "source": [
        "# Uses pandas functionality and optimized C code.\n",
        "# What is index, row?\n",
        "%%timeit\n",
        "dummy = 0\n",
        "for index, row in iris_df.iterrows():\n",
        "  dummy+=row['petal width (cm)']"
      ],
      "execution_count": 42,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "100 loops, best of 3: 9.81 ms per loop\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AMHJRRPEEX1b",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        },
        "outputId": "937364c5-5c97-41e1-8496-12da618960b1"
      },
      "source": [
        "# Vectorization. Just iterate and load into memory what we need.\n",
        "%%timeit\n",
        "dummy = 0\n",
        "dummy += iris_df['petal width (cm)']"
      ],
      "execution_count": 44,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "The slowest run took 4.03 times longer than the fastest. This could mean that an intermediate result is being cached.\n",
            "10000 loops, best of 3: 129 µs per loop\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T-REKYdLFKjQ",
        "colab_type": "text"
      },
      "source": [
        "**2.5)** The california housing dataframe is much larger than the iris dataframe. That means you'll be able to see an even greater difference in performance between the iteration techniques. Try two of the above techniques to sum the housing_median_age column and timeit!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "apyv6GiKCynu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "##### YOUR CODE HERE ######"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kMMOlIw1IyUg",
        "colab_type": "text"
      },
      "source": [
        "### Adding/Dropping Columns\n",
        "Of course, it may at some point become necessary to add or remove columns from our data frame.\n",
        "\n",
        "Dropping a dataframe is as easy as calling df.drop and supplying a column name."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xp7Bxx94I7vy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "print(iris_df.head())\n",
        "print(iris_df.drop([\"petal width (cm)\"], axis=1))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ohC5GfhYJdKl",
        "colab_type": "text"
      },
      "source": [
        "We can use a numpy function to add a column to a dataframe based on a condition."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "upfFFfGaJhbV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "iris_df[\"petalSize\"] = np.where(iris_df[\"petal width (cm)\"] > 1.3, \"Big\", \"Small\")\n",
        "print(iris_df)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9dOiCn9NGTco",
        "colab_type": "text"
      },
      "source": [
        "# Data Visualization\n",
        "To wrap up this week, we'll take a brief look at [matplotlib](https://matplotlib.org/), a popular libaray used to make both simple and complex visualizations from data. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Z5Mzq_6kIvsU",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        },
        "outputId": "1e644508-b6be-4803-a508-3a4f7ba21d63"
      },
      "source": [
        "import matplotlib.pyplot as plt\n",
        "iris_df.head()"
      ],
      "execution_count": 49,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sepal length (cm)</th>\n",
              "      <th>sepal width (cm)</th>\n",
              "      <th>petal length (cm)</th>\n",
              "      <th>petal width (cm)</th>\n",
              "      <th>target</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>5.1</td>\n",
              "      <td>3.5</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>4.9</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>4.7</td>\n",
              "      <td>3.2</td>\n",
              "      <td>1.3</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4.6</td>\n",
              "      <td>3.1</td>\n",
              "      <td>1.5</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5.0</td>\n",
              "      <td>3.6</td>\n",
              "      <td>1.4</td>\n",
              "      <td>0.2</td>\n",
              "      <td>0.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   sepal length (cm)  sepal width (cm)  ...  petal width (cm)  target\n",
              "0                5.1               3.5  ...               0.2     0.0\n",
              "1                4.9               3.0  ...               0.2     0.0\n",
              "2                4.7               3.2  ...               0.2     0.0\n",
              "3                4.6               3.1  ...               0.2     0.0\n",
              "4                5.0               3.6  ...               0.2     0.0\n",
              "\n",
              "[5 rows x 5 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 49
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yL_-JKOVImU_",
        "colab_type": "text"
      },
      "source": [
        "Once you understand how to use the pyplot (python plot) object, its easy to read the documentation and make graphs of any type. As an example, lets make a histogram to see the spread of sepal lengths in the iris_df."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jhzaLQMVJMNr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 350
        },
        "outputId": "54276680-91e9-432c-fd24-fdfc267e6e06"
      },
      "source": [
        "plt.hist(iris_df['sepal length (cm)'], bins=20)"
      ],
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(array([ 4.,  5.,  7., 16.,  9.,  5., 13., 14., 10.,  6., 10., 16.,  7.,\n",
              "        11.,  4.,  2.,  4.,  1.,  5.,  1.]),\n",
              " array([4.3 , 4.48, 4.66, 4.84, 5.02, 5.2 , 5.38, 5.56, 5.74, 5.92, 6.1 ,\n",
              "        6.28, 6.46, 6.64, 6.82, 7.  , 7.18, 7.36, 7.54, 7.72, 7.9 ]),\n",
              " <a list of 20 Patch objects>)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        },
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXIAAAD4CAYAAADxeG0DAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAPJUlEQVR4nO3df4xld13G8fdjl0pbGwruyI9uh2kMNAECaZ1gAUVggRSWdE3kj21SbREyilp+iDaLJpKYmKxKFBUD2bRLq5QFXAtWCtgGJI0JrOyWQn9skQpL2dKyWxqLgFI2fPxjbnG4zMy9c++Zufe7+34lk7n3nO/c8/Tb2WfOnLnnnFQVkqR2/cSkA0iSxmORS1LjLHJJapxFLkmNs8glqXGbNnJjmzdvrrm5uY3cpCQ17+DBgw9W1cxK6ze0yOfm5jhw4MBGblKSmpfkq6ut99CKJDXOIpekxlnkktQ4i1ySGmeRS1LjLHJJatzAIk+yJ8nRJHf0Lb8iyd1J7kzyZ+sXUZK0mmH2yK8BLlq6IMmLge3Ac6rqmcDbu48mSRrGwCKvqluAh/oWvx7YVVXf6405ug7ZJElDGPXMzqcDv5jkT4D/BX6vqj673MAkC8ACwOzs7Iiba9fczhtH/trDu7Z1mETryf/PmqRR/9i5CXgCcCHw+8AHk2S5gVW1u6rmq2p+ZmbFSwVIkkY0apEfAa6vRf8O/ADY3F0sSdKwRi3yDwMvBkjydOBU4MGuQkmShjfwGHmSvcCLgM1JjgBvA/YAe3pvSXwEuKy8i7MkTcTAIq+qS1ZYdWnHWSRJI/DMTklqnEUuSY2zyCWpcRa5JDXOIpekxlnkktQ4i1ySGmeRS1LjLHJJapxFLkmNs8glqXEWuSQ1ziKXpMZZ5JLUOItckhpnkUtS4wYWeZI9SY727gbUv+4tSSqJ9+uUpAkZZo/8GuCi/oVJzgFeDtzbcSZJ0hoMLPKqugV4aJlVfwlcCXivTkmaoJGOkSfZDtxXVZ/vOI8kaY0G3ny5X5LTgT9g8bDKMOMXgAWA2dnZtW5ODZrbeePIX3t417YOk0gnh1H2yH8WOBf4fJLDwBbg1iRPWm5wVe2uqvmqmp+ZmRk9qSRpWWveI6+q24GfefR5r8znq+rBDnNJkoY0zNsP9wKfBs5LciTJa9c/liRpWAP3yKvqkgHr5zpLI0laM8/slKTGWeSS1DiLXJIaZ5FLUuMscklqnEUuSY2zyCWpcRa5JDVuzafo68Q3zkWvJG0898glqXEWuSQ1ziKXpMZZ5JLUOItckhpnkUtS4yxySWqcRS5JjRvmVm97khxNcseSZX+e5O4kX0jyoSRnrW9MSdJKhtkjvwa4qG/ZzcCzqurZwH8Ab+04lyRpSAOLvKpuAR7qW3ZTVR3vPf0MsGUdskmShtDFMfJfBz620sokC0kOJDlw7NixDjYnSVpqrCJP8ofAceC6lcZU1e6qmq+q+ZmZmXE2J0laxshXP0xyOfAqYGtVVWeJJElrMlKRJ7kIuBL4par6breRJElrMczbD/cCnwbOS3IkyWuBdwJnAjcnuS3Ju9c5pyRpBQP3yKvqkmUWX70OWSRJI/DMTklqnEUuSY2zyCWpcRa5JDXOIpekxlnkktQ4i1ySGjfyKfqSJm9u540jf+3hXds6TKJJco9ckhpnkUtS4yxySWqcRS5JjbPIJalxFrkkNc4il6TGWeSS1DiLXJIaN8yt3vYkOZrkjiXLnpDk5iRf6n1+/PrGlCStZJg98muAi/qW7QQ+UVVPAz7Rey5JmoCBRV5VtwAP9S3eDlzbe3wt8Msd55IkDWnUi2Y9saru7z1+AHjiSgOTLAALALOzsyNuTieLSV0EapztSpM29h87q6qAWmX97qqar6r5mZmZcTcnSeozapF/I8mTAXqfj3YXSZK0FqMW+Q3AZb3HlwH/1E0cSdJaDfP2w73Ap4HzkhxJ8lpgF/CyJF8CXtp7LkmagIF/7KyqS1ZYtbXjLJKkEXhmpyQ1ziKXpMZZ5JLUOItckhpnkUtS4yxySWqcRS5JjRv1olnaAJO6gJSktrhHLkmNs8glqXEWuSQ1ziKXpMZZ5JLUOItckhpnkUtS4yxySWqcRS5JjRuryJO8OcmdSe5IsjfJY7sKJkkazshFnuRs4A3AfFU9CzgF2NFVMEnScMY9tLIJOC3JJuB04OvjR5IkrcXIRV5V9wFvB+4F7gcerqqb+sclWUhyIMmBY8eOjZ5UkrSscQ6tPB7YDpwLPAU4I8ml/eOqandVzVfV/MzMzOhJJUnLGufQykuBr1TVsar6PnA98PxuYkmShjVOkd8LXJjk9CQBtgKHuoklSRrWOMfI9wP7gFuB23uvtbujXJKkIY11h6Cqehvwto6ySJJG4JmdktQ4i1ySGmeRS1LjLHJJapxFLkmNs8glqXEWuSQ1bqz3kZ8s5nbeOOkIOoH5/aVxuUcuSY2zyCWpcRa5JDXOIpekxlnkktQ4i1ySGmeRS1LjLHJJapxFLkmNG6vIk5yVZF+Su5McSvK8roJJkoYz7in6fwV8vKpeneRU4PQOMkmS1mDkIk/yOOCFwOUAVfUI8Eg3sSRJwxpnj/xc4BjwniTPAQ4Cb6yq7ywdlGQBWACYnZ0dY3PS6rz4lE5W4xwj3wRcALyrqs4HvgPs7B9UVburar6q5mdmZsbYnCRpOeMU+RHgSFXt7z3fx2KxS5I20MhFXlUPAF9Lcl5v0Vbgrk5SSZKGNu67Vq4Aruu9Y+XLwGvGjyRJWouxiryqbgPmO8oiSRqBZ3ZKUuMscklqnEUuSY2zyCWpcRa5JDXOIpekxlnkktS4cU8IasLJeDGlk/G/WRpknH8Xh3dt6zBJt9wjl6TGWeSS1DiLXJIaZ5FLUuMscklqnEUuSY2zyCWpcRa5JDXOIpekxo1d5ElOSfK5JB/pIpAkaW262CN/I3Cog9eRJI1grCJPsgXYBlzVTRxJ0lqNe9GsdwBXAmeuNCDJArAAMDs7O/KGvAiU1K1JXkDqRL141aSMvEee5FXA0ao6uNq4qtpdVfNVNT8zMzPq5iRJKxjn0MoLgIuTHAbeD7wkyXs7SSVJGtrIRV5Vb62qLVU1B+wAPllVl3aWTJI0FN9HLkmN6+QOQVX1KeBTXbyWJGlt3COXpMZZ5JLUOItckhpnkUtS4yxySWqcRS5JjbPIJalxFrkkNc4il6TGWeSS1DiLXJIaZ5FLUuMscklqnEUuSY2zyCWpcRa5JDXOIpekxo1c5EnOSfKvSe5KcmeSN3YZTJI0nHFu9XYceEtV3ZrkTOBgkpur6q6OskmShjDyHnlV3V9Vt/Ye/zdwCDi7q2CSpOF0cvPlJHPA+cD+ZdYtAAsAs7OzXWxO0oTN7bxx0hG0xNh/7EzyU8A/Am+qqm/1r6+q3VU1X1XzMzMz425OktRnrCJP8hgWS/y6qrq+m0iSpLUY510rAa4GDlXVX3QXSZK0FuPskb8A+FXgJUlu6328sqNckqQhjfzHzqr6NyAdZpEkjcAzOyWpcRa5JDXOIpekxlnkktQ4i1ySGmeRS1LjLHJJalwnF82SpI0yqQt2jbvdw7u2dZTkx7lHLkmNs8glqXEWuSQ1ziKXpMZZ5JLUOItckhpnkUtS4yxySWqcRS5JjRv35ssXJfliknuS7OwqlCRpeOPcfPkU4G+BVwDPAC5J8oyugkmShjPOHvlzgXuq6stV9QjwfmB7N7EkScMa56JZZwNfW/L8CPDz/YOSLAALvaffTvLFMbY5rs3AgxPc/rBayQntZDVnt1rJCVOSNX86cMhqOZ+62heu+9UPq2o3sHu9tzOMJAeqan7SOQZpJSe0k9Wc3WolJ7STdZyc4xxauQ84Z8nzLb1lkqQNNE6RfxZ4WpJzk5wK7ABu6CaWJGlYIx9aqarjSX4H+BfgFGBPVd3ZWbL1MRWHeIbQSk5oJ6s5u9VKTmgn68g5U1VdBpEkbTDP7JSkxlnkktS4E7bIk5yS5HNJPrLMusuTHEtyW+/jdRPKeDjJ7b0MB5ZZnyR/3bsEwheSXDClOV+U5OEl8/lHk8jZy3JWkn1J7k5yKMnz+tZPy5wOyjnxOU1y3pLt35bkW0ne1DdmWuZzmKwTn9NejjcnuTPJHUn2Jnls3/qfTPKB3pzuTzI38EWr6oT8AH4XeB/wkWXWXQ68cwoyHgY2r7L+lcDHgAAXAvunNOeLlpvnCWW9Fnhd7/GpwFlTOqeDck7NnPbynAI8ADx1GudzyKwTn1MWT6T8CnBa7/kHgcv7xvwW8O7e4x3ABwa97gm5R55kC7ANuGrSWca0Hfi7WvQZ4KwkT550qGmV5HHAC4GrAarqkar6r75hE5/TIXNOm63Af1bVV/uWT3w+l7FS1mmxCTgtySbgdODrfeu3s/iDHmAfsDVJVnvBE7LIgXcAVwI/WGXMr/R+FdyX5JxVxq2nAm5KcrB3KYN+y10G4ewNSfajBuUEeF6Szyf5WJJnbmS4Jc4FjgHv6R1WuyrJGX1jpmFOh8kJ0zGnj9oB7F1m+TTMZ7+VssKE57Sq7gPeDtwL3A88XFU39Q374ZxW1XHgYeCnV3vdE67Ik7wKOFpVB1cZ9s/AXFU9G7iZ///pt9F+oaouYPEKkr+d5IUTyjHIoJy3svhr7HOAvwE+vNEBezYBFwDvqqrzge8A03h55WFyTsuc0jvh72LgHyaVYVgDsk58TpM8nsU97nOBpwBnJLl03Nc94YoceAFwcZLDLF6R8SVJ3rt0QFV9s6q+13t6FfBzGxvxhznu630+CnyIxStKLjUVl0EYlLOqvlVV3+49/ijwmCSbNzoni3uDR6pqf+/5PhYLc6lpmNOBOadoTmHxB/itVfWNZdZNw3wutWLWKZnTlwJfqapjVfV94Hrg+X1jfjinvcMvjwO+udqLnnBFXlVvraotVTXH4q9Yn6yqH/mJ13cM72Lg0AZGfDTDGUnOfPQx8HLgjr5hNwC/1ntnwIUs/hp2/7TlTPKkR4/hJXkui99Xq37jrYeqegD4WpLzeou2Anf1DZv4nA6Tc1rmtOcSVj5UMfH57LNi1imZ03uBC5Oc3suylR/vnxuAy3qPX81ih6165ua6X/1wWiT5Y+BAVd0AvCHJxcBx4CEW38Wy0Z4IfKj3fbUJeF9VfTzJbwJU1buBj7L4roB7gO8Cr5nSnK8GXp/kOPA/wI5B33jr6Argut6v2F8GXjOFczpMzqmY094P75cBv7Fk2TTO5zBZJz6nVbU/yT4WD/McBz4H7O7rp6uBv09yD4v9tGPQ63qKviQ17oQ7tCJJJxuLXJIaZ5FLUuMscklqnEUuSY2zyCWpcRa5JDXu/wDssfJp2hdqmQAAAABJRU5ErkJggg==\n",
            "text/plain": [
              "<Figure size 432x288 with 1 Axes>"
            ]
          },
          "metadata": {
            "tags": [],
            "needs_background": "light"
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tNf17UDPJ95T",
        "colab_type": "text"
      },
      "source": [
        "**2.6)** Lets combine our knowledge of plotting with some of the things we learned above. You're curious about the median house value in high density areas of California. Specifically, areas with a population greater than 1000. Write code that creates a histogram of the median house value of for blocks with a population greater than 1000.\n",
        "\n",
        "First, complete the task using row selection functions (iloc or loc).\n",
        "\n",
        "Second, complete the task by iterating over the dataframe and adding the median house values to a list (if the population is greater than 1000).\n",
        "\n",
        "You should construct the same histogram using both methods. Run a timeit to see which version is faster."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Uq1xKT3mKHJI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "### Create histogram using row selection here.\n",
        "\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_FLlLn6oMHlw",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "### Create histogram using iteration here.\n"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Bk1lA4NuFsSm",
        "colab_type": "text"
      },
      "source": [
        "Now that you know how to make a histogram manually, I'll show you how to make it using a built in pandas method.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WCT78H_UF0xf",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "iris_df.hist(figsize=(20,15))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_PBwrmlTGCWz",
        "colab_type": "text"
      },
      "source": [
        "**2.7)** Create a histogram below for all the columns in the california housing dataframe. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rf4YXLTcGBbA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#### YOUR CODE HERE ######\n",
        "housing_df.hist(figsize=(20,15))"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pE-V_frkGi1N",
        "colab_type": "text"
      },
      "source": [
        "Below is a more advanced sequence of visualizations from matlplotlib on the california housing dataframe."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4FutFxEfGiJh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "housing_df.plot(kind=\"scatter\", x=\"longitude\", y=\"latitude\")"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GR7qtMi-G1QE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "housing_df.plot(kind=\"scatter\", x=\"longitude\", y=\"latitude\", alpha=0.1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oFq1TQ8uG6bb",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "housing_df.plot(kind=\"scatter\", x=\"longitude\", y=\"latitude\", alpha=0.1,figsize=(20,15) ,\n",
        "                s=housing_df[\"population\"]/100, label=\"population\", c = \"median_house_value\", \n",
        "                cmap=plt.get_cmap(\"jet\"), colorbar=True)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "reaexxb7KmjH",
        "colab_type": "text"
      },
      "source": [
        "## Exploring Correlation\n",
        "\n",
        "When performing exploratory analysis of a dataset, we want to know how certain variable may impact other variables. This is especially important in ML when we're predicting a target value. Pandas provides built in functions that allows you to visualize the relationships between variables.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IwyD2DvfK9VF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from pandas.plotting import scatter_matrix\n",
        "\n",
        "atts = [\"sepal length (cm)\", \"sepal width (cm)\", \"petal width (cm)\", \"target\"]\n",
        "scatter_matrix(iris_df[atts], figsize=(12,8), alpha=1)"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j7_VA1MmLxov",
        "colab_type": "text"
      },
      "source": [
        "**2.8)** Choose some of the attributes from the California housing dataframe that you think may be useful to predict the median housing price. Create a scatter matrix to see which attributes are most correlated with the prices."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XPH4b3viL_2N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#### YOUR CODE HERE ######"
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_bHuaKy7LD4h",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "R3ZkZekCyCVW",
        "colab_type": "text"
      },
      "source": [
        "## Project\n",
        "\n",
        "Last week we saw a series of challenge exercies. This week, instead of exercises, you're going to work through a more complex notebook that does some real COVID-19 analysis. You won't have to write any code, but try your best to understand what is going on. Take some time to follow the author's workflow. Its very similar to the examples you just saw above.\n",
        "\n",
        "There are certainly going to be functions in the notebook that you haven't seen before. For most people, its impossible to be familiar with the entire pandas library. If you see a function call you don't recognize, look it up in the [documentation](https://pandas.pydata.org/pandas-docs/stable/index.html). If you're still stuck, don't hesistate to drop questions in the slack channel during the week!\n",
        "\n",
        "[Here](https://colab.research.google.com/github/hugobowne/COVID-19-EDA-tutorial/blob/master/notebooks/1-COVID-19-EDA-solution.ipynb#scrollTo=PttQGSbOp8E7) is the notebook.\n",
        "\n",
        "If you'd like more Dataframe practice, [here](https://www.w3resource.com/python-exercises/pandas/index-dataframe.php) is a list of problems."
      ]
    }
  ]
}