Recommender.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Testing the Viability of a Song Recommender\n",
    "\n",
    "<hr>\n",
    "\n",
    "Using the various attributes we were able to pull, we would like to see if it is possible to create a song recommender. It probably will not work too accurately for two reasons:\n",
    "\n",
    " - The Data\n",
    "      - We have one major hole in our data, and that is genre. Sadly genre wasn't a track feature in the API, it was only tied to artists, so we weren't able to collect it.\n",
    " - The Algorithm\n",
    "      - One algorithm alone isn't near enough to make an effective recommendation system\n",
    "      - in 2017 netflix had a recommendation algorithm challenge\n",
    "          - The winners used a combination of 107 different algorithms to create an effective system\n",
    "      - So even if ours was slightly effective, barring the previously stated genre issue, I would call that a win\n",
    "      \n",
    "## Outcome (Spoilers)\n",
    "<hr>\n",
    "I took an approach similar to how one might test document similarity using scikit's cosine_similarity function, and I was unsure that applying it to this problem would actually work, but...\n",
    "\n",
    "It worked suprisingly well, the matches on the top100 set seemed better, and I have hypotheses on why that might be the case, but I won't speculate.\n",
    "\n",
    "For the larger set, it can be hit or miss, but it seems like around half of the songs that it recommends can \"feel\" similar. It definitely outperformed my expectations without having a genre or other information that could guide the output though."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "### This block grabs the dataframes for you ###\n",
    "### This block doesn't need any modification ###\n",
    "\n",
    "from sqlalchemy import create_engine\n",
    "from pandas.io import sql\n",
    "import pandas as pd\n",
    "\n",
    "hostname=''\n",
    "dbname=''\n",
    "uname=''\n",
    "pwd=''\n",
    "\n",
    "engine = create_engine(\"mysql+pymysql://{user}:{pw}@{host}/{db}\".format(host=hostname, db=dbname, user=uname, pw=pwd))\n",
    "\n",
    "top100 = pd.read_sql('SELECT * FROM top100', con=engine)\n",
    "allSongs = pd.read_sql('SELECT * FROM final_Table', con=engine)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----- Top 100 Songs -----\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>rank</th>\n",
       "      <th>artist</th>\n",
       "      <th>title</th>\n",
       "      <th>album</th>\n",
       "      <th>release_date</th>\n",
       "      <th>length</th>\n",
       "      <th>popularity</th>\n",
       "      <th>danceability</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>energy</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>liveness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>tempo</th>\n",
       "      <th>time_signature</th>\n",
       "      <th>keys</th>\n",
       "      <th>valence</th>\n",
       "      <th>ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1990</td>\n",
       "      <td>1</td>\n",
       "      <td>Wilson Phillips</td>\n",
       "      <td>Hold On</td>\n",
       "      <td>Wilson Phillips</td>\n",
       "      <td>1990-01-01</td>\n",
       "      <td>266866</td>\n",
       "      <td>68</td>\n",
       "      <td>0.679</td>\n",
       "      <td>0.4000</td>\n",
       "      <td>0.657</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0497</td>\n",
       "      <td>-9.897</td>\n",
       "      <td>0.0255</td>\n",
       "      <td>97.800</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0.546</td>\n",
       "      <td>4VZDv8sASBS8UruUBGTFdk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1990</td>\n",
       "      <td>2</td>\n",
       "      <td>Roxette</td>\n",
       "      <td>It Must Have Been Love</td>\n",
       "      <td>It Must Have Been Love</td>\n",
       "      <td>1990-05-20</td>\n",
       "      <td>258786</td>\n",
       "      <td>76</td>\n",
       "      <td>0.520</td>\n",
       "      <td>0.3400</td>\n",
       "      <td>0.652</td>\n",
       "      <td>0.000055</td>\n",
       "      <td>0.2560</td>\n",
       "      <td>-6.655</td>\n",
       "      <td>0.0274</td>\n",
       "      <td>80.609</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0.722</td>\n",
       "      <td>6kvoHl80mfCVTv7XnZkjQn</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1990</td>\n",
       "      <td>3</td>\n",
       "      <td>Sinéad O'Connor</td>\n",
       "      <td>Nothing Compares 2 U</td>\n",
       "      <td>I Do Not Want What I Haven't Got</td>\n",
       "      <td>1990-07-01</td>\n",
       "      <td>280040</td>\n",
       "      <td>72</td>\n",
       "      <td>0.511</td>\n",
       "      <td>0.0425</td>\n",
       "      <td>0.574</td>\n",
       "      <td>0.000023</td>\n",
       "      <td>0.1050</td>\n",
       "      <td>-7.016</td>\n",
       "      <td>0.0273</td>\n",
       "      <td>119.917</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0.161</td>\n",
       "      <td>3nvuPQTw2zuFAVuLsC9IYQ</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   year  rank           artist                   title  \\\n",
       "0  1990     1  Wilson Phillips                 Hold On   \n",
       "1  1990     2          Roxette  It Must Have Been Love   \n",
       "2  1990     3  Sinéad O'Connor    Nothing Compares 2 U   \n",
       "\n",
       "                              album release_date  length  popularity  \\\n",
       "0                   Wilson Phillips   1990-01-01  266866          68   \n",
       "1            It Must Have Been Love   1990-05-20  258786          76   \n",
       "2  I Do Not Want What I Haven't Got   1990-07-01  280040          72   \n",
       "\n",
       "   danceability  acousticness  energy  instrumentalness  liveness  loudness  \\\n",
       "0         0.679        0.4000   0.657          0.000000    0.0497    -9.897   \n",
       "1         0.520        0.3400   0.652          0.000055    0.2560    -6.655   \n",
       "2         0.511        0.0425   0.574          0.000023    0.1050    -7.016   \n",
       "\n",
       "   speechiness    tempo  time_signature  keys  valence                     ids  \n",
       "0       0.0255   97.800               4     5    0.546  4VZDv8sASBS8UruUBGTFdk  \n",
       "1       0.0274   80.609               4     5    0.722  6kvoHl80mfCVTv7XnZkjQn  \n",
       "2       0.0273  119.917               4     5    0.161  3nvuPQTw2zuFAVuLsC9IYQ  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----- All Songs -----\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ids</th>\n",
       "      <th>name</th>\n",
       "      <th>album</th>\n",
       "      <th>artist</th>\n",
       "      <th>release_date</th>\n",
       "      <th>length</th>\n",
       "      <th>popularity</th>\n",
       "      <th>danceability</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>energy</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>liveness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>tempo</th>\n",
       "      <th>time_signature</th>\n",
       "      <th>keys</th>\n",
       "      <th>valence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2LKBDF6u2QNrzlUPzvpNAS</td>\n",
       "      <td>Tellin' Me Lies</td>\n",
       "      <td>Nature Of The Beast</td>\n",
       "      <td>April Wine</td>\n",
       "      <td>1981</td>\n",
       "      <td>3.025550</td>\n",
       "      <td>23</td>\n",
       "      <td>0.706</td>\n",
       "      <td>0.0543</td>\n",
       "      <td>0.637</td>\n",
       "      <td>0.000046</td>\n",
       "      <td>0.0616</td>\n",
       "      <td>-11.825</td>\n",
       "      <td>0.0340</td>\n",
       "      <td>137.752</td>\n",
       "      <td>3.0</td>\n",
       "      <td>9</td>\n",
       "      <td>0.965</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2Y0HDXWfE6KifNzo3GEScQ</td>\n",
       "      <td>Dancer</td>\n",
       "      <td>Hot Space</td>\n",
       "      <td>Queen</td>\n",
       "      <td>1982-05-03</td>\n",
       "      <td>3.818883</td>\n",
       "      <td>25</td>\n",
       "      <td>0.745</td>\n",
       "      <td>0.0435</td>\n",
       "      <td>0.336</td>\n",
       "      <td>0.000009</td>\n",
       "      <td>0.0766</td>\n",
       "      <td>-11.970</td>\n",
       "      <td>0.0398</td>\n",
       "      <td>100.093</td>\n",
       "      <td>4.0</td>\n",
       "      <td>9</td>\n",
       "      <td>0.625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2lRETJsdBygk2oVWbPpSRV</td>\n",
       "      <td>In Your Eyes</td>\n",
       "      <td>Emotions In Motion</td>\n",
       "      <td>Billy Squier</td>\n",
       "      <td>1982</td>\n",
       "      <td>3.777333</td>\n",
       "      <td>25</td>\n",
       "      <td>0.526</td>\n",
       "      <td>0.2200</td>\n",
       "      <td>0.558</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.0589</td>\n",
       "      <td>-6.918</td>\n",
       "      <td>0.0291</td>\n",
       "      <td>137.122</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.566</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                      ids             name                album        artist  \\\n",
       "0  2LKBDF6u2QNrzlUPzvpNAS  Tellin' Me Lies  Nature Of The Beast    April Wine   \n",
       "1  2Y0HDXWfE6KifNzo3GEScQ           Dancer            Hot Space         Queen   \n",
       "2  2lRETJsdBygk2oVWbPpSRV     In Your Eyes   Emotions In Motion  Billy Squier   \n",
       "\n",
       "  release_date    length  popularity  danceability  acousticness  energy  \\\n",
       "0         1981  3.025550          23         0.706        0.0543   0.637   \n",
       "1   1982-05-03  3.818883          25         0.745        0.0435   0.336   \n",
       "2         1982  3.777333          25         0.526        0.2200   0.558   \n",
       "\n",
       "   instrumentalness  liveness  loudness  speechiness    tempo  time_signature  \\\n",
       "0          0.000046    0.0616   -11.825       0.0340  137.752             3.0   \n",
       "1          0.000009    0.0766   -11.970       0.0398  100.093             4.0   \n",
       "2          0.000000    0.0589    -6.918       0.0291  137.122             4.0   \n",
       "\n",
       "   keys  valence  \n",
       "0     9    0.965  \n",
       "1     9    0.625  \n",
       "2     2    0.566  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(\"----- Top 100 Songs -----\")\n",
    "display(top100.head(3))\n",
    "print(\"----- All Songs -----\")\n",
    "display(allSongs.head(3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Start of Testing Below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.metrics.pairwise import cosine_similarity\n",
    "import sklearn_recommender as skr\n",
    "from sklearn import preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the features we want to use to make a recommendation\n",
    "# for the sake of testing, we'll look at the two sets differently\n",
    "# since top100 is smaller and likely has all songs meant for american audience\n",
    "\n",
    "features = ['danceability','acousticness','energy','liveness','speechiness','tempo','loudness','valence']\n",
    "\n",
    "recDF = top100[features]\n",
    "items = tf.transform(recDF)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf = skr.transformer.SimilarityTransformer(cols=(0,-1),normalize=True)\n",
    "sim_mat = tf.transform(items)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "rec = skr.recommender.SimilarityRecommender(5).fit(sim_mat)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>year</th>\n",
       "      <th>rank</th>\n",
       "      <th>artist</th>\n",
       "      <th>title</th>\n",
       "      <th>album</th>\n",
       "      <th>release_date</th>\n",
       "      <th>length</th>\n",
       "      <th>popularity</th>\n",
       "      <th>danceability</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>energy</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>liveness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>tempo</th>\n",
       "      <th>time_signature</th>\n",
       "      <th>keys</th>\n",
       "      <th>valence</th>\n",
       "      <th>ids</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1302</th>\n",
       "      <td>2003</td>\n",
       "      <td>5</td>\n",
       "      <td>3 Doors Down</td>\n",
       "      <td>When I'm Gone</td>\n",
       "      <td>Away From The Sun</td>\n",
       "      <td>2002-11-12</td>\n",
       "      <td>260333</td>\n",
       "      <td>68</td>\n",
       "      <td>0.530</td>\n",
       "      <td>0.00385</td>\n",
       "      <td>0.768</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.1030</td>\n",
       "      <td>-5.611</td>\n",
       "      <td>0.0284</td>\n",
       "      <td>148.095</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>0.374</td>\n",
       "      <td>3WbphvawbMZ8FyqDxYGdSQ</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1587</th>\n",
       "      <td>2006</td>\n",
       "      <td>90</td>\n",
       "      <td>Eminem</td>\n",
       "      <td>When I'm Gone</td>\n",
       "      <td>When I'm Gone</td>\n",
       "      <td>2005-12-06</td>\n",
       "      <td>281120</td>\n",
       "      <td>62</td>\n",
       "      <td>0.608</td>\n",
       "      <td>0.05510</td>\n",
       "      <td>0.745</td>\n",
       "      <td>0.00000</td>\n",
       "      <td>0.2700</td>\n",
       "      <td>-5.496</td>\n",
       "      <td>0.3650</td>\n",
       "      <td>150.068</td>\n",
       "      <td>4</td>\n",
       "      <td>10</td>\n",
       "      <td>0.725</td>\n",
       "      <td>2lRI7n5b9jlPYsEpRuots6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2218</th>\n",
       "      <td>2013</td>\n",
       "      <td>21</td>\n",
       "      <td>Anna Kendrick</td>\n",
       "      <td>Cups (Pitch Perfect's When I'm Gone)</td>\n",
       "      <td>Ultimate Pitch Perfect (Original Motion Pictur...</td>\n",
       "      <td>2015-02-10</td>\n",
       "      <td>128013</td>\n",
       "      <td>59</td>\n",
       "      <td>0.878</td>\n",
       "      <td>0.04620</td>\n",
       "      <td>0.455</td>\n",
       "      <td>0.00484</td>\n",
       "      <td>0.0353</td>\n",
       "      <td>-8.845</td>\n",
       "      <td>0.0770</td>\n",
       "      <td>129.953</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>0.790</td>\n",
       "      <td>6mH3qVIeOsnQIAho5eWwhH</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      year  rank         artist                                 title  \\\n",
       "1302  2003     5   3 Doors Down                         When I'm Gone   \n",
       "1587  2006    90         Eminem                         When I'm Gone   \n",
       "2218  2013    21  Anna Kendrick  Cups (Pitch Perfect's When I'm Gone)   \n",
       "\n",
       "                                                  album release_date  length  \\\n",
       "1302                                  Away From The Sun   2002-11-12  260333   \n",
       "1587                                      When I'm Gone   2005-12-06  281120   \n",
       "2218  Ultimate Pitch Perfect (Original Motion Pictur...   2015-02-10  128013   \n",
       "\n",
       "      popularity  danceability  acousticness  energy  instrumentalness  \\\n",
       "1302          68         0.530       0.00385   0.768           0.00000   \n",
       "1587          62         0.608       0.05510   0.745           0.00000   \n",
       "2218          59         0.878       0.04620   0.455           0.00484   \n",
       "\n",
       "      liveness  loudness  speechiness    tempo  time_signature  keys  valence  \\\n",
       "1302    0.1030    -5.611       0.0284  148.095               4     7    0.374   \n",
       "1587    0.2700    -5.496       0.3650  150.068               4    10    0.725   \n",
       "2218    0.0353    -8.845       0.0770  129.953               4     0    0.790   \n",
       "\n",
       "                         ids  \n",
       "1302  3WbphvawbMZ8FyqDxYGdSQ  \n",
       "1587  2lRI7n5b9jlPYsEpRuots6  \n",
       "2218  6mH3qVIeOsnQIAho5eWwhH  "
      ]
     },
     "execution_count": 107,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# to test I want to use a song I am familiar with\n",
    "# after shuffling the first one I knew enough\n",
    "# was \"when I'm gone\" sadly...\n",
    "\n",
    "top100.loc[top100['title'].str.contains(\"When I'm Gone\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[163 277  51   5 289]\n",
      " [163  51 277   5 289]]\n"
     ]
    }
   ],
   "source": [
    "# ohh, I know the em song too, so I can test both\n",
    "\n",
    "print(rec.predict([1302,1587]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "LL Cool J - Around the Way Girl\n"
     ]
    }
   ],
   "source": [
    "# for some reason this is the top recommendation for both\n",
    "print(top100.iloc[163].artist + \" - \" + top100.iloc[163].title)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3 doors down and em don't seem too close, so something\n",
    "# has to be wrong this looked like the easy approach, but\n",
    "# sadly im going to have to do a little bit more work apparently\n",
    "# I think I want to preprocess some of the info myself,\n",
    "# and then reapproach with another method\n",
    "\n",
    "\n",
    "x = recDF.values\n",
    "min_max_scaler = preprocessing.MinMaxScaler()\n",
    "x_scaled = min_max_scaler.fit_transform(x)\n",
    "normalized = pd.DataFrame(x_scaled)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "cosined = cosine_similarity(normalized)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "OneRepublic - Secrets\n",
      "Nickelback - Photograph\n",
      "Kelly Clarkson - My Life Would Suck Without You\n",
      "Trace Adkins - You're Gonna Miss This\n",
      "Miley Cyrus - Malibu\n",
      "Hanson - I Will Come to You\n"
     ]
    }
   ],
   "source": [
    "def top100Searcher(songID,matrix):\n",
    "    similar = list(enumerate(matrix[songID]))\n",
    "    sortedSimilar = sorted(similar,key=lambda x:x[1],reverse=True)[1:]\n",
    "    i=0\n",
    "    for j in sortedSimilar:\n",
    "        print(top100.iloc[j[0]].artist + \" - \" + top100.iloc[j[0]].title)\n",
    "        i=i+1\n",
    "        if i>5:\n",
    "            break\n",
    "\n",
    "top100Searcher(1302,cosined)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ludacris - Splash Waterfalls\n",
      "Lou Bega - Mambo No. 5\n",
      "Eminem Featuring Juice WRLD - Godzilla\n",
      "Luther Vandross and Janet Jackson - The Best Things in Life Are Free\n",
      "DJ Khaled Featuring Justin Bieber, Chance The Rapper & Quavo - No Brainer\n",
      "Lil Wayne - A Milli\n"
     ]
    }
   ],
   "source": [
    "# okay wow. I'm suprised. The first two seem kinda close actually.\n",
    "# since that eminem song came up too we'll try that one too\n",
    "\n",
    "top100Searcher(1587,cosined)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Miley Cyrus - Party In The U.S.A.\n",
      "Harry Styles - Adore You\n",
      "TLC - Unpretty\n",
      "Moby featuring Gwen Stefani - South Side\n",
      "Adele - Set Fire To The Rain\n",
      "Ella Henderson - Ghost\n"
     ]
    }
   ],
   "source": [
    "# searching party in the USA (it returns itself because theres a duplicate in the df)\n",
    "top100Searcher(1826,cosined)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 151,
   "metadata": {},
   "outputs": [],
   "source": [
    "# I'm suprised how well it works. Now to go and attempt it with \n",
    "# the bigger list, first lets clean up and make functions\n",
    "\n",
    "def createCosMatrix(df, features):\n",
    "    # creates the df with the features we want\n",
    "    recDF = df[features]\n",
    "    \n",
    "    # normalized all the features\n",
    "    x = recDF.values\n",
    "    min_max_scaler = preprocessing.MinMaxScaler()\n",
    "    x_scaled = min_max_scaler.fit_transform(x)\n",
    "    normalized = pd.DataFrame(x_scaled)\n",
    "    \n",
    "    # creates an NxN matrix of cosine similarities\n",
    "    cosined = cosine_similarity(normalized)\n",
    "    \n",
    "    return cosined\n",
    "\n",
    "def idSearch(df,songname):\n",
    "    # our naming conventions were off for titles, so this\n",
    "    # will do in a pinch\n",
    "    try:\n",
    "        display(df.loc[df['title'].str.contains(songname)])\n",
    "    except:\n",
    "        display(df.loc[df['name'].str.contains(songname)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test = createCosMatrix(allSongs,features)\n",
    "idSearch(allSongs,\"Shake\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 179,
   "metadata": {},
   "outputs": [],
   "source": [
    "# oof thats a lot of numbers\n",
    "# I tried scaling it down to a less accurate float\n",
    "# but were still at 441 GB of memory needed\n",
    "# we only technically need the row we query though,\n",
    "# so maybe a tweak will fix it\n",
    "\n",
    "def getSongCosine(df, features, songID):\n",
    "    # creates the df with the features we want\n",
    "    recDF = df[features]\n",
    "    \n",
    "    # probably dont need this anymore because were not saving\n",
    "    # the full matrix\n",
    "    #recDF = recDF.astype(np.float32)\n",
    "    \n",
    "    # normalized all the features\n",
    "    x = recDF.values\n",
    "    min_max_scaler = preprocessing.MinMaxScaler()\n",
    "    x_scaled = min_max_scaler.fit_transform(x)\n",
    "    normalized = pd.DataFrame(x_scaled)\n",
    "    \n",
    "    # creates an NxN matrix of cosine similarities\n",
    "    cosined = cosine_similarity(normalized[songID:songID+1],normalized)\n",
    "    \n",
    "    return cosined\n",
    "\n",
    "\n",
    "# now we also need a way to search songs in allSongs\n",
    "\n",
    "def allSongSearcher(matrix):\n",
    "    similar = list(enumerate(matrix[0]))\n",
    "    sortedSimilar = sorted(similar,key=lambda x:x[1],reverse=True)[1:]\n",
    "    i=0\n",
    "    for j in sortedSimilar:\n",
    "        print(allSongs.iloc[j[0]].artist + \" - \" + allSongs['name'].iloc[j[0]])\n",
    "        i=i+1\n",
    "        if i>5:\n",
    "            break"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 180,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we'll test with taylor swift's shake it off (2906)\n",
    "\n",
    "test = getSongCosine(allSongs,features,2906)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 181,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Taylor Swift - Shake It Off\n",
      "Taylor Swift - Shake It Off\n",
      "Taylor Swift - Shake It Off\n",
      "Taylor Swift - Shake It Off\n",
      "Cage The Elephant - Ain't No Rest for the Wicked\n",
      "Charli XCX - Break the Rules - ODESZA Remix\n"
     ]
    }
   ],
   "source": [
    "# all right, there seems to be A LOT of\n",
    "# duplicates in this, so i want to try\n",
    "# this again after deleting the dupilcates\n",
    "\n",
    "# EDIT: nevermind, apparently spotify has\n",
    "# 4 different releases of this song all\n",
    "# with unique IDs, so we'll do a different song\n",
    "\n",
    "allSongSearcher(test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 185,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ids</th>\n",
       "      <th>name</th>\n",
       "      <th>album</th>\n",
       "      <th>artist</th>\n",
       "      <th>release_date</th>\n",
       "      <th>length</th>\n",
       "      <th>popularity</th>\n",
       "      <th>danceability</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>energy</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>liveness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>tempo</th>\n",
       "      <th>time_signature</th>\n",
       "      <th>keys</th>\n",
       "      <th>valence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>121576</th>\n",
       "      <td>22oEJW6r2rMb9z4IntfyEa</td>\n",
       "      <td>Hey Look Ma, I Made It</td>\n",
       "      <td>Pray for the Wicked</td>\n",
       "      <td>Panic! At The Disco</td>\n",
       "      <td>2018-06-22</td>\n",
       "      <td>0.283</td>\n",
       "      <td>71</td>\n",
       "      <td>0.577</td>\n",
       "      <td>0.0137</td>\n",
       "      <td>0.833</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.121</td>\n",
       "      <td>-3.337</td>\n",
       "      <td>0.0695</td>\n",
       "      <td>107.936</td>\n",
       "      <td>4.0</td>\n",
       "      <td>5</td>\n",
       "      <td>0.58</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           ids                    name                album  \\\n",
       "121576  22oEJW6r2rMb9z4IntfyEa  Hey Look Ma, I Made It  Pray for the Wicked   \n",
       "\n",
       "                     artist release_date  length  popularity  danceability  \\\n",
       "121576  Panic! At The Disco   2018-06-22   0.283          71         0.577   \n",
       "\n",
       "        acousticness  energy  instrumentalness  liveness  loudness  \\\n",
       "121576        0.0137   0.833               0.0     0.121    -3.337   \n",
       "\n",
       "        speechiness    tempo  time_signature  keys  valence  \n",
       "121576       0.0695  107.936             4.0     5     0.58  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "idSearch(allSongs,\"Hey Look Ma, I Made It\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 186,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Tenille Arts - Somebody Like That\n",
      "Moby - Extreme Ways\n",
      "Poom - Les voiles - Christine Remix\n",
      "BØRNS - Dopamine\n",
      "Moby - Extreme Ways\n",
      "Punk Goes - In My Head\n"
     ]
    }
   ],
   "source": [
    "# I could see the first one making sense, not\n",
    "# the same vibe, but the vocalization and band\n",
    "# prescence is kind of similar\n",
    "\n",
    "allSongSearcher(getSongCosine(allSongs,features,121576))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 187,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ids</th>\n",
       "      <th>name</th>\n",
       "      <th>album</th>\n",
       "      <th>artist</th>\n",
       "      <th>release_date</th>\n",
       "      <th>length</th>\n",
       "      <th>popularity</th>\n",
       "      <th>danceability</th>\n",
       "      <th>acousticness</th>\n",
       "      <th>energy</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>liveness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>tempo</th>\n",
       "      <th>time_signature</th>\n",
       "      <th>keys</th>\n",
       "      <th>valence</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>285419</th>\n",
       "      <td>40wb1DugiG4c4ztCt9oaWp</td>\n",
       "      <td>Me Like Yuh (feat. Hoody) [Korean Version]</td>\n",
       "      <td>Everything You Wanted</td>\n",
       "      <td>Jay Park</td>\n",
       "      <td>2016-10-20</td>\n",
       "      <td>0.373</td>\n",
       "      <td>57</td>\n",
       "      <td>0.690</td>\n",
       "      <td>0.305</td>\n",
       "      <td>0.820</td>\n",
       "      <td>0.000002</td>\n",
       "      <td>0.138</td>\n",
       "      <td>-6.763</td>\n",
       "      <td>0.1130</td>\n",
       "      <td>99.965</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>318662</th>\n",
       "      <td>5S5gGOsUarzRXKoaSstwba</td>\n",
       "      <td>Me Like Yuh</td>\n",
       "      <td>Everything You Wanted</td>\n",
       "      <td>Jay Park</td>\n",
       "      <td>2016-10-20</td>\n",
       "      <td>0.373</td>\n",
       "      <td>0</td>\n",
       "      <td>0.715</td>\n",
       "      <td>0.223</td>\n",
       "      <td>0.831</td>\n",
       "      <td>0.000004</td>\n",
       "      <td>0.132</td>\n",
       "      <td>-6.161</td>\n",
       "      <td>0.0729</td>\n",
       "      <td>100.006</td>\n",
       "      <td>4.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.805</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           ids                                        name  \\\n",
       "285419  40wb1DugiG4c4ztCt9oaWp  Me Like Yuh (feat. Hoody) [Korean Version]   \n",
       "318662  5S5gGOsUarzRXKoaSstwba                                 Me Like Yuh   \n",
       "\n",
       "                        album    artist release_date  length  popularity  \\\n",
       "285419  Everything You Wanted  Jay Park   2016-10-20   0.373          57   \n",
       "318662  Everything You Wanted  Jay Park   2016-10-20   0.373           0   \n",
       "\n",
       "        danceability  acousticness  energy  instrumentalness  liveness  \\\n",
       "285419         0.690         0.305   0.820          0.000002     0.138   \n",
       "318662         0.715         0.223   0.831          0.000004     0.132   \n",
       "\n",
       "        loudness  speechiness    tempo  time_signature  keys  valence  \n",
       "285419    -6.763       0.1130   99.965             4.0     2    0.750  \n",
       "318662    -6.161       0.0729  100.006             4.0     2    0.805  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "idSearch(allSongs,\"Me Like Yuh\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 188,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "MC Rich - Down the Coast\n",
      "Billy Paul - Let 'Em In\n",
      "Onyx - Hold Up\n",
      "Andra - Camarero\n",
      "Keko Salata - Pari kilometriä (feat. Diandra)\n",
      "Sofiane - Nouveaux parrains\n"
     ]
    }
   ],
   "source": [
    "# The first song feels REALLY similar, but\n",
    "# the second one is waaaay off. \n",
    "# third seems to be like a podcast? and \n",
    "# 4th is a different genre, but it feels similar\n",
    "\n",
    "allSongSearcher(getSongCosine(allSongs,features,285419))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}