chapter-12-web-scraping.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8fe5b251",
   "metadata": {},
   "source": [
    "# Chapter 12: WEB SCRAPING"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91801838",
   "metadata": {},
   "source": [
    "- `webbrowser` - Comes with Python and opens a browser to a specific page.\n",
    "\n",
    "- `requests` - Downloads files and web pages from the internet.\n",
    "\n",
    "- `bs4` - Parses HTML, the format that web pages are written in.\n",
    "\n",
    "- `selenium` - Launches and controls a web browser. The selenium module is able to fill in forms and\n",
    "simulate mouse clicks in this browser."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "86dbefae",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import webbrowser\n",
    "\n",
    "webbrowser.open('https://inventwithpython.com/')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7d385dc",
   "metadata": {},
   "source": [
    "## Project: mapIt.py with the webbrowser Module"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "22f5bacf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# ADDRESS: mapit 870 Valencia St, San Francisco, CA 94110\n",
    "\n",
    "#! python3\n",
    "# mapIt.py - Launches a map in the browser using an address from the\n",
    "\n",
    "import sys\n",
    "import pyperclip\n",
    "import webbrowser\n",
    "\n",
    "# Command line or clipboard.\n",
    "if len(sys.argv) > 1:\n",
    "    # get address from command line.\n",
    "    address = \" \".join(sys.argv[1:])\n",
    "else:\n",
    "# get address from clipboard.\n",
    "    address = pyperclip.paste()\n",
    "\n",
    "webbrowser.open(\"https://google.com/maps/place/\" + address)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "565fe60f",
   "metadata": {},
   "source": [
    "### Ideas for Similar Programs\n",
    "\n",
    "- Open all links on a page in separate browser tabs.\n",
    "- Open the browser to the URL for your local weather.\n",
    "- Open several social network sites that you regularly check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e43cd67c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open all links on a page in separate browser tabs.\n",
    "links = [\"google.com\", \"amazon.com\", \"telegram.org\"]\n",
    "\n",
    "for link in links:\n",
    "    webbrowser.open(\"https://\" + link)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8ee1a4b",
   "metadata": {},
   "source": [
    "## Downloading Files from the Web with the requests Module\n",
    "\n",
    "The `requests` module lets you easily download files from the web without having to worry about complicated issues such as network errors, connection problems, and data compression. The `requests` module doesn’t come with Python, so you’ll have to install it first"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd4f45a9",
   "metadata": {},
   "source": [
    "### Downloading a Web Page with the requests.get() Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "e02fa2bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "a6f5fe0f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [200]>"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = requests.get(\"https://automatetheboringstuff.com/files/rj.txt\")\n",
    "res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "f6e4e820",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "requests.models.Response"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "ef6f4a37",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res.status_code == requests.codes.ok"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "beb265bd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "178978"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(res.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "3a8ffd84",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Project Gutenberg EBook of Romeo and Juliet, by William Shakespeare\r\n",
      "\r\n",
      "This eBook is for the use of anyone anywhere at no cost and with\r\n",
      "almost no restrictions whatsoever.  You may copy it, give it away or\r\n",
      "re-use it under the terms of the Project\n"
     ]
    }
   ],
   "source": [
    "print(res.text[:251])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7f3a644d",
   "metadata": {},
   "source": [
    "### Checking for Errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "bb724d52",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Response [404]>"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "res = requests.get(\"https://inventwithpython.com/page_that_does_not_exist\")\n",
    "res"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "f9ae077f",
   "metadata": {},
   "outputs": [
    {
     "ename": "HTTPError",
     "evalue": "404 Client Error: Not Found for url: https://inventwithpython.com/page_that_does_not_exist",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mHTTPError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[0;32mIn [76], line 1\u001b[0m\n\u001b[0;32m----> 1\u001b[0m \u001b[43mres\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mraise_for_status\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/miniconda3/lib/python3.9/site-packages/requests/models.py:1021\u001b[0m, in \u001b[0;36mResponse.raise_for_status\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   1016\u001b[0m     http_error_msg \u001b[38;5;241m=\u001b[39m (\n\u001b[1;32m   1017\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mstatus_code\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m Server Error: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mreason\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m for url: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00m\u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39murl\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m   1018\u001b[0m     )\n\u001b[1;32m   1020\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m http_error_msg:\n\u001b[0;32m-> 1021\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m HTTPError(http_error_msg, response\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m)\n",
      "\u001b[0;31mHTTPError\u001b[0m: 404 Client Error: Not Found for url: https://inventwithpython.com/page_that_does_not_exist"
     ]
    }
   ],
   "source": [
    "res.raise_for_status()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "59607cb1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "There was a problem: 404 Client Error: Not Found for url: https://inventwithpython.com/page_that_does_not_exist\n"
     ]
    }
   ],
   "source": [
    "res = requests.get(\"https://inventwithpython.com/page_that_does_not_exist\")\n",
    "\n",
    "try:\n",
    "    res.raise_for_status()\n",
    "except Exception as exc:\n",
    "    print(f\"There was a problem: {exc}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5372cc99",
   "metadata": {},
   "source": [
    "## Saving Downloaded Files to the Hard Drive\n",
    "\n",
    "From here, you can save the web page to a file on your hard drive with the standard `open()` function and `write()` method. There are some slight differences, though. First, you must open the file in *write binary* mode by passing the string `'wb'` as the second argument to `open()`. Even if the page is in plaintext (such as the *Romeo and Juliet* text you downloaded earlier), you need to write binary data instead of text data in order to maintain the *Unicode encoding* of the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "id": "f4701d0e",
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "\n",
    "res = requests.get(\"https://automatetheboringstuff.com/files/rj.txt\")\n",
    "res.raise_for_status()\n",
    "playFile = open('RomeoAndJuliet.txt', 'wb')\n",
    "for chunk in res.iter_content(100_000):\n",
    "    playFile.write(chunk)\n",
    "\n",
    "playFile.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "id": "17faa59f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The Project Gutenberg EBook of Romeo and Juliet, by William Shakespeare\n",
      "\n",
      "This eBook is for the use of anyone anywhere at no cost and with\n",
      "almost no restrictions whatsoever.  You may copy it, give it away or\n",
      "re-use it under the terms of the Project\n"
     ]
    }
   ],
   "source": [
    "with open(\"RomeoAndJuliet.txt\", 'r') as f:\n",
    "    rj_content = f.read()[:247]\n",
    "print(rj_content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "6e59f3dc",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "174126\n"
     ]
    }
   ],
   "source": [
    "with open(\"RomeoAndJuliet.txt\", 'r') as f:\n",
    "    rj_content = f.read()\n",
    "print(len(rj_content))\n",
    "print(rj_content)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "a02eb0b3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "size: 130277 bytes\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import sys\n",
    "\n",
    "res = requests.get(\"https://automatetheboringstuff.com/2e/chapter12\")\n",
    "res.raise_for_status()\n",
    "size = sys.getsizeof(res.text)\n",
    "playFile = open('chapter12.html', 'wb')\n",
    "print(f\"size: {size} bytes\")\n",
    "for chunk in res.iter_content(size):\n",
    "    playFile.write(chunk)\n",
    "playFile.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "2d45570e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\"\n",
      "    \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd\">\n",
      "\n",
      "<html lang=\"en\" xmlns=\"http://www.w3.org/1999/xhtml\" xml:lang=\"en\">\n",
      "<head>\n",
      "  <link rel=\"stylesheet\" type=\"text/css\" href=\"/automate2_website.css\" />\n",
      "  <meta charset=\"UTF-8\" />\n",
      "  <title>Automate the Boring Stuff with Python</title>\n",
      "</head>\n",
      "\n",
      "<body>\n",
      "  <header class=\"top_header\">\n",
      "  <a href=\"https://automatetheboringstuff.com/\">Home</a> | <a href=\"https://www.nostarch.com/automatestuff2\">Buy Direct from Publisher</a> | \n"
     ]
    }
   ],
   "source": [
    "with open(\"chapter12.html\", 'r') as f:\n",
    "    html_content = f.read()[:545]\n",
    "print(html_content)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2746ede6",
   "metadata": {},
   "source": [
    "## Parsing HTML with the bs4 Module\n",
    "\n",
    "Beautiful Soup is a module for extracting information from an HTML page (and is much better for this purpose than regular expressions). The Beautiful Soup module’s name is `bs4` (for Beautiful Soup, version 4).\n",
    "\n",
    "### Creating a BeautifulSoup Object from HTML"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "d83530ea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'bs4.BeautifulSoup'>\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "import bs4\n",
    "\n",
    "res = requests.get(\"https://nostarch.com\")\n",
    "res.raise_for_status()\n",
    "noStarchSoup = bs4.BeautifulSoup(res.text, 'html.parser')\n",
    "print(type(noStarchSoup))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "9810cc0f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'bs4.BeautifulSoup'>\n"
     ]
    }
   ],
   "source": [
    "exampleFile = open(\"example.html\")\n",
    "exampleSoup = bs4.BeautifulSoup(exampleFile, 'html.parser')\n",
    "print(type(exampleSoup))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc7bc4ab",
   "metadata": {},
   "source": [
    "The `'html.parser'` parser used here comes with Python. However, you can use the faster `'lxml'` parser if you install the third-party `lxml` module."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08e5a638",
   "metadata": {},
   "source": [
    "### Finding an Element with the select() Method\n",
    "\n",
    "| Selector passed to the select() method | Will match . . . |\n",
    "| :- | :- |\n",
    "| **`soup.select('div')`** | All elements named `<div>` |\n",
    "| **`soup.select('#author')`** | The element with an id attribute of author |\n",
    "| **`soup.select('.notice')`** | All elements that use a CSS class attribute named notice |\n",
    "| **`soup.select('div span')`** | All elements named `<span>` that are within an element named `<div>` |\n",
    "| **`soup.select('div > span')`** | All elements named `<span>` that are directly within an element named `<div>`, with no other element in between |\n",
    "| **`soup.select('input[name]')`** | All elements named `<input>` that have a name attribute with any value |\n",
    "| **`soup.select('input[type=\"button\"]')`** | All elements named `<input>` that have an attribute named type with value button |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "62dd537c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import bs4\n",
    "\n",
    "exampleFile = open('example.html')\n",
    "exampleSoup = bs4.BeautifulSoup(exampleFile.read(), 'html.parser')\n",
    "elems = exampleSoup.select('#author')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "4e43320d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'bs4.element.ResultSet'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[<span id=\"author\">Al Sweigart</span>]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(type(elems))\n",
    "elems"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "eaa8a6ea",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'bs4.element.Tag'>\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<span id=\"author\">Al Sweigart</span>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(type(elems[0]))\n",
    "elems[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "1e9e4581",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Al Sweigart'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elems[0].getText()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "cf7c889b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 'author'}"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "elems[0].attrs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "a8a90c56",
   "metadata": {},
   "outputs": [],
   "source": [
    "pElems = exampleSoup.select('p')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "5185f3e1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<p>Download my <strong>Python</strong> book from <a href=\"https://\n",
       "inventwithpython.com\">my website</a>.</p>"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pElems[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "9777ac96",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Download my Python book from my website.'"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pElems[0].getText()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "5770caef",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Learn Python the easy way!'"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pElems[1].getText()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "bea59399",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'By Al Sweigart'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pElems[2].getText()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5efabfb4",
   "metadata": {},
   "source": [
    "### Getting Data from an Element’s Attributes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "451e9ac2",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<span id=\"author\">Al Sweigart</span>"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import bs4\n",
    "\n",
    "soup = bs4.BeautifulSoup(open('example.html'), 'html.parser')\n",
    "spanElem = soup.select('span')[0]\n",
    "spanElem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "609fb321",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'author'"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spanElem.get('id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "f34c091f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spanElem.get('some_nonexistent_attr') == None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "b51bc41c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'id': 'author'}"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spanElem.attrs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "953e0787",
   "metadata": {},
   "source": [
    "## Project: Opening All Search Results\n",
    "\n",
    "It would be nice if I could simply type a search term on the command line and have my computer automatically open a browser with all the top search results in new tabs. Let’s write a script to do this with the search results page for the Python Package Index at https://pypi.org/. A program like this can be adapted to many other websites, although the Google and DuckDuckGo often employ measures that make scraping their search results pages difficult."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "693685a0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Searching ...\n",
      "Opeing https://pypi.org/project/f16774e1d64c/\n",
      "Opeing https://pypi.org/project/6d657461666c6f77/\n",
      "Opeing https://pypi.org/project/3636c788d0392f7e84453434eea18c59/\n",
      "Opeing https://pypi.org/project/c3d/\n",
      "Opeing https://pypi.org/project/d2c/\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "import requests\n",
    "import bs4\n",
    "import webbrowser\n",
    "\n",
    "print(\"Searching ...\")  # display text while downlading the search result page\n",
    "\n",
    "res = requests.get(\"https://pypi.org/search/?q=\" + \" \".join(sys.argv[1:]))\n",
    "res.raise_for_status()\n",
    "\n",
    "# Retrieve top search result links.\n",
    "soup = bs4.BeautifulSoup(res.text, 'lxml')\n",
    "# Open a browser tab for each result.\n",
    "linkElems = soup.select('.package-snippet')\n",
    "numOpen = min(5, len(linkElems))\n",
    "for i in range(numOpen):\n",
    "    urlToOpen = 'https://pypi.org' + linkElems[i].get('href')\n",
    "    print('Opeing', urlToOpen)\n",
    "    webbrowser.open(urlToOpen)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e53249e2",
   "metadata": {},
   "source": [
    "### Ideas for Similar Programs\n",
    "\n",
    "- Open all the product pages after searching a shopping site such as Amazon.\n",
    "- Open all the links to reviews for a single product.\n",
    "- Open the result links to photos after performing a search on a photo site such as Flickr or Imgur."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "104a91bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import requests\n",
    "import bs4\n",
    "import webbrowser\n",
    "\n",
    "search_term = \"ibm quantum\"\n",
    "link = \"https://google.com/search?q=ibm+quantum\"\n",
    "classes = [\"MjjYud\", \"ULSxyf\"]\n",
    "\n",
    "print(\"Searching...\")\n",
    "resp = requests.get(link + \"+\".join(sys.argv[1:]))\n",
    "resp"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e62e12fa",
   "metadata": {},
   "source": [
    "## Project: Downloading All XKCD Comics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c9fefb1",
   "metadata": {
    "hide_input": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scraping https://xkcd.com ...\n",
      "Downloading image https://imgs.xkcd.com/comics/account_problems.png ...\n",
      "Scraping https://xkcd.com/2699/ ...\n",
      "Downloading image https://imgs.xkcd.com/comics/feature_comparison.png ...\n",
      "Scraping https://xkcd.com/2698/ ...\n",
      "Downloading image https://imgs.xkcd.com/comics/bad_date.png ...\n",
      "Scraping https://xkcd.com/2697/ ...\n",
      "Downloading image https://imgs.xkcd.com/comics/y2k_and_2038.png ...\n",
      "Scraping https://xkcd.com/2696/ ...\n"
     ]
    }
   ],
   "source": [
    "# downloadXkcd.py\n",
    "\n",
    "import os\n",
    "import requests\n",
    "import bs4\n",
    "\n",
    "url = \"https://xkcd.com\"  # starting url\n",
    "os.makedirs('xkcd', exist_ok=True)  # store comics in ./xkcd\n",
    "\n",
    "while not url.endswith('#'):\n",
    "\n",
    "    # Scrape the page\n",
    "    print(f\"Downloading page {url}\")\n",
    "    res = requests.get(url)\n",
    "    res.raise_for_status()\n",
    "    soup = bs4.BeautifulSoup(res.text, 'lxml')\n",
    "\n",
    "    # Find the URL of the comic image\n",
    "    comic_elem = soup.select(\"#comic img\")\n",
    "    if comic_elem == []:\n",
    "        print(\"Could not find comic image.\")\n",
    "    else:\n",
    "        comic_url = 'https:' + comic_elem[0].get('src')\n",
    "\n",
    "        # Download the image\n",
    "        res = requests.get(comic_url)\n",
    "        res.raise_for_status()\n",
    "\n",
    "        # Save the image to ./xkcd\n",
    "        with open(os.path.join('xkcd', os.path.basename(comic_url)), 'wb') as imgFile:\n",
    "            imgFile.write(res.content)\n",
    "\n",
    "        # Get the Prev button's url\n",
    "        prev_link = soup.select('a[rel=\"prev\"]')[0]\n",
    "        url = 'https://xkcd.com' + prev_link.get('href')\n",
    "\n",
    "print('Done.')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "800b70f4",
   "metadata": {},
   "source": [
    "### Ideas for Similar Programs\n",
    "\n",
    "- Back up an entire site by following all of its links.\n",
    "- Copy all the messages off a web forum.\n",
    "- Duplicate the catalog of items for sale on an online store."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1265816b",
   "metadata": {},
   "source": [
    "## Controlling the Browser with the selenium Module\n",
    "\n",
    "### Starting a selenium-Controlled Browser"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "8a189884",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "selenium.webdriver.chrome.webdriver.WebDriver"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "\n",
    "browser = webdriver.Chrome()\n",
    "type(browser)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "99d6de15",
   "metadata": {},
   "outputs": [],
   "source": [
    "browser.get('https://inventwithpython.com')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0495a94b",
   "metadata": {},
   "source": [
    "### Finding Elements on the Page\n",
    "\n",
    "| **Method name** | **`WebElement` object/list returned** |\n",
    "| :- | :- |\n",
    "| **`browser.find_element_by_class_name(name)`, `browser.find_elements_by_class_name(name)`** | Elements that use the CSS class name |\n",
    "| **`browser.find_element_by_css_selector(selector)`, `browser.find_elements_by_css_selector(selector)`** | Elements that match the CSS selector |\n",
    "| **`browser.find_element_by_id(id)`, `browser.find_elements_by_id(id)`** | Elements with a matching id attribute value |\n",
    "| **`browser.find_element_by_link_text(text)`, `browser.find_elements_by_link_text(text)`** | `<a>` elements that completely match the text provided |\n",
    "| **`browser.find_element_by_partial_link_text(text)`, `browser.find_elements_by_partial_link_text(text)`** | `<a>` elements that contain the text provided |\n",
    "| **`browser.find_element_by_name(name)`, `browser.find_elements_by_name(name)`** | Elements with a matching name attribute value |\n",
    "| **`browser.find_element_by_tag_name(name)`, `browser.find_elements_by_tag_name(name)`** | Elements with a matching tag name (case-insensitive; an `<a>` element is matched by 'a' and 'A') |\n",
    "\n",
    "---\n",
    "\n",
    "### `WebElement` Attributes and Methods\n",
    "\n",
    "| Attribute or method | Description |\n",
    "| :- | :- |\n",
    "| **`tag_name`** | The tag name, such as 'a' for an `<a>` element |\n",
    "| **`get_attribute(name)`** | The value for the element’s name attribute |\n",
    "| **`text`** | The text within the element, such as 'hello' in `<span>`hello`</span>` |\n",
    "| **`clear()`** | For text field or text area elements, clears the text typed into it |\n",
    "| **`is_displayed()`** | Returns True if the element is visible; otherwise returns False |\n",
    "| **`is_enabled()`** | For input elements, returns True if the element is enabled; otherwise returns False |\n",
    "| **`is_selected()`** | For checkbox or radio button elements, returns True if the element is selected; otherwise returns False |\n",
    "| **`location`** | A dictionary with keys `'x'` and `'y'` for the position of the element in the page |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "31d34709",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found <img> element with that class name!\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.by import By\n",
    "\n",
    "browser = webdriver.Chrome()\n",
    "browser.get('https://inventwithpython.com')\n",
    "\n",
    "try:\n",
    "    web_elem = browser.find_element(By.CLASS_NAME, \"cover-thumb\")  # WebElement object\n",
    "    print(f\"Found <{web_elem.tag_name}> element with that class name!\")\n",
    "except:\n",
    "    print(\"Was not able to find an element with that name.\")\n",
    "\n",
    "browser.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "957391d5",
   "metadata": {},
   "source": [
    "### Clicking the Page"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "56204523",
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.by import By\n",
    "\n",
    "browser = webdriver.Chrome()\n",
    "browser.get('https://inventwithpython.com')\n",
    "\n",
    "linkElem = browser.find_element(By.LINK_TEXT, \"Read Online for Free\")\n",
    "linkElem.click()  # follows the \"Read Online for Free\" link\n",
    "\n",
    "# browser.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bea8ed43",
   "metadata": {},
   "source": [
    "### Filling Out and Submitting Forms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "6510b9d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.by import By\n",
    "\n",
    "browser = webdriver.Chrome()\n",
    "browser.get(\"https://login.metafilter.com\")\n",
    "userElem = browser.find_element(By.ID, 'user_name')\n",
    "userElem.send_keys(\"your_username_here\")\n",
    "\n",
    "passwordElem = browser.find_element(By.ID, 'user_pass')\n",
    "passwordElem.send_keys('your_password_here')\n",
    "passwordElem.submit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c981be21",
   "metadata": {},
   "source": [
    "Calling the `submit()` method on any element will have the same result as clicking the Submit button for the form that element is in. (You could have just as easily called `emailElem.submit()`, and the code would have done the same thing.)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "373de10c",
   "metadata": {},
   "source": [
    "### Sending Special Keys\n",
    "\n",
    "**Commonly Used Variables in the `selenium.webdriver.common.keys` Module**\n",
    "| **Attributes** | **Meanings** |\n",
    "| :- | :- |\n",
    "| **`Keys.DOWN`, `Keys.UP`, `Keys.LEFT`, `Keys.RIGHT`** | The keyboard arrow keys |\n",
    "| **`Keys.ENTER`, `Keys.RETURN`** | The ENTER and RETURN keys |\n",
    "| **`Keys.HOME`, `Keys.END`, `Keys.PAGE_DOWN`, `Keys.PAGE_UP`** | The HOME, END, PAGEDOWN, and PAGEUP keys |\n",
    "| **`Keys.ESCAPE`, `Keys.BACK_SPACE, Keys.DELETE`** | The ESC, BACKSPACE, and DELETE keys |\n",
    "| **`Keys.F1`, `Keys.F2`, ... , `Keys.F12`** | The F1 to F12 keys at the top of the keyboard |\n",
    "| **`Keys.TAB`** | The TAB key |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "14bdac53",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "\n",
    "browser = webdriver.Chrome()\n",
    "browser.get(\"https://nostarch.com\")\n",
    "htmlElem = browser.find_element(By.TAG_NAME, \"html\")\n",
    "htmlElem.send_keys(Keys.END)  # scrolls to bottom\n",
    "time.sleep(1)\n",
    "htmlElem.send_keys(Keys.HOME)  # scrolls to top"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2cda440",
   "metadata": {},
   "source": [
    "### Clicking Browser Buttons\n",
    "\n",
    "- `browser.back()` - Clicks the Back button.\n",
    "- `browser.forward()` - Clicks the Forward button.\n",
    "- `browser.refresh()` - Clicks the Refresh/Reload button.\n",
    "- `browser.quit()` - Clicks the Close Window button."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "4e4d3443",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "\n",
    "browser = webdriver.Chrome()\n",
    "browser.maximize_window()\n",
    "browser.get(\"http://inventwithpython.com\")\n",
    "link_elem = browser.find_element(By.LINK_TEXT, \"Read Online for Free\")\n",
    "link_elem.click()\n",
    "browser.find_element(By.TAG_NAME, \"html\").send_keys(Keys.END)  # scrolls to bottom\n",
    "time.sleep(2)\n",
    "browser.back()  # click back button in browser\n",
    "time.sleep(2)\n",
    "browser.find_element(By.TAG_NAME, \"html\").send_keys(Keys.HOME)  # scrolls to top\n",
    "# browser.quit()  # close browser window"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbbef0a1",
   "metadata": {},
   "source": [
    "## Practice Projects"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d82d193",
   "metadata": {},
   "source": [
    "### Command Line Emailer\n",
    "Write a program that takes an email address and string of text on the command line and then, using selenium, logs in to your email account and sends an email of the string to the provided address. (You might want to set up a separate email account for this program.)\n",
    "This would be a nice way to add a notification feature to your programs. You could also write a similar program to send messages from a Facebook or Twitter account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "51be3c32",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\" Script to send email from mail.ru account to any email. \"\"\"\n",
    "\n",
    "import sys\n",
    "import time\n",
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from selenium.webdriver.support.ui import WebDriverWait\n",
    "from selenium.webdriver.support import expected_conditions as EC\n",
    "import pyinputplus as pyip\n",
    "\n",
    "# get email address and text message\n",
    "emailTo = sys.argv[1]\n",
    "message = \" \".join(sys.argv[2:])\n",
    "# open chrome browser\n",
    "browser = webdriver.Chrome()\n",
    "browser.maximize_window()\n",
    "# login in to account\n",
    "browser.get(\"https://mail.ru\")\n",
    "browser.find_element(By.CSS_SELECTOR, 'button[data-testid=\"enter-mail-primary\"]').click()  # click 'Log in' button\n",
    "time.sleep(20)  # time for login to your account\n",
    "\n",
    "unameElem = browser.find_element(By.CSS_SELECTOR, 'input[name=\"username\"]')\n",
    "login = pyip.inputEmail(prompt='Enter your login: ')\n",
    "unameElem.send_keys(login)\n",
    "unameElem.submit()\n",
    "# enter the password\n",
    "passwElem = browser.find_element(By.NAME, \"password\")\n",
    "passwElem.click()\n",
    "password = pyip.inputPassword(prompt='Enter your password: ')\n",
    "passwElem.send_keys(password)\n",
    "passwElem.submit()\n",
    "\n",
    "\n",
    "WebDriverWait(browser, 10).until(EC.presence_of_element_located(\n",
    "    (By.CSS_SELECTOR, 'div.sidebar__compose-btn-box > a'))\n",
    "    ).click()\n",
    "\n",
    "WebDriverWait(browser, 10).until(EC.presence_of_element_located(\n",
    "    (By.CSS_SELECTOR, 'label div input'))\n",
    "    ).send_keys(emailTo)\n",
    "\n",
    "browser.find_element(By.CSS_SELECTOR, 'div[role=\"textbox\"]').send_keys(message)  # put message to text field\n",
    "browser.find_element(By.CSS_SELECTOR, 'button[data-test-id=\"send\"]').click()  # click 'Send' button\n",
    "\n",
    "browser.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e6e3033",
   "metadata": {},
   "source": [
    "### Image Site Downloader\n",
    "Write a program that goes to a photo-sharing site like Flickr or Imgur, searches for a category of photos, and then downloads all the resulting images. You could write a program that works with any photo site that has a search feature."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "226084e5",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\" Searches command line text from yandex.com/images and Downloads the images \"\"\"\n",
    "\n",
    "import sys\n",
    "import os\n",
    "import requests\n",
    "import bs4\n",
    "\n",
    "search_term = \" \".join(sys.argv[1:])\n",
    "link = f\"https://yandex.com/images/search?text={search_term}\"\n",
    "folder_name = \"_\".join(sys.argv[1:]) + \"_images\"\n",
    "os.makedirs(folder_name, exist_ok=True)  # make the images folder\n",
    "\n",
    "print(f\"Searching '{link}'\")\n",
    "res = requests.get(link)\n",
    "soup = bs4.BeautifulSoup(res.text, 'lxml')\n",
    "img_elems = soup.select(\"div > a > img\")\n",
    "img_urls = [\"http:\" + img.get('src') for img in img_elems]\n",
    "\n",
    "img_names = []\n",
    "for elem in img_elems:\n",
    "    alt = elem.get('alt')\n",
    "    name = \"\"\n",
    "    for char in alt:\n",
    "        if char.isalnum():\n",
    "            name += char\n",
    "        if char.isspace():\n",
    "            name += \" \"\n",
    "    img_names.append(name)\n",
    "\n",
    "for url, name in zip(img_urls, img_names):\n",
    "    print(f\"Downloading image {url} ...\")\n",
    "    open(os.path.join(folder_name, f\"{name}.png\"), 'wb').write(requests.get(url).content)\n",
    "\n",
    "print(\"===== Done. =====\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "59079d11",
   "metadata": {},
   "source": [
    "### 2048\n",
    "2048 is a simple game where you combine tiles by sliding them up, down, left, or right with the arrow keys. You can actually get a fairly high score by repeatedly sliding in an up, right, down, and left pattern over and over again. Write a program that will open the game at https://gabrielecirulli.github.io/2048/ and keep sending up, right, down, and left keystrokes to automatically play the game."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9004a98d",
   "metadata": {},
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "from selenium.webdriver.common.by import By\n",
    "\n",
    "browser = webdriver.Chrome()\n",
    "browser.get(\"https://2048game.com\")\n",
    "\n",
    "# play the game\n",
    "body_elem = browser.find_element(By.CSS_SELECTOR, \"body\")\n",
    "for i in range(100):\n",
    "    body_elem.send_keys(Keys.UP)\n",
    "    body_elem.send_keys(Keys.RIGHT)\n",
    "    body_elem.send_keys(Keys.DOWN)\n",
    "    body_elem.send_keys(Keys.LEFT)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c498e9a6",
   "metadata": {},
   "source": [
    "### Link Verification\n",
    "Write a program that, given the URL of a web page, will attempt to download every linked page on the page. The program should flag any pages that have a 404 “Not Found” status code and print them out as broken links."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1bc22405",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "2c7666c1320a956fa05893d10993164efa76d3af81fc7470d6f4943526f3df36"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}