chapter-20-controlling-keyboard-and-mouse-with-gui-automation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "8fe5b251",
   "metadata": {},
   "source": [
    "# Chapter 20: CONTROLLING THE KEYBOARD AND MOUSE WITH GUI AUTOMATION\n",
    "\n",
    "`pip install PyAutoGUI`\n",
    "\n",
    "For linux:\n",
    "- `sudo apt install scrot python3-tk python3-dev`"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9c44d4b",
   "metadata": {},
   "source": [
    "### Pauses and Fail-Safes\n",
    "\n",
    "If your program has a bug and you’re unable to use the keyboard and mouse to shut it down, you can use PyAutoGUI’s fail-safe feature. Quickly slide the mouse to one of the four corners of the screen. Every PyAutoGUI function call has a 10th-of-a-second delay after performing its action to give you enough time to move the mouse to a corner. If PyAutoGUI then finds that the mouse cursor is in a corner, it raises the `pyautogui.FailSafeException` exception. Non-PyAutoGUI instructions will not have this 10th-of-a-second delay.\n",
    "\n",
    "If you find yourself in a situation where you need to stop your PyAutoGUI program, just slam the mouse toward a corner to stop it."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6b854372",
   "metadata": {},
   "source": [
    "### Controlling Mouse Movement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "48164c41",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Size(width=1920, height=1080)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pyautogui\n",
    "\n",
    "wh = pyautogui.size()  # Obtain the screen resolution.\n",
    "wh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ae378050",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1920\n",
      "1080\n"
     ]
    }
   ],
   "source": [
    "print(wh[0])\n",
    "print(wh[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "85a47e34",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1920\n",
      "1080\n"
     ]
    }
   ],
   "source": [
    "print(wh.width)\n",
    "print(wh.height)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f21bc428",
   "metadata": {},
   "source": [
    "### Moving the Mouse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "ae972be1",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyautogui\n",
    "\n",
    "for i in range(2):  # Move mouse in a square\n",
    "    pyautogui.moveTo(100, 100, duration=0.2)\n",
    "    pyautogui.moveTo(200, 100, duration=0.2)\n",
    "    pyautogui.moveTo(200, 200, duration=0.2)\n",
    "    pyautogui.moveTo(100, 200, duration=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1c14b8f",
   "metadata": {},
   "source": [
    "The `pyautogui.move()` function moves the mouse cursor *relative to its current position*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "49fc98a9",
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(2):\n",
    "    pyautogui.move(100, 0, duration=0.2) # right\n",
    "    pyautogui.move(0, 100, duration=0.2) # down\n",
    "    pyautogui.move(-100, 0, duration=0.2) # left\n",
    "    pyautogui.move(0, -100, duration=0.2) # up"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7987b848",
   "metadata": {},
   "source": [
    "### Getting the Mouse Position"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "dc0fc43e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Point(x=552, y=807)"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyautogui.position() # Get current mouse position."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "ac5a0bbc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Point(x=552, y=806)"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p = pyautogui.position()\n",
    "p"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "41f3bffb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "552"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p[0]  # The x-coordinate is at index 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "c23209bb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "552"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p.x  # The x-coordinate is also in the x attribute."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "7221d7c1",
   "metadata": {},
   "source": [
    "### Controlling Mouse Interaction"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "8c759e48",
   "metadata": {},
   "source": [
    "#### Clicking the Mouse\n",
    "\n",
    "`pyautogui.click(100, 150, button='left')`\n",
    "\n",
    "button: 'left', 'middle', 'right'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "21fc0a08",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyautogui\n",
    "\n",
    "pyautogui.click(10, 5)  # Move mouse to (10, 5) and click."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "485e489c",
   "metadata": {},
   "outputs": [],
   "source": [
    "pyautogui.mouseDown(30, 70)\n",
    "pyautogui.move(0, 150, duration=0.2)\n",
    "pyautogui.mouseUp()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "902c779f",
   "metadata": {},
   "outputs": [],
   "source": [
    "pyautogui.doubleClick()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "2db91018",
   "metadata": {},
   "outputs": [],
   "source": [
    "pyautogui.rightClick()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "b06c56ca",
   "metadata": {},
   "outputs": [],
   "source": [
    "pyautogui.leftClick()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "pyautogui.middleClick()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a1552ca2",
   "metadata": {},
   "source": [
    "#### Dragging the Mouse\n",
    "\n",
    "- `pyautogui.drag()`\n",
    "- `pyautogui.dragTo()`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "53d06592",
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "import pyautogui\n",
    "\n",
    "time.sleep(3)\n",
    "pyautogui.click()  # Click to make the window active.\n",
    "distance = 400\n",
    "change = 20\n",
    "\n",
    "while distance > 0:\n",
    "    pyautogui.drag(distance, 0)  # Move right.\n",
    "    distance -= change\n",
    "    pyautogui.drag(0, distance)  # Move down.\n",
    "    pyautogui.drag(-distance, 0)  # Move left.\n",
    "    distance -= change\n",
    "    pyautogui.drag(0, -distance)  # Move up."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9f7243b0",
   "metadata": {},
   "source": [
    "#### Scrolling the Mouse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d53308bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyautogui\n",
    "\n",
    "pyautogui.scroll(200)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e3f54354",
   "metadata": {},
   "source": [
    "### Planning Your Mouse Movements\n",
    "\n",
    "By default, the 3 Sec. Button Delay checkbox is checked, causing a three-second delay between clicking a Copy or Log button and the copying or logging taking place. This gives you a short amount of time in which to click the button and then move the mouse into your desired position. It may be easier to uncheck this box, move the mouse into position, and press the F1 to F8 keys to copy or log the mouse position.\n",
    "\n",
    "For example, uncheck the 3 Sec. Button Delay, then move the mouse around the screen while pressing the F6 button, and notice how the x- and y-coordinates of the mouse are recorded in the large text field in the middle of the window. You can later use these coordinates in your PyAutoGUI scripts.\n",
    "\n",
    "For more information on MouseInfo, review the complete documentation at https://mouseinfo.readthedocs.io/."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "394b5480",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyautogui\n",
    "\n",
    "pyautogui.mouseInfo()  # provides mouse coordinate information"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b03fb95d",
   "metadata": {},
   "source": [
    "### Working with the Screen"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "318a7650",
   "metadata": {},
   "source": [
    "#### Getting a Screenshot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45a3059d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pyautogui\n",
    "\n",
    "im = pyautogui.screenshot()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e865c8be",
   "metadata": {},
   "source": [
    "The `im` variable will contain the `Image` object of the screenshot. You can now call methods on the `Image` object in the `im` variable, just like any other `Image` object."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d164c354",
   "metadata": {},
   "source": [
    "#### Analyzing the Screenshot"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4ec07ce4",
   "metadata": {},
   "source": [
    "Say that one of the steps in your GUI automation program is to click a gray button. Before calling the `click()` method, you could take a screenshot and look at the pixel where the script is about to click. If it’s not the same gray as the gray button, then your program knows something is wrong. Maybe the window moved unexpectedly, or maybe a pop-up dialog has blocked the button. At this point, instead of continuing—and possibly wreaking havoc by clicking the wrong thing—your program can “see” that it isn’t clicking the right thing and stop itself. You can obtain the RGB color value of a particular pixel on the screen with the `pixel()` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "800e42a2",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 17, 38)"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pyautogui\n",
    "\n",
    "pyautogui.pixel(0, 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47bd9bf6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(40, 44, 52)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyautogui.pixel(50, 200)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "a0a2b415",
   "metadata": {},
   "source": [
    "The return value from `pixel()` is an RGB tuple of three integers for the amount of red, green, and blue in the pixel. (There is no fourth value for alpha, because screenshot images are fully opaque.)\n",
    "\n",
    "PyAutoGUI’s `pixelMatchesColor()` function will return True if the pixel at the given x- and y-coordinates on the screen matches the given color. The first and second arguments are integers for the x- and y-coordinates, and the third argument is a tuple of three integers for the RGB color the screen pixel must match."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "98c5d777",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(40, 44, 52)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyautogui.pixel(50, 200)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ab65d7e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyautogui.pixelMatchesColor(50, 200, ((40, 44, 52)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "868d3915",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyautogui.pixelMatchesColor(50, 200, (255, 135, 144))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "1b422d3a",
   "metadata": {},
   "source": [
    "### Image Recognition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0f850131",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Box(left=0, top=0, width=63, height=52)"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pyautogui\n",
    "\n",
    "b = pyautogui.locateOnScreen('submit.png')\n",
    "b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a05f9d1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e5b45ccb",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "b.left"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a65cec39",
   "metadata": {},
   "outputs": [],
   "source": [
    "pyautogui.click((643, 745, 70, 29))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "010eaca0",
   "metadata": {},
   "outputs": [],
   "source": [
    "pyautogui.click(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2462de52",
   "metadata": {},
   "outputs": [],
   "source": [
    "pyautogui.click('submit.png')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc51e18c",
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    location = pyautogui.locateOnScreen('submit.png')\n",
    "except:\n",
    "    print(\"Image could not be found.\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9775cecf",
   "metadata": {},
   "source": [
    "### Getting Window Information"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "548e027a",
   "metadata": {},
   "source": [
    "#### Obtaining the Active Window"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "47e98032",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Win32Window(hWnd=918830)"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import time\n",
    "import pyautogui\n",
    "\n",
    "pyautogui.getActiveWindow()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "05b0ac02",
   "metadata": {},
   "source": [
    "In the interactive shell, call the `pyautogui.getActiveWindow()` function to get a Window object.\n",
    "\n",
    "Once you have that Window object, you can retrieve any of the object’s attributes, which describe its size, position, and title:\n",
    "\n",
    "- `left, right, top, bottom` - A single integer for the x- or y-coordinate of the window’s side\n",
    "- `topleft, topright, bottomleft, bottomright` - A named tuple of two integers for the (x, y) coordinates of the window’s corner\n",
    "- `midleft, midright, midleft, midright` - A named tuple of two integers for the (x, y) coordinate of the middle of the window’s side\n",
    "- `width, height` - A single integer for one of the window’s dimensions, in pixels\n",
    "- `size` - A named tuple of two integers for the (width, height) of the window\n",
    "- `area` - A single integer representing the area of the window, in pixels\n",
    "- `center` - A named tuple of two integers for the (x, y) coordinate of the window’s center\n",
    "- `centerx, centery` - A single integer for the x- or y-coordinate of the window’s center\n",
    "- `box` - A named tuple of four integers for the (left, top, width, height) measurements of the window\n",
    "- `title` - A string of the text in the title bar at the top of the window"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a2ee2216",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Win32Window(hWnd=918830)"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pyautogui\n",
    "\n",
    "wo = pyautogui.getActiveWindow()\n",
    "wo"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "89ad0ba5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<Win32Window left=\"-9\", top=\"-9\", width=\"1938\", height=\"1048\", title=\"● chapter-20-controlling-keyboard-and-mouse-with-gui-automation.ipynb - Visual Studio Code\">'"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "str(wo)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7363333f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'● chapter-20-controlling-keyboard-and-mouse-with-gui-automation.ipynb - Visual Studio Code'"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wo.title"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e6bd475f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Size(width=1938, height=1048)"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wo.size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "91f170d1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(-9, -9, 1929, 1039)"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wo.left, wo.top, wo.right, wo.bottom"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "108f2cb8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Point(x=1929, y=-9)"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "wo.topright"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7dacef2c",
   "metadata": {},
   "outputs": [],
   "source": [
    "pyautogui.click(wo.left+10, wo.right+20)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "ddf9ea2e",
   "metadata": {},
   "source": [
    "#### Other Ways of Obtaining Windows\n",
    "\n",
    "The following functions return a list of `Window` objects. If they’re unable to find any windows, they return an empty list:\n",
    "\n",
    "- `pyautogui.getAllWindows()` - returns a list of Window objects for every visible window on the screen.\n",
    "- `pyautogui.getWindowsAt(x, y)` - returns a list of Window objects for every visible window that includes the point (x, y).\n",
    "- `pyautogui.getActiveWindow()` - returns the Window object for the window that is currently receiving keyboard focus.\n",
    "- `pyautogui.getWindowsWithTitle(title)` - returns a list of Window objects for every visible window that includes the string title in its title bar.\n",
    "- `pyautogui.getAllTitles()` - returns a list of strings of every visible window."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "65d1d6ed",
   "metadata": {},
   "source": [
    "#### Manipulating Windows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7c11040b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1938"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pyautogui\n",
    "\n",
    "fw = pyautogui.getActiveWindow()\n",
    "fw.width # Gets the current width of the window. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6021f2b5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Point(x=-9, y=-9)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fw.topleft # Gets the current position of the window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4bc4655e",
   "metadata": {},
   "outputs": [],
   "source": [
    "fw.width = 1000 # Resizes the width.\n",
    "fw.topleft = (800, 400) # Moves the window"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "fcd9c0a9",
   "metadata": {},
   "source": [
    "You can also find out and change the window’s minimized, maximized, and activated states. Try entering the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a1f01091",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pyautogui\n",
    "\n",
    "fw = pyautogui.getActiveWindow()\n",
    "fw.isMaximized # Returns True if window is maximized. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3af327d0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fw.isMinimized # Returns True if window is minimized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c07f8327",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fw.isActive  # Returns True if window is the active window."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aaa80c43",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fw.maximize()  # Maximizes the window.\n",
    "fw.isMaximized"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "17deaaa4",
   "metadata": {},
   "outputs": [],
   "source": [
    "fw.restore() # Undoes a minimize/maximize action.\n",
    "fw.minimize() # Minimizes the window. \n",
    "\n",
    "import time\n",
    "# Wait 5 seconds while you activate a different window:\n",
    "time.sleep(5); fw.activate()\n",
    "fw.close() # This will close the window you're typing in"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "e73b6e63",
   "metadata": {},
   "source": [
    "### Controlling the Keyboard\n",
    "\n",
    "PyAutoGUI also has functions for sending virtual keypresses to your computer, which enables you to fill out forms or enter text into applications."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "9b34eb6c",
   "metadata": {},
   "source": [
    "#### Sending a String from the Keyboard\n",
    "\n",
    "The `pyautogui.write()` function sends virtual keypresses to the computer. What these keypresses do depends on what window is active and what text field has focus. You may want to first send a mouse click to the text field you want in order to ensure that it has focus.\n",
    "\n",
    "As a simple example, let’s use Python to automatically type the words *Hello, world!* into a file editor window. First, open a new file editor window and position it in the upper-left corner of your screen so that PyAutoGUI will click in the right place to bring it into focus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "171c0487",
   "metadata": {},
   "outputs": [],
   "source": [
    "## Create a new text file and write 'Hello, world!' in VS Code.\n",
    "import pyautogui\n",
    "\n",
    "pyautogui.click(x=76, y=20)  # click 'File' option\n",
    "pyautogui.click(x=123, y=59)  # click 'New Text File'\n",
    "pyautogui.write('Hello, world!')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "dbf05a47",
   "metadata": {},
   "source": [
    "#### Key Names\n",
    "\n",
    "| Keyboard key string | Meaning |\n",
    "| :- | :- |\n",
    "| `'a'`, `'b'`, `'c'`, `'A'`, `'B'`, `'C'`, `'1'`, `'2'`, `'3'`, `'!'`, `'@'`, `'#'`, and so on | The keys for single characters |\n",
    "| `'enter'` (or `'return'` or `'\\n'`) | The `enter` key |\n",
    "| `'esc'` | The `esc` key |\n",
    "| `'shiftleft'`, `'shiftright'` | The left and right `shift` keys |\n",
    "| `'altleft'`, `'altright'` | The left and right `alt` keys |\n",
    "| `'ctrlleft'`, `'ctrlright'` | The left and right `ctrl` keys |\n",
    "| `'tab'` (or '`\\t'`) | The `tab` key |\n",
    "| `'backspace'`, `'delete'` | The `backspace` and `delete` keys |\n",
    "| `'pageup'`, `'pagedown'` | The `page up` and `page down` keys |\n",
    "| `'home'`, `'end'` | The home and end keys |\n",
    "| `'up'`, `'down'`, `'left'`, `'right'` | The `up`, `down`, `left`, and `right arrow` keys |\n",
    "| `'f1'`, `'f2'`, `'f3'`, and so on | The `F1` to `F12` keys |\n",
    "| `'capslock'`, '`numlock'`, `'scrolllock'` | The `caps lock`, `num lock`, and `scroll lock` keys |\n",
    "| `'pause'` | The `pause` key |\n",
    "| `'volumemute'`, `'volumedown'`, `'volumeup'` | The `mute`, `volume down`, and `volume up` keys (some keyboards do not have these keys, but your operating system will still be able to understand these simulated keypresses) |\n",
    "| `'insert'` | The `ins` or `insert` key |\n",
    "| `'printscreen'` | The `prtsc` or `print screen` key |\n",
    "| `'winleft'`, `'winright'`  | The `left` and `right win` keys (on Windows) |\n",
    "| `'command'` | The `Command` key (on macOS) |\n",
    "| `'option'` | The `option` key (on macOS) |"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5ce55d6e",
   "metadata": {},
   "source": [
    "#### Pressing and Releasing the Keyboard\n",
    "\n",
    "Much like the `mouseDown()` and `mouseUp()` functions, `pyautogui.keyDown()` and `pyautogui.keyUp()` will send virtual keypresses and releases to the computer. They are passed a keyboard key string for their argument. For convenience, `PyAutoGUI` provides the `pyautogui.press()` function, which calls both of these functions to simulate a complete keypress."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "358a573d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# type a dollar sign character (obtained by holding the shiFt key and pressing 4):\n",
    "pyautogui.keyDown('shift'); pyautogui.press('4'); pyautogui.keyUp('shift')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "c399b1a3",
   "metadata": {},
   "source": [
    "#### Hotkey Combinations\n",
    "\n",
    "A *hotkey* or *shortcut* is a combination of keypresses to invoke some application function. The common hotkey for copying a selection is `ctrl-C`. The user presses and holds the `ctrl` key, then presses the `C` key, and then releases the `C` and `ctrl` keys."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "fcab1e38",
   "metadata": {},
   "outputs": [],
   "source": [
    "pyautogui.keyDown('ctrl')\n",
    "pyautogui.keyDown('c')\n",
    "pyautogui.keyUp('c')\n",
    "pyautogui.keyUp('ctrl')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6cb5c1ef",
   "metadata": {},
   "source": [
    "This is rather complicated. Instead, use the `pyautogui.hotkey()` function, which takes multiple keyboard key string arguments, presses them in order, and releases them in the reverse order."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "96745158",
   "metadata": {},
   "outputs": [],
   "source": [
    "pyautogui.hotkey('ctrl', 'c')"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bcccec1e",
   "metadata": {},
   "source": [
    "### Setting Up Your GUI Automation Scripts\n",
    "\n",
    "- Use the same screen resolution each time you run the script so that the position of windows doesn’t change.\n",
    "- The application window that your script clicks should be maximized so that its buttons and menus are in the same place each time you run the script.\n",
    "- Add generous pauses while waiting for content to load; you don’t want your script to begin clicking before the application is ready.\n",
    "- Use locateOnScreen() to find buttons and menus to click, rather than relying on XY coordinates. If your script can’t find the thing it needs to click, stop the program rather than let it continue blindly clicking.\n",
    "- Use getWindowsWithTitle() to ensure that the application window you think your script is clicking on exists, and use the activate() method to put that window in the foreground.\n",
    "- Use the logging module from Chapter 11 to keep a log file of what your script has done. This way, if you have to stop your script halfway through a process, you can change it to pick up from where it left off.\n",
    "- Add as many checks as you can to your script. Think about how it could fail if an unexpected pop-up window appears or if your computer loses its internet connection.\n",
    "- You may want to supervise the script when it first begins to ensure that it’s working correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "97fe5e2d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "10 9 8 7 6 5 4 3 2 1 \n"
     ]
    }
   ],
   "source": [
    "import pyautogui\n",
    "\n",
    "pyautogui.sleep(3) # Pauses the program for 3 seconds.\n",
    "pyautogui.countdown(10) # Counts down over 10 seconds."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "0e3c49ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting in 3 2 1 \n"
     ]
    }
   ],
   "source": [
    "print('Starting in ', end=''); pyautogui.countdown(3)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "d48fea04",
   "metadata": {},
   "source": [
    "### Project: Automatic Form Filler\n",
    "\n",
    "https://autbor.com/form"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "e6d15e2f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Ensure that the browser window is active and the form is loaded!\n",
      ">>> 5 SECOND PAUSE TO LET USER PRESS CTRL-C <<<\n",
      "5 4 3 2 1 \n",
      "Entering Alice info...\n",
      "Submitted form.\n",
      "Restarting in 5 4 3 2 1 \n",
      ">>> 5 SECOND PAUSE TO LET USER PRESS CTRL-C <<<\n",
      "5 4 3 2 1 \n",
      "Entering Bob info...\n",
      "Submitted form.\n",
      "Restarting in 5 4 3 2 1 \n",
      ">>> 5 SECOND PAUSE TO LET USER PRESS CTRL-C <<<\n",
      "5 4 3 2 1 \n",
      "Entering Carol info...\n",
      "Submitted form.\n",
      "Restarting in 5 4 3 2 1 \n",
      ">>> 5 SECOND PAUSE TO LET USER PRESS CTRL-C <<<\n",
      "5 4 3 2 1 \n",
      "Entering Alex Murphy info...\n",
      "Submitted form.\n",
      "Restarting in 5 4 3 2 1 \n"
     ]
    }
   ],
   "source": [
    "# Automatically fills in the form at autbor.com/form\n",
    "\n",
    "import time\n",
    "import pyautogui\n",
    "\n",
    "formData = [{'name': 'Alice', 'fear': 'eavesdroppers', 'source': 'wand', 'robocop': 4, 'comments': 'Tell Bob I said hi.'},\n",
    "            {'name': 'Bob', 'fear': 'bees', 'source': 'amulet', 'robocop': 4, 'comments': 'n/a'},\n",
    "            {'name': 'Carol', 'fear': 'puppets', 'source': 'crystal ball', 'robocop': 1, 'comments': 'Please take the puppets out of the break room.'},\n",
    "            {'name': 'Alex Murphy', 'fear': 'ED-209', 'source': 'money', 'robocop': 5, 'comments': 'Protect the innocent. Serve the public trust. Uphold the law.'},\n",
    "            ]\n",
    "\n",
    "# pyautogui.PAUSE = 0.5\n",
    "print('Ensure that the browser window is active and the form is loaded!')\n",
    "\n",
    "for person in formData:\n",
    "    # Give the user a chance to kill the script\n",
    "    print('>>> 5 SECOND PAUSE TO LET USER PRESS CTRL-C <<<')\n",
    "    pyautogui.countdown(5)\n",
    "\n",
    "    print(f\"Entering {person['name']} info...\")\n",
    "    pyautogui.write(['\\t']*4)\n",
    "\n",
    "    # Fill out Name field\n",
    "    pyautogui.write(person['name'] + '\\t')\n",
    "\n",
    "    # Fill out Greatest Fear(s) field\n",
    "    pyautogui.write(person['fear'] + '\\t')\n",
    "\n",
    "    # Fill out Source of Wizard Powers field\n",
    "    if person['source'] == 'wand':\n",
    "        pyautogui.write(['down', '\\n', '\\t'], 0.5)\n",
    "    elif person['source'] == 'amulet':\n",
    "        pyautogui.write(['down']*2 + ['\\n', '\\t'], 0.5)\n",
    "    elif person['source'] == 'crystal ball':\n",
    "        pyautogui.write(['down']*3 + ['\\n', '\\t'], 0.5)\n",
    "    elif person['source'] == 'money':\n",
    "        pyautogui.write(['down']*4 + ['\\n', '\\t'], 0.5)\n",
    "\n",
    "    # Fill out Robocop field\n",
    "    if person['robocop'] == 1:\n",
    "        pyautogui.write([' '] + ['\\t']*2, 0.5)\n",
    "    elif person['robocop'] == 2:\n",
    "        pyautogui.write(['right'] + ['\\t']*2, 0.5)\n",
    "    elif person['robocop'] == 3:\n",
    "        pyautogui.write(['right']*2 + ['\\t']*2, 0.5)\n",
    "    elif person['robocop'] == 4:\n",
    "        pyautogui.write(['right']*3 + ['\\t']*2, 0.5)\n",
    "    elif person['robocop'] == 5:\n",
    "        pyautogui.write(['right']*4 + ['\\t']*2, 0.5)\n",
    "\n",
    "    # Fill out Additional comments\n",
    "    pyautogui.write(person['comments'] + '\\t')\n",
    "\n",
    "    # \"Click\" Submit button by pressing Enter.\n",
    "    time.sleep(0.5) # Wait for the button to activate.\n",
    "    pyautogui.press('enter')\n",
    "\n",
    "    # Wait until form page has loaded.\n",
    "    print('Submitted form.')\n",
    "    print(\"Restarting in \", end=\"\")\n",
    "    pyautogui.countdown(5)\n",
    "\n",
    "    # \"Click\" the \"Submit another response\" link\n",
    "    pyautogui.write(['\\t', '\\n'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8488d0ee",
   "metadata": {},
   "source": [
    "### Displaying Message Boxes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b102d1b4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'OK'"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pyautogui\n",
    "\n",
    "pyautogui.alert(\"This   is a message\", 'Important')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a5bbaee0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'OK'"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyautogui.confirm(\"Do you want to continue?\")  # Click Cancel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "0ce5b519",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Baroq'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyautogui.prompt(\"What is your cat's name?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "18b1b2c0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'secretpass'"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pyautogui.password(\"What is the password?\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3acea842",
   "metadata": {},
   "source": [
    "### Practice Projects"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "341c398b",
   "metadata": {},
   "source": [
    "#### Looking Busy\n",
    "\n",
    "Many instant messaging programs determine whether you are idle, or away from your computer, by detecting a lack of mouse movement over some period of time — say, 10 minutes. Maybe you’re away from your computer but don’t want others to see your instant messenger status go into idle mode. Write a script to nudge your mouse cursor slightly every 10 seconds. The nudge should be small and infrequent enough so that it won’t get in the way if you do happen to need to use your computer while the script is running."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "98489768",
   "metadata": {},
   "source": [
    "#### Using the Clipboard to Read a Text Field\n",
    "\n",
    "While you can send keystrokes to an application’s text fields with `pyautogui.write()`, you can’t use `PyAutoGUI` alone to read the text already inside a text field. This is where the `Pyperclip` module can help. You can use `PyAutoGUI` to obtain the window for a text editor such as Mu or Notepad, bring it to the front of the screen by clicking on it, click inside the text field, and then send the `ctrl-A` hotkey to “select all” and `ctrl-C` hotkey to “copy to clipboard.” Your Python script can then read the clipboard text by running `import pyperclip` and `pyperclip.paste()`.\n",
    "\n",
    "Write a program that follows this procedure for copying the text from a window’s text fields. Use `pyautogui.getWindowsWithTitle('Notepad')` (or whichever text editor you choose) to obtain a Window object. The top and left attributes of this Window object can tell you where this window is, while the `activate()` method will ensure it is at the front of the screen. You can then click the main text field of the text editor by adding, say,  100 or 200 pixels to the top and left attribute values with `pyautogui.click()` to put the keyboard focus there. Call `pyautogui.hotkey('ctrl', 'a')` and `pyautogui.hotkey('ctrl', 'c')` to select all the text and copy it to the clipboard. Finally, call `pyperclip.paste()` to retrieve the text from the clipboard and paste it into your Python program. From there, you can use this string however you want, but just pass it to `print()` for now.\n",
    "\n",
    "Note that the window functions of `PyAutoGUI` only work on Windows as of PyAutoGUI version 1.0.0, and not on macOS or Linux."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "333c294b",
   "metadata": {},
   "source": [
    "#### Instant Messenger Bot\n",
    "\n",
    "Google Talk, Skype, Yahoo Messenger, AIM, and other instant messaging applications often use proprietary protocols that make it difficult for others to write Python modules that can interact with these programs. But even these proprietary protocols can’t stop you from writing a GUI automation tool.\n",
    "\n",
    "The Google Talk application has a search bar that lets you enter a username on your friend list and open a messaging window when you press enter. The keyboard focus automatically moves to the new window. Other instant messenger applications have similar ways to open new message windows. Write a program that will automatically send out a notification message to a select group of people on your friend list. Your program may have to deal with exceptional cases, such as friends being offline, the chat window appearing at different coordinates on the screen, or confirmation boxes that interrupt your messaging. Your program will have to take screenshots to guide its GUI interaction and adopt ways of detecting when its virtual keystrokes aren’t being sent.\n",
    "\n",
    "*NOTE: You may want to set up some fake test accounts so that you don’t accidentally spam your real friends while writing this program.*"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "5a700f41",
   "metadata": {},
   "source": [
    "#### Game-Playing Bot Tutorial\n",
    "\n",
    "There is a great tutorial titled “How to Build a Python Bot That Can Play Web Games” that you can find a link to at https://nostarch.com/automatestuff2/. This tutorial explains how to create a GUI automation program in Python that plays a Flash game called Sushi Go Round. The game involves clicking the correct ingredient buttons to fill customers’ sushi orders. The faster you fill orders without mistakes, the more points you get. This is a perfectly suited task for a GUI automation program—and a way to cheat to a high score! The tutorial covers many of the same topics that this chapter covers but also includes descriptions of PyAutoGUI’s basic image recognition features. The source code for this bot is at https://github.com/asweigart/sushigoroundbot/ and a video of the bot playing the game is at https://youtu.be/lfk_T6VKhTE."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.5"
  },
  "vscode": {
   "interpreter": {
    "hash": "2c7666c1320a956fa05893d10993164efa76d3af81fc7470d6f4943526f3df36"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}