Boring_Python_Chapter_7.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1>Chapter 7: PATTERN MATCHING WITH REGULAR EXPRESSIONS</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may be familiar with searching for text by pressing CTRL-F and entering the words you’re looking for. Regular expressions go one step further: they allow you to specify a pattern of text to search for. You may not know a business’s exact phone number, but if you live in the United States or Canada, you know it will be three digits, followed by a hyphen, and then four more digits (and optionally, a three-digit area code at the start). This is how you, as a human, know a phone number when you see it: 415-555-1234 is a phone number, but 4,155,551,234 is not."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also recognize all sorts of other text patterns every day: email addresses have @ symbols in the middle, US social security numbers have nine digits and two hyphens, website URLs often have periods and forward slashes, news headlines use title case, social media hashtags begin with # and contain no spaces, and more."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regular expressions are helpful, but few non-programmers know about them even though most modern text editors and word processors, such as Microsoft Word or OpenOffice, have find and find-and-replace features that can search based on regular expressions. Regular expressions are huge time-savers, not just for software users but also for programmers. In fact, tech writer Cory Doctorow argues that we should be teaching regular expressions even before programming:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Knowing [regular expressions] can mean the difference between solving a problem in 3 steps and solving it in 3,000 steps. When you’re a nerd, you forget that the problems you solve with a couple keystrokes can take other people days of tedious, error-prone work to slog through.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this chapter, you’ll start by writing a program to find text patterns without using regular expressions and then see how to use regular expressions to make the code much less bloated. I’ll show you basic matching with regular expressions and then move on to some more powerful features, such as string substitution and creating your own character classes. Finally, at the end of the chapter, you’ll write a program that can automatically extract phone numbers and email addresses from a block of text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding Patterns of Text Without Regular Expressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say you want to find an American phone number in a string. You know the pattern if you’re American: three numbers, a hyphen, three numbers, a hyphen, and four numbers. Here’s an example: 415-555-4242."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let’s use a function named isPhoneNumber() to check whether a string matches this pattern, returning either True or False. Open a new file editor tab and enter the following code; then save the file as isPhoneNumber.py:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def isPhoneNumber(text):\n",
    "    if len(text) != 12:\n",
    "         return False\n",
    "    for i in range(0, 3):\n",
    "        if not text[i].isdecimal():\n",
    "             return False\n",
    "    if text[3] != '-':\n",
    "         return False\n",
    "    for i in range(4, 7):\n",
    "        if not text[i].isdecimal():\n",
    "             return False\n",
    "    if text[7] != '-':\n",
    "         return False\n",
    "    for i in range(8, 12):\n",
    "        if not text[i].isdecimal():\n",
    "             return False\n",
    "    return True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Is 415-555-4242 a phone number?\n",
      "True\n",
      "Is Moshi moshi a phone number?\n",
      "False\n"
     ]
    }
   ],
   "source": [
    "print('Is 415-555-4242 a phone number?')\n",
    "print(isPhoneNumber('415-555-4242'))\n",
    "print('Is Moshi moshi a phone number?')\n",
    "print(isPhoneNumber('Moshi moshi'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The isPhoneNumber() function has code that does several checks to see whether the string in text is a valid phone number. If any of these checks fail, the function returns False. First the code checks that the string is exactly 12 characters ➊. Then it checks that the area code (that is, the first three characters in text) consists of only numeric characters ➋. The rest of the function checks that the string follows the pattern of a phone number: the number must have the first hyphen after the area code ➌, three more numeric characters ➍, then another hyphen ➎, and finally four more numbers ➏. If the program execution manages to get past all the checks, it returns True ➐."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Calling isPhoneNumber() with the argument '415-555-4242' will return True. Calling isPhoneNumber() with 'Moshi moshi' will return False; the first test fails because 'Moshi moshi' is not 12 characters long.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you wanted to find a phone number within a larger string, you would have to add even more code to find the phone number pattern. Replace the last four print() function calls in isPhoneNumber.py with the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Phone number found: 415-555-1011\n",
      "Phone number found: 415-555-9999\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "message = 'Call me at 415-555-1011 tomorrow. 415-555-9999 is my office.'\n",
    "for i in range(len(message)):\n",
    "    chunk = message[i:i+12]\n",
    "    if isPhoneNumber(chunk):\n",
    "            print('Phone number found: ' + chunk)\n",
    "print('Done')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On each iteration of the for loop, a new chunk of 12 characters from message is assigned to the variable chunk ➊. For example, on the first iteration, i is 0, and chunk is assigned message[0:12] (that is, the string 'Call me at 4'). On the next iteration, i is 1, and chunk is assigned message[1:13] (the string 'all me at 41'). In other words, on each iteration of the for loop, chunk takes on the following values:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "'Call me at 4'\n",
    ", 'all me at 41'\n",
    ", 'll me at 415'\n",
    ", 'l me at 415-'\n",
    " . . . and so on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You pass chunk to isPhoneNumber() to see whether it matches the phone number pattern ➋, and if so, you print the chunk.\n",
    "\n",
    "Continue to loop through message, and eventually the 12 characters in chunk will be a phone number. The loop goes through the entire string, testing each 12-character piece and printing any chunk it finds that satisfies isPhoneNumber(). Once we’re done going through message, we print Done.\n",
    "\n",
    "While the string in message is short in this example, it could be millions of characters long and the program would still run in less than a second. A similar program that finds phone numbers using regular expressions would also run in less than a second, but regular expressions make it quicker to write these programs."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Finding Patterns of Text with Regular Expressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous phone number–finding program works, but it uses a lot of code to do something limited: the isPhoneNumber() function is 17 lines but can find only one pattern of phone numbers. What about a phone number formatted like 415.555.4242 or (415) 555-4242? What if the phone number had an extension, like 415-555-4242 x99? The isPhoneNumber() function would fail to validate them. You could add yet more code for these additional patterns, but there is an easier way.\n",
    "\n",
    "Regular expressions, called regexes for short, are descriptions for a pattern of text. For example, a \\d in a regex stands for a digit character—that is, any single numeral from 0 to 9. The regex \\d\\d\\d-\\d\\d\\d-\\d\\d\\d\\d is used by Python to match the same text pattern the previous isPhoneNumber() function did: a string of three numbers, a hyphen, three more numbers, another hyphen, and four numbers. Any other string would not match the \\d\\d\\d-\\d\\d\\d-\\d\\d\\d\\d regex.\n",
    "\n",
    "But regular expressions can be much more sophisticated. For example, adding a 3 in braces ({3}) after a pattern is like saying, “Match this pattern three times.” So the slightly shorter regex \\d{3}-\\d{3}-\\d{4} also matches the correct phone number format."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating Regex Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All the regex functions in Python are in the re module. Enter the following into the interactive shell to import this module:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    " import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Passing a string value representing your regular expression to re.compile() returns a Regex pattern object (or simply, a Regex object)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To create a Regex object that matches the phone number pattern, enter the following into the interactive shell. (Remember that \\d means “a digit character” and \\d\\d\\d-\\d\\d\\d-\\d\\d\\d\\d is the regular expression for a phone number pattern.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    " phoneNumRegex = re.compile(r'\\d\\d\\d-\\d\\d\\d-\\d\\d\\d\\d')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now the phoneNumRegex variable contains a Regex object."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Matching Regex Objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Regex object’s search() method searches the string it is passed for any matches to the regex. The search() method will return None if the regex pattern is not found in the string. If the pattern is found, the search() method returns a Match object, which have a group() method that will return the actual matched text from the searched string. (I’ll explain groups shortly.) For example, enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Phone number found: 415-555-4242\n"
     ]
    }
   ],
   "source": [
    ">>> phoneNumRegex = re.compile(r'\\d\\d\\d-\\d\\d\\d-\\d\\d\\d\\d')\n",
    ">>> mo = phoneNumRegex.search('My number is 415-555-4242.')\n",
    ">>> print('Phone number found: ' + mo.group())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mo variable name is just a generic name to use for Match objects. This example might seem complicated at first, but it is much shorter than the earlier isPhoneNumber.py program and does the same thing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, we pass our desired pattern to re.compile() and store the resulting Regex object in phoneNumRegex. Then we call search() on phoneNumRegex and pass search() the string we want to match for during the search. The result of the search gets stored in the variable mo. In this example, we know that our pattern will be found in the string, so we know that a Match object will be returned. Knowing that mo contains a Match object and not the null value None, we can call group() on mo to return the match. Writing mo.group() inside our print() function call displays the whole match, 415-555-4242."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Review of Regular Expression Matching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Import the regex module with import re.\n",
    "2. Create a Regex object with the re.compile() function. (Remember to use a raw string.)\n",
    "3. Pass the string you want to search into the Regex object’s search() method. This returns a Match object.\n",
    "4. Call the Match object’s group() method to return a string of the actual matched text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# More Pattern Matching with Regular Expressions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you know the basic steps for creating and finding regular expression objects using Python, you’re ready to try some of their more powerful pattern-matching capabilities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Grouping with Parentheses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say you want to separate the area code from the rest of the phone number. Adding parentheses will create groups in the regex: (\\d\\d\\d)-(\\d\\d\\d-\\d\\d\\d\\d). Then you can use the group() match object method to grab the matching text from just one group.\n",
    "\n",
    "The first set of parentheses in a regex string will be group 1. The second set will be group 2. By passing the integer 1 or 2 to the group() match object method, you can grab different parts of the matched text. Passing 0 or nothing to the group() method will return the entire matched text. Enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'415'"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> phoneNumRegex = re.compile(r'(\\d\\d\\d)-(\\d\\d\\d-\\d\\d\\d\\d)')\n",
    ">>> mo = phoneNumRegex.search('My number is 415-555-4242.')\n",
    ">>> mo.group(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'555-4242'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo.group(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'415-555-4242'"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo.group(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'415-555-4242'"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo.group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to retrieve all the groups at once, use the groups() method—note the plural form for the name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "('415', '555-4242')"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo.groups()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "415\n"
     ]
    }
   ],
   "source": [
    ">>> areaCode, mainNumber = mo.groups()\n",
    ">>> print(areaCode)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "555-4242\n"
     ]
    }
   ],
   "source": [
    ">>> print(mainNumber)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since mo.groups() returns a tuple of multiple values, you can use the multiple-assignment trick to assign each value to a separate variable, as in the previous areaCode, mainNumber = mo.groups() line.\n",
    "\n",
    "Parentheses have a special meaning in regular expressions, but what do you do if you need to match a parenthesis in your text? For instance, maybe the phone numbers you are trying to match have the area code set in parentheses. In this case, you need to escape the ( and ) characters with a backslash. Enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'(415)'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> phoneNumRegex = re.compile(r'(\\(\\d\\d\\d\\)) (\\d\\d\\d-\\d\\d\\d\\d)')\n",
    ">>> mo = phoneNumRegex.search('My phone number is (415) 555-4242.')\n",
    ">>> mo.group(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'555-4242'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo.group(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \\( and \\) escape characters in the raw string passed to re.compile() will match actual parenthesis characters. In regular expressions, the following characters have special meanings:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ".  ^  $  *  +  ?  {  }  [  ]  \\  |  (  )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to detect these characters as part of your text pattern, you need to escape them with a backslash:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\.  \\^  \\$  \\*  \\+  \\?  \\{  \\}  \\[  \\]  \\\\  \\|  \\(  \\)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Make sure to double-check that you haven’t mistaken escaped parentheses \\( and \\) for parentheses ( and ) in a regular expression. If you receive an error message about “missing )” or “unbalanced parenthesis,” you may have forgotten to include the closing unescaped parenthesis for a group, like in this example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "ename": "error",
     "evalue": "missing ), unterminated subpattern at position 0",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31merror\u001b[0m                                     Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-26-051679312224>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[0mre\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcompile\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34mr'(\\(Parentheses\\)'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[1;32mC:\\ProgramData\\Anaconda3\\lib\\re.py\u001b[0m in \u001b[0;36mcompile\u001b[1;34m(pattern, flags)\u001b[0m\n\u001b[0;32m    232\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mcompile\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpattern\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mflags\u001b[0m\u001b[1;33m=\u001b[0m\u001b[1;36m0\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    233\u001b[0m     \u001b[1;34m\"Compile a regular expression pattern, returning a Pattern object.\"\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 234\u001b[1;33m     \u001b[1;32mreturn\u001b[0m \u001b[0m_compile\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpattern\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mflags\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    235\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    236\u001b[0m \u001b[1;32mdef\u001b[0m \u001b[0mpurge\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\ProgramData\\Anaconda3\\lib\\re.py\u001b[0m in \u001b[0;36m_compile\u001b[1;34m(pattern, flags)\u001b[0m\n\u001b[0;32m    284\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0msre_compile\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0misstring\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpattern\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    285\u001b[0m         \u001b[1;32mraise\u001b[0m \u001b[0mTypeError\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"first argument must be string or compiled pattern\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 286\u001b[1;33m     \u001b[0mp\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0msre_compile\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcompile\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mpattern\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mflags\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    287\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;33m(\u001b[0m\u001b[0mflags\u001b[0m \u001b[1;33m&\u001b[0m \u001b[0mDEBUG\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    288\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[0mlen\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0m_cache\u001b[0m\u001b[1;33m)\u001b[0m \u001b[1;33m>=\u001b[0m \u001b[0m_MAXCACHE\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\ProgramData\\Anaconda3\\lib\\sre_compile.py\u001b[0m in \u001b[0;36mcompile\u001b[1;34m(p, flags)\u001b[0m\n\u001b[0;32m    762\u001b[0m     \u001b[1;32mif\u001b[0m \u001b[0misstring\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mp\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    763\u001b[0m         \u001b[0mpattern\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0mp\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 764\u001b[1;33m         \u001b[0mp\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0msre_parse\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mparse\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mp\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mflags\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    765\u001b[0m     \u001b[1;32melse\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    766\u001b[0m         \u001b[0mpattern\u001b[0m \u001b[1;33m=\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\ProgramData\\Anaconda3\\lib\\sre_parse.py\u001b[0m in \u001b[0;36mparse\u001b[1;34m(str, flags, pattern)\u001b[0m\n\u001b[0;32m    928\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    929\u001b[0m     \u001b[1;32mtry\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m--> 930\u001b[1;33m         \u001b[0mp\u001b[0m \u001b[1;33m=\u001b[0m \u001b[0m_parse_sub\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0msource\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mpattern\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mflags\u001b[0m \u001b[1;33m&\u001b[0m \u001b[0mSRE_FLAG_VERBOSE\u001b[0m\u001b[1;33m,\u001b[0m \u001b[1;36m0\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m    931\u001b[0m     \u001b[1;32mexcept\u001b[0m \u001b[0mVerbose\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    932\u001b[0m         \u001b[1;31m# the VERBOSE flag was switched on inside the pattern.  to be\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\ProgramData\\Anaconda3\\lib\\sre_parse.py\u001b[0m in \u001b[0;36m_parse_sub\u001b[1;34m(source, state, verbose, nested)\u001b[0m\n\u001b[0;32m    424\u001b[0m     \u001b[1;32mwhile\u001b[0m \u001b[1;32mTrue\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    425\u001b[0m         itemsappend(_parse(source, state, verbose, nested + 1,\n\u001b[1;32m--> 426\u001b[1;33m                            not nested and not items))\n\u001b[0m\u001b[0;32m    427\u001b[0m         \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0msourcematch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\"|\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    428\u001b[0m             \u001b[1;32mbreak\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;32mC:\\ProgramData\\Anaconda3\\lib\\sre_parse.py\u001b[0m in \u001b[0;36m_parse\u001b[1;34m(source, state, verbose, nested, first)\u001b[0m\n\u001b[0;32m    817\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[0msource\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmatch\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m\")\"\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    818\u001b[0m                 raise source.error(\"missing ), unterminated subpattern\",\n\u001b[1;32m--> 819\u001b[1;33m                                    source.tell() - start)\n\u001b[0m\u001b[0;32m    820\u001b[0m             \u001b[1;32mif\u001b[0m \u001b[0mgroup\u001b[0m \u001b[1;32mis\u001b[0m \u001b[1;32mnot\u001b[0m \u001b[1;32mNone\u001b[0m\u001b[1;33m:\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m    821\u001b[0m                 \u001b[0mstate\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mclosegroup\u001b[0m\u001b[1;33m(\u001b[0m\u001b[0mgroup\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mp\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31merror\u001b[0m: missing ), unterminated subpattern at position 0"
     ]
    }
   ],
   "source": [
    ">>> re.compile(r'(\\(Parentheses\\)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The error message tells you that there is an opening parenthesis at index 0 of the r'(\\(Parentheses\\)' string that is missing its corresponding closing parenthesis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Matching Multiple Groups with the Pipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The | character is called a pipe. You can use it anywhere you want to match one of many expressions. For example, the regular expression r'Batman|Tina Fey' will match either 'Batman' or 'Tina Fey'.\n",
    "\n",
    "When both Batman and Tina Fey occur in the searched string, the first occurrence of matching text will be returned as the Match object. Enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Batman'"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> heroRegex = re.compile (r'Batman|Tina Fey')\n",
    ">>> mo1 = heroRegex.search('Batman and Tina Fey')\n",
    ">>> mo1.group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Tina Fey'"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo2 = heroRegex.search('Tina Fey and Batman')\n",
    ">>> mo2.group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use the pipe to match one of several patterns as part of your regex. For example, say you wanted to match any of the strings 'Batman', 'Batmobile', 'Batcopter', and 'Batbat'. Since all these strings start with Bat, it would be nice if you could specify that prefix only once. This can be done with parentheses. Enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Batmobile'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> batRegex = re.compile(r'Bat(man|mobile|copter|bat)')\n",
    ">>> mo = batRegex.search('Batmobile lost a wheel')\n",
    ">>> mo.group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'mobile'"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo.group(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The method call mo.group() returns the full matched text 'Batmobile', while mo.group(1) returns just the part of the matched text inside the first parentheses group, 'mobile'. By using the pipe character and grouping parentheses, you can specify several alternative patterns you would like your regex to match.\n",
    "\n",
    "If you need to match an actual pipe character, escape it with a backslash, like \\|."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optional Matching with the Question Mark"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes there is a pattern that you want to match only optionally. That is, the regex should find a match regardless of whether that bit of text is there. The ? character flags the group that precedes it as an optional part of the pattern. For example, enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Batman'"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> batRegex = re.compile(r'Bat(wo)?man')\n",
    ">>> mo1 = batRegex.search('The Adventures of Batman')\n",
    ">>> mo1.group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Batwoman'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo2 = batRegex.search('The Adventures of Batwoman')\n",
    ">>> mo2.group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The (wo)? part of the regular expression means that the pattern wo is an optional group. The regex will match text that has zero instances or one instance of wo in it. This is why the regex matches both 'Batwoman' and 'Batman'.\n",
    "\n",
    "Using the earlier phone number example, you can make the regex look for phone numbers that do or do not have an area code. Enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'415-555-4242'"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> phoneRegex = re.compile(r'(\\d\\d\\d-)?\\d\\d\\d-\\d\\d\\d\\d')\n",
    ">>> mo1 = phoneRegex.search('My number is 415-555-4242')\n",
    ">>> mo1.group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'555-4242'"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo2 = phoneRegex.search('My number is 555-4242')\n",
    ">>> mo2.group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can think of the ? as saying, “Match zero or one of the group preceding this question mark.”\n",
    "\n",
    "If you need to match an actual question mark character, escape it with \\?.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Matching Zero or More with the Star"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The * (called the star or asterisk) means “match zero or more”—the group that precedes the star can occur any number of times in the text. It can be completely absent or repeated over and over again. Let’s look at the Batman example again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Batman'"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> batRegex = re.compile(r'Bat(wo)*man')\n",
    ">>> mo1 = batRegex.search('The Adventures of Batman')\n",
    ">>> mo1.group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Batwoman'"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo2 = batRegex.search('The Adventures of Batwoman')\n",
    ">>> mo2.group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Batwowowowoman'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo3 = batRegex.search('The Adventures of Batwowowowoman')\n",
    ">>> mo3.group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For 'Batman', the (wo)* part of the regex matches zero instances of wo in the string; for 'Batwoman', the (wo)* matches one instance of wo; and for 'Batwowowowoman', (wo)* matches four instances of wo.\n",
    "\n",
    "If you need to match an actual star character, prefix the star in the regular expression with a backslash, \\*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Matching One or More with the Plus"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While * means “match zero or more,” the + (or plus) means “match one or more.” Unlike the star, which does not require its group to appear in the matched string, the group preceding a plus must appear at least once. It is not optional. Enter the following into the interactive shell, and compare it with the star regexes in the previous section:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Batwoman'"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> batRegex = re.compile(r'Bat(wo)+man')\n",
    ">>> mo1 = batRegex.search('The Adventures of Batwoman')\n",
    ">>> mo1.group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Batwowowowoman'"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo2 = batRegex.search('The Adventures of Batwowowowoman')\n",
    ">>> mo2.group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo3 = batRegex.search('The Adventures of Batman')\n",
    ">>> mo3 == None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The regex Bat(wo)+man will not match the string 'The Adventures of Batman', because at least one wo is required by the plus sign.\n",
    "\n",
    "If you need to match an actual plus sign character, prefix the plus sign with a backslash to escape it: \\+."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Matching Specific Repetitions with Braces"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you have a group that you want to repeat a specific number of times, follow the group in your regex with a number in braces. For example, the regex (Ha){3} will match the string 'HaHaHa', but it will not match 'HaHa', since the latter has only two repeats of the (Ha) group.\n",
    "\n",
    "Instead of one number, you can specify a range by writing a minimum, a comma, and a maximum in between the braces. For example, the regex (Ha){3,5} will match 'HaHaHa', 'HaHaHaHa', and 'HaHaHaHaHa'.\n",
    "\n",
    "You can also leave out the first or second number in the braces to leave the minimum or maximum unbounded. For example, (Ha){3,} will match three or more instances of the (Ha) group, while (Ha){,5} will match zero to five instances. Braces can help make your regular expressions shorter. These two regular expressions match identical patterns:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. (Ha){3}\n",
    "2. (Ha)(Ha)(Ha)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And these two regular expressions also match identical patterns:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. (Ha){3,5}\n",
    "2. ((Ha)(Ha)(Ha))|((Ha)(Ha)(Ha)(Ha))|((Ha)(Ha)(Ha)(Ha)(Ha))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'HaHaHa'"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> haRegex = re.compile(r'(Ha){3}')\n",
    ">>> mo1 = haRegex.search('HaHaHa')\n",
    ">>> mo1.group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo2 = haRegex.search('Ha')\n",
    ">>> mo2 == None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here, (Ha){3} matches 'HaHaHa' but not 'Ha'. Since it doesn’t match 'Ha', search() returns None."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Greedy and Non-greedy Matching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since (Ha){3,5} can match three, four, or five instances of Ha in the string 'HaHaHaHaHa', you may wonder why the Match object’s call to group() in the previous brace example returns 'HaHaHaHaHa' instead of the shorter possibilities. After all, 'HaHaHa' and 'HaHaHaHa' are also valid matches of the regular expression (Ha){3,5}.\n",
    "\n",
    "Python’s regular expressions are greedy by default, which means that in ambiguous situations they will match the longest string possible. The non-greedy (also called lazy) version of the braces, which matches the shortest string possible, has the closing brace followed by a question mark.\n",
    "\n",
    "Enter the following into the interactive shell, and notice the difference between the greedy and non-greedy forms of the braces searching the same string:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'HaHaHaHaHa'"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> greedyHaRegex = re.compile(r'(Ha){3,5}')\n",
    ">>> mo1 = greedyHaRegex.search('HaHaHaHaHa')\n",
    ">>> mo1.group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'HaHaHa'"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> nongreedyHaRegex = re.compile(r'(Ha){3,5}?')\n",
    ">>> mo2 = nongreedyHaRegex.search('HaHaHaHaHa')\n",
    ">>> mo2.group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the question mark can have two meanings in regular expressions: declaring a non-greedy match or flagging an optional group. These meanings are entirely unrelated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The findall() Method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to the search() method, Regex objects also have a findall() method. While search() will return a Match object of the first matched text in the searched string, the findall() method will return the strings of every match in the searched string. To see how search() returns a Match object only on the first instance of matching text, enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'415-555-9999'"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> phoneNumRegex = re.compile(r'\\d\\d\\d-\\d\\d\\d-\\d\\d\\d\\d')\n",
    ">>> mo = phoneNumRegex.search('Cell: 415-555-9999 Work: 212-555-0000')\n",
    ">>> mo.group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the other hand, findall() will not return a Match object but a list of strings—as long as there are no groups in the regular expression. Each string in the list is a piece of the searched text that matched the regular expression. Enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['415-555-9999', '212-555-0000']"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> phoneNumRegex = re.compile(r'\\d\\d\\d-\\d\\d\\d-\\d\\d\\d\\d') # has no groups\n",
    ">>> phoneNumRegex.findall('Cell: 415-555-9999 Work: 212-555-0000')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If there are groups in the regular expression, then findall() will return a list of tuples. Each tuple represents a found match, and its items are the matched strings for each group in the regex. To see findall() in action, enter the following into the interactive shell (notice that the regular expression being compiled now has groups in parentheses):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('415', '555', '9999'), ('212', '555', '0000')]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> phoneNumRegex = re.compile(r'(\\d\\d\\d)-(\\d\\d\\d)-(\\d\\d\\d\\d)') # has groups\n",
    ">>> phoneNumRegex.findall('Cell: 415-555-9999 Work: 212-555-0000')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To summarize what the findall() method returns, remember the following:\n",
    "\n",
    "When called on a regex with no groups, such as \\d\\d\\d-\\d\\d\\d-\\d\\d\\d\\d, the method findall() returns a list of string matches, such as ['415-555-9999', '212-555-0000'].\n",
    "When called on a regex that has groups, such as (\\d\\d\\d)-(\\d\\d\\d)-(\\d\\d\\d\\d), the method findall() returns a list of tuples of strings (one string for each group), such as [('415', '555', '9999'), ('212', '555', '0000')]."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Character Classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the earlier phone number regex example, you learned that \\d could stand for any numeric digit. That is, \\d is shorthand for the regular expression (0|1|2|3|4|5|6|7|8|9). There are many such shorthand character classes, as shown in Table 7-1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Shorthand Character Class      | Represents |\n",
    "| ----------- | ----------- |\n",
    "| \\d      | Any numeric digit from 0 to 9.    |\n",
    "| \\D   | Any character that is not a numeric digit from 0 to 9.        |\n",
    "| \\w | Any letter, numeric digit, or the underscore character. (Think of this as matching “word” characters.) |\n",
    "| \\W | \tAny character that is not a letter, numeric digit, or the underscore character. |\n",
    "| \\s | \tAny space, tab, or newline character. (Think of this as matching “space” characters.) |\n",
    "| \\S | \tAny character that is not a space, tab, or newline. |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Character classes are nice for shortening regular expressions. The character class [0-5] will match only the numbers 0 to 5; this is much shorter than typing (0|1|2|3|4|5). Note that while \\d matches digits and \\w matches digits, letters, and the underscore, there is no shorthand character class that matches only letters. (Though you can use the [a-zA-Z] character class, as explained next.)\n",
    "\n",
    "For example, enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['12 drummers',\n",
       " '11 pipers',\n",
       " '10 lords',\n",
       " '9 ladies',\n",
       " '8 maids',\n",
       " '7 swans',\n",
       " '6 geese',\n",
       " '5 rings',\n",
       " '4 birds',\n",
       " '3 hens',\n",
       " '2 doves',\n",
       " '1 partridge']"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> xmasRegex = re.compile(r'\\d+\\s\\w+')\n",
    ">>> xmasRegex.findall('12 drummers, 11 pipers, 10 lords, 9 ladies, 8 maids, 7 swans, 6 geese, 5 rings, 4 birds, 3 hens, 2 doves, 1 partridge')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The regular expression \\d+\\s\\w+ will match text that has one or more numeric digits (\\d+), followed by a whitespace character (\\s), followed by one or more letter/digit/underscore characters (\\w+). The findall() method returns all matching strings of the regex pattern in a list."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making Your Own Character Classes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are times when you want to match a set of characters but the shorthand character classes (\\d, \\w, \\s, and so on) are too broad. You can define your own character class using square brackets. For example, the character class [aeiouAEIOU] will match any vowel, both lowercase and uppercase. Enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['o', 'o', 'o', 'e', 'a', 'a', 'o', 'o', 'A', 'O', 'O']"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> vowelRegex = re.compile(r'[aeiouAEIOU]')\n",
    ">>> vowelRegex.findall('RoboCop eats baby food. BABY FOOD.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also include ranges of letters or numbers by using a hyphen. For example, the character class [a-zA-Z0-9] will match all lowercase letters, uppercase letters, and numbers.\n",
    "\n",
    "Note that inside the square brackets, the normal regular expression symbols are not interpreted as such. This means you do not need to escape the ., *, ?, or () characters with a preceding backslash. For example, the character class [0-5.] will match digits 0 to 5 and a period. You do not need to write it as [0-5\\.].\n",
    "\n",
    "By placing a caret character (^) just after the character class’s opening bracket, you can make a negative character class. A negative character class will match all the characters that are not in the character class. For example, enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['R',\n",
       " 'b',\n",
       " 'C',\n",
       " 'p',\n",
       " ' ',\n",
       " 't',\n",
       " 's',\n",
       " ' ',\n",
       " 'b',\n",
       " 'b',\n",
       " 'y',\n",
       " ' ',\n",
       " 'f',\n",
       " 'd',\n",
       " '.',\n",
       " ' ',\n",
       " 'B',\n",
       " 'B',\n",
       " 'Y',\n",
       " ' ',\n",
       " 'F',\n",
       " 'D',\n",
       " '.']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> consonantRegex = re.compile(r'[^aeiouAEIOU]')\n",
    ">>> consonantRegex.findall('RoboCop eats baby food. BABY FOOD.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, instead of matching every vowel, we’re matching every character that isn’t a vowel."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Caret and Dollar Sign Characters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also use the caret symbol (^) at the start of a regex to indicate that a match must occur at the beginning of the searched text. Likewise, you can put a dollar sign ($) at the end of the regex to indicate the string must end with this regex pattern. And you can use the ^ and $ together to indicate that the entire string must match the regex—that is, it’s not enough for a match to be made on some subset of the string.\n",
    "\n",
    "For example, the r'^Hello' regular expression string matches strings that begin with 'Hello'. Enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 5), match='Hello'>"
      ]
     },
     "execution_count": 56,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> beginsWithHello = re.compile(r'^Hello')\n",
    ">>> beginsWithHello.search('Hello, world!')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> beginsWithHello.search('He said hello.') == None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The r'\\d$' regular expression string matches strings that end with a numeric character from 0 to 9. Enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(16, 17), match='2'>"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> endsWithNumber = re.compile(r'\\d$')\n",
    ">>> endsWithNumber.search('Your number is 42')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> endsWithNumber.search('Your number is forty two.') == None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The r'^\\d+$' regular expression string matches strings that both begin and end with one or more numeric characters. Enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<re.Match object; span=(0, 10), match='1234567890'>"
      ]
     },
     "execution_count": 61,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> wholeStringIsNum = re.compile(r'^\\d+$')\n",
    ">>> wholeStringIsNum.search('1234567890')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 62,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> wholeStringIsNum.search('12345xyz67890') == None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> wholeStringIsNum.search('12  34567890') == None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last two search() calls in the previous interactive shell example demonstrate how the entire string must match the regex if ^ and $ are used.\n",
    "\n",
    "I always confuse the meanings of these two symbols, so I use the mnemonic “Carrots cost dollars” to remind myself that the caret comes first and the dollar sign comes last."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Wildcard Character"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The . (or dot) character in a regular expression is called a wildcard and will match any character except for a newline. For example, enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['cat', 'hat', 'sat', 'lat', 'mat']"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> atRegex = re.compile(r'.at')\n",
    ">>> atRegex.findall('The cat in the hat sat on the flat mat.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember that the dot character will match just one character, which is why the match for the text flat in the previous example matched only lat. To match an actual dot, escape the dot with a backslash: \\.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Matching Everything with Dot-Star"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes you will want to match everything and anything. For example, say you want to match the string 'First Name:', followed by any and all text, followed by 'Last Name:', and then followed by anything again. You can use the dot-star (.*) to stand in for that “anything.” Remember that the dot character means “any single character except the newline,” and the star character means “zero or more of the preceding character.”\n",
    "\n",
    "Enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Al'"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> nameRegex = re.compile(r'First Name: (.*) Last Name: (.*)')\n",
    ">>> mo = nameRegex.search('First Name: Al Last Name: Sweigart')\n",
    ">>> mo.group(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Sweigart'"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> mo.group(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dot-star uses greedy mode: It will always try to match as much text as possible. To match any and all text in a non-greedy fashion, use the dot, star, and question mark (.*?). Like with braces, the question mark tells Python to match in a non-greedy way.\n",
    "\n",
    "Enter the following into the interactive shell to see the difference between the greedy and non-greedy versions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<To serve man>'"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> nongreedyRegex = re.compile(r'<.*?>')\n",
    ">>> mo = nongreedyRegex.search('<To serve man> for dinner.>')\n",
    ">>> mo.group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<To serve man> for dinner.>'"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> greedyRegex = re.compile(r'<.*>')\n",
    ">>> mo = greedyRegex.search('<To serve man> for dinner.>')\n",
    ">>> mo.group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both regexes roughly translate to “Match an opening angle bracket, followed by anything, followed by a closing angle bracket.” But the string '<To serve man> for dinner.>' has two possible matches for the closing angle bracket. In the non-greedy version of the regex, Python matches the shortest possible string: '<To serve man>'. In the greedy version, Python matches the longest possible string: '<To serve man> for dinner.>'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Matching Newlines with the Dot Character"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dot-star will match everything except a newline. By passing re.DOTALL as the second argument to re.compile(), you can make the dot character match all characters, including the newline character.\n",
    "\n",
    "Enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Serve the public trust.'"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> noNewlineRegex = re.compile('.*')\n",
    ">>> noNewlineRegex.search('Serve the public trust.\\nProtect the innocent. \\nUphold the law.').group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Serve the public trust.\\nProtect the innocent.\\nUphold the law.'"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> newlineRegex = re.compile('.*', re.DOTALL)\n",
    ">>> newlineRegex.search('Serve the public trust.\\nProtect the innocent.\\nUphold the law.').group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The regex noNewlineRegex, which did not have re.DOTALL passed to the re.compile() call that created it, will match everything only up to the first newline character, whereas newlineRegex, which did have re.DOTALL passed to re.compile(), matches everything. This is why the newlineRegex.search() call matches the full string, including its newline characters."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Review of Regex Symbols"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This chapter covered a lot of notation, so here’s a quick review of what you learned about basic regular expression syntax:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. The ? matches zero or one of the preceding group.\n",
    "2. The * matches zero or more of the preceding group.\n",
    "3. The + matches one or more of the preceding group.\n",
    "4. The {n} matches exactly n of the preceding group.\n",
    "5. The {n,} matches n or more of the preceding group.\n",
    "6. The {,m} matches 0 to m of the preceding group.\n",
    "7. The {n,m} matches at least n and at most m of the preceding group.\n",
    "8. {n,m}? or *? or +? performs a non-greedy match of the preceding group.\n",
    "9. ^spam means the string must begin with spam.\n",
    "10. spam$ means the string must end with spam.\n",
    "11. The . matches any character, except newline characters.\n",
    "12. \\d, \\w, and \\s match a digit, word, or space character, respectively.\n",
    "13. \\D, \\W, and \\S match anything except a digit, word, or space character, respectively.\n",
    "14. [abc] matches any character between the brackets (such as a, b, or c).\n",
    "15. [^abc] matches any character that isn’t between the brackets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Case-Insensitive Matching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normally, regular expressions match text with the exact casing you specify. For example, the following regexes match completely different strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    ">>> regex1 = re.compile('RoboCop')\n",
    ">>> regex2 = re.compile('ROBOCOP')\n",
    ">>> regex3 = re.compile('robOcop')\n",
    ">>> regex4 = re.compile('RobocOp')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But sometimes you care only about matching the letters without worrying whether they’re uppercase or lowercase. To make your regex case-insensitive, you can pass re.IGNORECASE or re.I as a second argument to re.compile(). Enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'RoboCop'"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> robocop = re.compile(r'robocop', re.I)\n",
    ">>> robocop.search('RoboCop is part man, part machine, all cop.').group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'ROBOCOP'"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> robocop.search('ROBOCOP protects the innocent.').group()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'robocop'"
      ]
     },
     "execution_count": 76,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> robocop.search('Al, why does your programming book talk about robocop so much?').group()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Substituting Strings with the sub() Method"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regular expressions can not only find text patterns but can also substitute new text in place of those patterns. The sub() method for Regex objects is passed two arguments. The first argument is a string to replace any matches. The second is the string for the regular expression. The sub() method returns a string with the substitutions applied.\n",
    "\n",
    "For example, enter the following into the interactive shell:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'CENSORED gave the secret documents to CENSORED.'"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> namesRegex = re.compile(r'Agent \\w+')\n",
    ">>> namesRegex.sub('CENSORED', 'Agent Alice gave the secret documents to Agent Bob.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes you may need to use the matched text itself as part of the substitution. In the first argument to sub(), you can type \\1, \\2, \\3, and so on, to mean “Enter the text of group 1, 2, 3, and so on, in the substitution.”\n",
    "\n",
    "For example, say you want to censor the names of the secret agents by showing just the first letters of their names. To do this, you could use the regex Agent (\\w)\\w* and pass r'\\1****' as the first argument to sub(). The \\1 in that string will be replaced by whatever text was matched by group 1—that is, the (\\w) group of the regular expression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'A**** told C**** that E**** knew B**** was a double agent.'"
      ]
     },
     "execution_count": 78,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    ">>> agentNamesRegex = re.compile(r'Agent (\\w)\\w*')\n",
    ">>> agentNamesRegex.sub(r'\\1****', 'Agent Alice told Agent Carol that Agent Eve knew Agent Bob was a double agent.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Managing Complex Regexes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regular expressions are fine if the text pattern you need to match is simple. But matching complicated text patterns might require long, convoluted regular expressions. You can mitigate this by telling the re.compile() function to ignore whitespace and comments inside the regular expression string. This “verbose mode” can be enabled by passing the variable re.VERBOSE as the second argument to re.compile().\n",
    "\n",
    "Now instead of a hard-to-read regular expression like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [],
   "source": [
    "phoneRegex = re.compile(r'((\\d{3}|\\(\\d{3}\\))?(\\s|-|\\.)?\\d{3}(\\s|-|\\.)\\d{4} (\\s*(ext|x|ext.)\\s*\\d{2,5})?)')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "you can spread the regular expression over multiple lines with comments like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [],
   "source": [
    "phoneRegex = re.compile(r'''(\n",
    "    (\\d{3}|\\(\\d{3}\\))?            # area code\n",
    "    (\\s|-|\\.)?                    # separator\n",
    "    \\d{3}                         # first 3 digits\n",
    "    (\\s|-|\\.)                     # separator\n",
    "    \\d{4}                         # last 4 digits\n",
    "    (\\s*(ext|x|ext.)\\s*\\d{2,5})?  # extension\n",
    "    )''', re.VERBOSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how the previous example uses the triple-quote syntax (''') to create a multiline string so that you can spread the regular expression definition over many lines, making it much more legible.\n",
    "\n",
    "The comment rules inside the regular expression string are the same as regular Python code: the # symbol and everything after it to the end of the line are ignored. Also, the extra spaces inside the multiline string for the regular expression are not considered part of the text pattern to be matched. This lets you organize the regular expression so it’s easier to read."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Combining re.IGNORECASE, re.DOTALL, and re.VERBOSE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if you want to use re.VERBOSE to write comments in your regular expression but also want to use re.IGNORECASE to ignore capitalization? Unfortunately, the re.compile() function takes only a single value as its second argument. You can get around this limitation by combining the re.IGNORECASE, re.DOTALL, and re.VERBOSE variables using the pipe character (|), which in this context is known as the bitwise or operator.\n",
    "\n",
    "So if you want a regular expression that’s case-insensitive and includes newlines to match the dot character, you would form your re.compile() call like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [],
   "source": [
    ">>> someRegexValue = re.compile('foo', re.IGNORECASE | re.DOTALL)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Including all three options in the second argument will look like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {},
   "outputs": [],
   "source": [
    ">>> someRegexValue = re.compile('foo', re.IGNORECASE | re.DOTALL | re.VERBOSE)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This syntax is a little old-fashioned and originates from early versions of Python. The details of the bitwise operators are beyond the scope of this book, but check out the resources at https://nostarch.com/automatestuff2/ for more information. You can also pass other options for the second argument; they’re uncommon, but you can read more about them in the resources, too."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Project: Phone Number and Email Address Extractor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Say you have the boring task of finding every phone number and email address in a long web page or document. If you manually scroll through the page, you might end up searching for a long time. But if you had a program that could search the text in your clipboard for phone numbers and email addresses, you could simply press CTRL-A to select all the text, press CTRL-C to copy it to the clipboard, and then run your program. It could replace the text on the clipboard with just the phone numbers and email addresses it finds.\n",
    "\n",
    "Whenever you’re tackling a new project, it can be tempting to dive right into writing code. But more often than not, it’s best to take a step back and consider the bigger picture. I recommend first drawing up a high-level plan for what your program needs to do. Don’t think about the actual code yet—you can worry about that later. Right now, stick to broad strokes.\n",
    "\n",
    "For example, your phone and email address extractor will need to do the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Get the text off the clipboard.\n",
    "2. Find all phone numbers and email addresses in the text.\n",
    "3. Paste them onto the clipboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you can start thinking about how this might work in code. The code will need to do the following:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Use the pyperclip module to copy and paste strings.\n",
    "2. Create two regexes, one for matching phone numbers and the other for matching email addresses.\n",
    "3. Find all matches, not just the first match, of both regexes.\n",
    "4. Neatly format the matched strings into a single string to paste.\n",
    "5. Display some kind of message if no matches were found in the text."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This list is like a road map for the project. As you write the code, you can focus on each of these steps separately. Each step is fairly manageable and expressed in terms of things you already know how to do in Python."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 1: Create a Regex for Phone Numbers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, you have to create a regular expression to search for phone numbers. Create a new file, enter the following, and save it as phoneAndEmail.py:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'pyperclip'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-85-19d446e1fd14>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[1;31m# phoneAndEmail.py - Finds phone numbers and email addresses on the clipboard.\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m \u001b[1;32mimport\u001b[0m \u001b[0mpyperclip\u001b[0m\u001b[1;33m,\u001b[0m \u001b[0mre\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      5\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m phoneRegex = re.compile(r'''(\n",
      "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'pyperclip'"
     ]
    }
   ],
   "source": [
    "#! python3\n",
    "# phoneAndEmail.py - Finds phone numbers and email addresses on the clipboard.\n",
    "\n",
    "import pyperclip, re\n",
    "\n",
    "phoneRegex = re.compile(r'''(\n",
    "    (\\d{3}|\\(\\d{3}\\))?                # area code\n",
    "    (\\s|-|\\.)?                        # separator\n",
    "    (\\d{3})                           # first 3 digits\n",
    "    (\\s|-|\\.)                         # separator\n",
    "    (\\d{4})                           # last 4 digits\n",
    "    (\\s*(ext|x|ext.)\\s*(\\d{2,5}))?    # extension\n",
    "    )''', re.VERBOSE)\n",
    "\n",
    "# TODO: Create email regex.\n",
    "\n",
    "# TODO: Find matches in clipboard text.\n",
    "\n",
    "# TODO: Copy results to the clipboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The TODO comments are just a skeleton for the program. They’ll be replaced as you write the actual code.\n",
    "\n",
    "The phone number begins with an optional area code, so the area code group is followed with a question mark. Since the area code can be just three digits (that is, \\d{3}) or three digits within parentheses (that is, \\(\\d{3}\\)), you should have a pipe joining those parts. You can add the regex comment # Area code to this part of the multiline string to help you remember what (\\d{3}|\\(\\d{3}\\))? is supposed to match.\n",
    "\n",
    "The phone number separator character can be a space (\\s), hyphen (-), or period (.), so these parts should also be joined by pipes. The next few parts of the regular expression are straightforward: three digits, followed by another separator, followed by four digits. The last part is an optional extension made up of any number of spaces followed by ext, x, or ext., followed by two to five digits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 2: Create a Regex for Email Addresses"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You will also need a regular expression that can match email addresses. Make your program look like the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid character in identifier (<ipython-input-86-e14d516742cc>, line 11)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  File \u001b[1;32m\"<ipython-input-86-e14d516742cc>\"\u001b[1;36m, line \u001b[1;32m11\u001b[0m\n\u001b[1;33m    ➊ [a-zA-Z0-9._%+-]+      # username\u001b[0m\n\u001b[1;37m    ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid character in identifier\n"
     ]
    }
   ],
   "source": [
    "#! python3\n",
    "# phoneAndEmail.py - Finds phone numbers and email addresses on the clipboard.\n",
    "\n",
    "import pyperclip, re\n",
    "\n",
    "phoneRegex = re.compile(r'''(\n",
    "--snip--\n",
    "\n",
    "# Create email regex.\n",
    "emailRegex = re.compile(r'''(\n",
    "  ➊ [a-zA-Z0-9._%+-]+      # username\n",
    "  ➋ @                      # @ symbol\n",
    "  ➌ [a-zA-Z0-9.-]+         # domain name\n",
    "    (\\.[a-zA-Z]{2,4})       # dot-something\n",
    "    )''', re.VERBOSE)\n",
    "\n",
    "# TODO: Find matches in clipboard text.\n",
    "\n",
    "# TODO: Copy results to the clipboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The username part of the email address ➊ is one or more characters that can be any of the following: lowercase and uppercase letters, numbers, a dot, an underscore, a percent sign, a plus sign, or a hyphen. You can put all of these into a character class: [a-zA-Z0-9._%+-].\n",
    "\n",
    "The domain and username are separated by an @ symbol ➋. The domain name ➌ has a slightly less permissive character class with only letters, numbers, periods, and hyphens: [a-zA-Z0-9.-]. And last will be the “dot-com” part (technically known as the top-level domain), which can really be dot-anything. This is between two and four characters.\n",
    "\n",
    "The format for email addresses has a lot of weird rules. This regular expression won’t match every possible valid email address, but it’ll match almost any typical email address you’ll encounter."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 3: Find All Matches in the Clipboard Text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have specified the regular expressions for phone numbers and email addresses, you can let Python’s re module do the hard work of finding all the matches on the clipboard. The pyperclip.paste() function will get a string value of the text on the clipboard, and the findall() regex method will return a list of tuples.\n",
    "\n",
    "Make your program look like the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-90-1bff153768fe>, line 10)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  File \u001b[1;32m\"<ipython-input-90-1bff153768fe>\"\u001b[1;36m, line \u001b[1;32m10\u001b[0m\n\u001b[1;33m    text = str(pyperclip.paste())\u001b[0m\n\u001b[1;37m       ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "#! python3\n",
    "# phoneAndEmail.py - Finds phone numbers and email addresses on the clipboard.\n",
    "\n",
    "import pyperclip, re\n",
    "\n",
    "phoneRegex = re.compile(r'''(\n",
    "--snip--\n",
    "'''\n",
    "# Find matches in clipboard text.\n",
    "text = str(pyperclip.paste())\n",
    "\n",
    "matches = []\n",
    "for groups in phoneRegex.findall(text):\n",
    "   phoneNum = '-'.join([groups[1], groups[3], groups[5]])\n",
    "   if groups[8] != '':\n",
    "       phoneNum += ' x' + groups[8]\n",
    "   matches.append(phoneNum)\n",
    "for groups in emailRegex.findall(text):\n",
    "   matches.append(groups[0])\n",
    "\n",
    "# TODO: Copy results to the clipboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is one tuple for each match, and each tuple contains strings for each group in the regular expression. Remember that group 0 matches the entire regular expression, so the group at index 0 of the tuple is the one you are interested in.\n",
    "\n",
    "As you can see at ➊, you’ll store the matches in a list variable named matches. It starts off as an empty list, and a couple for loops. For the email addresses, you append group 0 of each match ➌. For the matched phone numbers, you don’t want to just append group 0. While the program detects phone numbers in several formats, you want the phone number appended to be in a single, standard format. The phoneNum variable contains a string built from groups 1, 3, 5, and 8 of the matched text ➋. (These groups are the area code, first three digits, last four digits, and extension.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Step 4: Join the Matches into a String for the Clipboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have the email addresses and phone numbers as a list of strings in matches, you want to put them on the clipboard. The pyperclip.copy() function takes only a single string value, not a list of strings, so you call the join() method on matches.\n",
    "\n",
    "To make it easier to see that the program is working, let’s print any matches you find to the terminal. If no phone numbers or email addresses were found, the program should tell the user this.\n",
    "\n",
    "Make your program look like the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "ename": "SyntaxError",
     "evalue": "invalid syntax (<ipython-input-91-3c8aa0bd2977>, line 4)",
     "output_type": "error",
     "traceback": [
      "\u001b[1;36m  File \u001b[1;32m\"<ipython-input-91-3c8aa0bd2977>\"\u001b[1;36m, line \u001b[1;32m4\u001b[0m\n\u001b[1;33m    --snip--\u001b[0m\n\u001b[1;37m            ^\u001b[0m\n\u001b[1;31mSyntaxError\u001b[0m\u001b[1;31m:\u001b[0m invalid syntax\n"
     ]
    }
   ],
   "source": [
    "#! python3\n",
    "# phoneAndEmail.py - Finds phone numbers and email addresses on the clipboard.\n",
    "\n",
    "--snip--\n",
    "for groups in emailRegex.findall(text):\n",
    "    matches.append(groups[0])\n",
    "\n",
    "# Copy results to the clipboard.\n",
    "if len(matches) > 0:\n",
    "    pyperclip.copy('\\n'.join(matches))\n",
    "    print('Copied to clipboard:')\n",
    "    print('\\n'.join(matches))\n",
    "else:\n",
    "    print('No phone numbers or email addresses found.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Running the Program"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For an example, open your web browser to the No Starch Press contact page at https://nostarch.com/contactus/, press CTRL-A to select all the text on the page, and press CTRL-C to copy it to the clipboard. When you run this program, the output will look something like this:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copied to clipboard:\n",
    "800-420-7240\n",
    "415-863-9900\n",
    "415-863-9950\n",
    "info@nostarch.com\n",
    "media@nostarch.com\n",
    "academic@nostarch.com\n",
    "info@nostarch.com"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ideas for Similar Programs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Identifying patterns of text (and possibly substituting them with the sub() method) has many different potential applications. For example, you could:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Find website URLs that begin with http:// or https://.\n",
    "2. Clean up dates in different date formats (such as 3/14/2019, 03-14-2019, and 2015/3/19) by replacing them with dates in a single, standard format.\n",
    "3. Remove sensitive information such as Social Security or credit card numbers.\n",
    "4. Find common typos such as multiple spaces between words, accidentally accidentally repeated words, or multiple exclamation marks at the end of sentences. Those are annoying!!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Summary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While a computer can search for text quickly, it must be told precisely what to look for. Regular expressions allow you to specify the pattern of characters you are looking for, rather than the exact text itself. In fact, some word processing and spreadsheet applications provide find-and-replace features that allow you to search using regular expressions.\n",
    "\n",
    "The re module that comes with Python lets you compile Regex objects. These objects have several methods: search() to find a single match, findall() to find all matching instances, and sub() to do a find-and-replace substitution of text.\n",
    "\n",
    "You can find out more in the official Python documentation at https://docs.python.org/3/library/re.html. Another useful resource is the tutorial website https://www.regular-expressions.info/."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}