iterators-generators-and-uncertainty.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "# Iterators, Generators, and Uncertainty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Suppose you are working on a Python API that provides access to a real-time data stream (perhaps from an array of sensors or from a web service that handles user requests). You would like to deliver to the consumers of your API a simple but flexible abstraction that allows them to operate on new items from the stream when they choose to do so. Furthermore, you would like the API to allow users to do the following three things:\n",
    "\n",
    "* specify fall-back or default data streams (*e.g.*, if their first choice of stream is exhausted);\n",
    "* interleave items coming from multiple streams (presenting them as a single, new stream); and\n",
    "* process the items from a stream in parallel using multiprocessing.\n",
    "\n",
    "What abstraction should you use? How much of it must be custom-built and how much can be done using native Python features? When working with data streams, state spaces, and other abstractions that represent large or unbounded structures, it can be tempting to custom-build solutions that may become increasingly complex and difficult to maintain. Understanding a range of features that are already available in a language or its built-in libraries can help mitigate this while saving significant time and effort (both your own and that of others who build upon your work)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Iterators and generators are powerful tools in the Python language that have compelling applications in a number of contexts. This article reviews how they are defined, how they can be used, how they are related to one another, and how they can help you work in an elegant and flexible way with data structures and data streams of an unknown or infinite size."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Iterables, Iterators, and Generators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "When discussing Python, the terms *iterable*, *iterator*, and *generator* often appear in similar contexts or are even used interchangeably. These language features also solve similar problems. This can lead to some confusion, but there is a reason that on occasion these are conflated."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "One way to understand the term *iterator* is that it refers to *any* Python data structure that *has an interface* that supports iteration over objects one at a time. A data structure is *iterable* if there is *at least one way* to construct an iterator that traverses it in some way. On the other hand, a *generator* is a particular kind of data structure, defined in a specific way within Python, that maintains an internal state and constructs or retrieves zero or more objects or values one at a time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Thus, a generator by virtue of its characteristics can have an interface that allows it to qualify as an iterator, which consequently also makes it an iterable. In fact, all generators are iterators and iterable. However, not all iterators or iterable data structures are generators because there exist other approaches for building a Python object that possesses the kind of interface an iterator or iterable is expected to have."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Iterators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If you want to implement an iterator data structure directly, you need to include a special method `__next__` in the class definition, which will be invoked whenever the built-in [`next`](https://docs.python.org/3/library/functions.html#next) function is applied to an instance of that data structure. The `skips` data structure below can emit every other positive integer via its definition of a `__next__` method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class skips:\n",
    "    def __init__(self):\n",
    "        self.integer = 0\n",
    "\n",
    "    def __next__(self):\n",
    "        self.integer += 2\n",
    "        return self.integer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now it is possible to use the built-in [`next`](https://docs.python.org/3/library/functions.html#next) function to retrieve each item one at a time from an instance of the `skips` data structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[2, 4, 6]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ns = skips()\n",
    "[next(ns), next(ns), next(ns)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The number of items over which the data structure will iterate can be limited by raising the `StopIteration` exception when more items can not (or should not) be returned."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class skips:\n",
    "    def __init__(self, start, end):\n",
    "        self.integer = start-2\n",
    "        self.end = end\n",
    "\n",
    "    def __next__(self):\n",
    "        self.integer += 2\n",
    "        if self.integer > self.end:\n",
    "            raise StopIteration\n",
    "        return self.integer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It is then the responsibility of any code that uses an instance of this iterator to catch this exception and handle it appropriately. It is worth acknowledging that this is a somewhat unusual use of a language feature normally associated with catching errors (because an iterator being exhausted is not always an error condition)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0\n",
      "2\n",
      "4\n",
      "6\n",
      "8\n",
      "10\n"
     ]
    }
   ],
   "source": [
    "ns = skips(0, 10)\n",
    "while True:\n",
    "    try:\n",
    "        print(next(ns))\n",
    "    except StopIteration:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Iterables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In Python, there is a distinction between an *iterator* and an *iterable data structure*. This distinction is useful to maintain for a variety of reasons, including the ones below.\n",
    "\n",
    "* You may not want to clutter a data structure (as it may represent a spreadsheet, a database table, a large graph, and so on) with the state necessary to keep track of an iteration process.\n",
    "* You may want the data structure to support *multiple* iterators, either semantically (*e.g.*, iteration over rows versus over columns) or in terms of implementation (*e.g.*, breadth-first search versus depth-first search).\n",
    "* You may want to make it easy to *reset* iteration without fiddling with the internal state of a data structure instance (*i.e.*, resetting a traversal of the data structure instance could involve simply creating a fresh iterator)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As an example, consider a data structure `interval` that represents all positive integers in some range. Users might be allowed to obtain two different kinds of iterators for an instance of this data structure: those that iterate over only the even integers and those that iterate over only the odd integers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class interval:\n",
    "    def __init__(self, lower, upper):\n",
    "        self.lower = lower\n",
    "        self.upper = upper\n",
    "\n",
    "    def evens(self):\n",
    "        return skips(\n",
    "            self.lower + (0 if (self.lower % 2) == 0 else 1),\n",
    "            self.upper\n",
    "        )\n",
    "    \n",
    "    def odds(self):\n",
    "        return skips(\n",
    "            self.lower + (0 if (self.lower % 2) == 1 else 1),\n",
    "            self.upper\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The example below illustrates how an iterator returned by one of the methods in the definition of `interval` can be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "3\n",
      "5\n",
      "7\n",
      "9\n"
     ]
    }
   ],
   "source": [
    "ns = interval(0, 10).odds()\n",
    "while True: # Keep iterating and printing until exhaustion.\n",
    "    try:\n",
    "        print(next(ns))\n",
    "    except StopIteration:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "So far in this article, the distinction between *iterators* and *iterable data structures* has been explicit for clarity. However, the convention that is supported (and sometimes expected) throughout Python is that an iterable data structure has a *single* iterator that can be used to iterate over it. This iterator is returned by a special method [`__iter__`](https://docs.python.org/3/reference/datamodel.html#object.__iter__) that is included in the class definition. In the example below, the `interval` class supports the creation of an iterator that visits every integer in the interval."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class every:\n",
    "    def __init__(self, start, end):\n",
    "        self.integer = start - 1\n",
    "        self.end = end\n",
    "\n",
    "    def __next__(self):\n",
    "        self.integer += 1\n",
    "        if self.integer > self.end:\n",
    "            raise StopIteration\n",
    "        return self.integer\n",
    "\n",
    "class interval:\n",
    "    def __init__(self, lower, upper):\n",
    "        self.lower = lower\n",
    "        self.upper = upper\n",
    "\n",
    "    def __iter__(self):\n",
    "        return every(self.lower, self.upper)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Python's built-in [`iter`](https://docs.python.org/3/library/functions.html#iter) function can be used to invoke `__iter__` for an instance of this data structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1\n",
      "2\n",
      "3\n"
     ]
    }
   ],
   "source": [
    "ns = iter(interval(1, 3))\n",
    "while True: # Keep iterating and printing until exhaustion.\n",
    "    try:\n",
    "        print(next(ns))\n",
    "    except StopIteration:\n",
    "        break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Including a definition for an `__iter__` method also makes it possible to use many of Python's built-in functions and language constructs that expect an iterable data structure. This includes functions such as [`list`](https://docs.python.org/3/library/functions.html#func-list) and [`set`](https://docs.python.org/3/library/functions.html#func-set), which use `iter` to obtain an iterator for their inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10})"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(interval(0, 10)), set(interval(0, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This also includes comprehensions and `for` loops."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\n",
      "[1, 2]\n",
      "[1, 2, 3]\n",
      "[1, 2, 3, 4]\n"
     ]
    }
   ],
   "source": [
    "for n in interval(1, 4):\n",
    "    print([k for k in interval(1, n)])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "There is nothing stopping you from making the iterator itself an iterable by having it return itself, as in the variant below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class every:\n",
    "    def __init__(self, start, end):\n",
    "        self.integer = start - 1\n",
    "        self.end = end\n",
    "\n",
    "    def __next__(self):\n",
    "        self.integer += 1\n",
    "        if self.integer > self.end:\n",
    "            raise StopIteration\n",
    "        return self.integer\n",
    "\n",
    "    def __iter__(self):\n",
    "        return self"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This approach ensures that there is no ambiguity (from a programmer's perspective) about what will happen when built-in functions such as `list` are applied to an instance of the data structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(every(0, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This practice is common and is the cause of some of the confusion and conflation that occurs between iterators and iterables. In addition to the potential for confusion, users of such a data structure must be careful to use the iterator as an iterable only once (or, alternatively, the object must reset its internal state every time `__iter__` is invoked)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ns = every(0, 10)\n",
    "list(ns), list(ns) # Only returns contents the first time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Nevertheless, this can also be a useful practice. Going back to the example with `evens` and `odds`, ensuring the iterators returned by these methods are also iterable means they can be fed directly into contexts that expect an iterable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class skips:\n",
    "    def __init__(self, start, end):\n",
    "        self.integer = start - 2\n",
    "        self.end = end\n",
    "\n",
    "    def __next__(self):\n",
    "        self.integer += 2\n",
    "        if self.integer > self.end:\n",
    "            raise StopIteration\n",
    "        return self.integer\n",
    "\n",
    "    def __iter__(self):\n",
    "        return self\n",
    "\n",
    "class interval:\n",
    "    def __init__(self, lower, upper):\n",
    "        self.lower = lower\n",
    "        self.upper = upper\n",
    "\n",
    "    def evens(self):\n",
    "        return skips(\n",
    "            self.lower + (0 if (self.lower % 2) == 0 else 1),\n",
    "            self.upper\n",
    "        )\n",
    "    \n",
    "    def odds(self):\n",
    "        return skips(\n",
    "            self.lower + (0 if (self.lower % 2) == 1 else 1),\n",
    "            self.upper\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The example below illustrates how this kind of interface can be used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0, 2, 4, 6, 8, 10], {1, 3, 5, 7, 9})"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "i = interval(0, 10)\n",
    "list(i.evens()), set(i.odds())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Generators"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Generators are data structures defined using either the `yield` statement or comprehension notation (also known as a [generator expression](https://docs.python.org/3/glossary.html#term-generator-expression)). The example below defines a generator `skips` using both approaches. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def skips(start, end):\n",
    "    integer = start\n",
    "    while integer <= end:\n",
    "        yield integer\n",
    "        integer += 2\n",
    "\n",
    "def skips(start, end):\n",
    "    return (\n",
    "        integer\n",
    "        for integer in range(start, end)\n",
    "        if (integer - start) % 2 == 0\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "When it is evaluated, a generator returns an iterator (more precisely called a [generator iterator](https://docs.python.org/3/glossary.html#term-generator-iterator)). These are technically both iterators and iterables. For example, as with any iterator, `next` can be applied directly to instances of this data structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 2, 4)"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ns = skips(0, 10)\n",
    "next(ns), next(ns), next(ns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As with any iterator, exhaustion can be detected by catching the `StopIteration` exception."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Exhausted generator iterator.\n"
     ]
    }
   ],
   "source": [
    "ns = skips(0, 2)\n",
    "try:\n",
    "    next(ns), next(ns), next(ns)\n",
    "except StopIteration:\n",
    "    print(\"Exhausted generator iterator.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Finally, an instance of the data structure can be used in any context that expects an iterable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 2, 4, 6, 8]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(skips(0, 10))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It is possible to confirm that the result of evaluating `skips` is indeed a generator by checking its type.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import types\n",
    "isinstance(skips(0, 10), types.GeneratorType)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "It is also possible to inspect its type to confirm that `skips` indeed evaluates to an iterator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import collections\n",
    "isinstance(skips(0, 10), collections.abc.Iterator)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Data Structures of Infinite or Unknown Size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Among the use cases that demonstrate how iterators/generators serve as a powerful language feature are scenarios involving data structures whose size is unknown or unbounded/infinite (such as streams, very large files, databases, and so on). You have already seen that you can define an iterable that can produce new objects or values indefinitely, so iterables are an effective way to represent and encapsulate such structures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Returning to the example described at the beginning of the article, recall that you are faced with creating a Python API for working with data streams that might (or might not) run out of items that can be drawn from them. The advantages of leveraging iterables and generators should be evident at this point, so suppose you move ahead with this option and implement an iterable to represent a data stream. How can you address the three specific requirements (*i.e.*, default/fall-back streams, interleaving, and splitting for parallelism) using these features?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To satisfy the first requirement, you must allow a user to exhaust one iterable and then switch to another one. This is straightforward to do by constructing a generator that concatenates two iterables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def concatenate(xs, ys):\n",
    "    for x in xs:\n",
    "        yield x\n",
    "    for y in ys:\n",
    "        yield y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Concatenating two instances of an iterable data structure is now straightforward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 2, 4, 6, 8, 10]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(concatenate(skips(0,5), skips(6,11)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Notice that if the first iterable is never exhausted, the second one will never be used."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "To address the second requirement, first consider a simpler scenario. What if you would like to \"line up\" or \"pair up\" entries in two or more iterables? You can use the built-in [`zip`](https://docs.python.org/3/library/functions.html#zip) function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[(0, 6), (2, 8), (4, 10)]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(zip(skips(0,5), skips(6,11)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Notice that the result of evaluating `zip` is indeed an iterator."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import collections\n",
    "isinstance(\n",
    "    zip(skips(0,5), skips(6,11)),\n",
    "    collections.abc.Iterator\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Combining `zip` with comprehension syntax, you can now define a generator that *interleaves* two iterables (switching back and forth between emitting an item from one and then the other)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "def interleave(xs, ys):\n",
    "    return (\n",
    "        z \n",
    "        for (x, y) in zip(xs, ys) \n",
    "        for z in (x, y)\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "As with concatenation, interleaving is now concise and straightforward."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0, 6, 2, 8, 4, 10]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(interleave(skips(0,5), skips(6,11)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Finally, how can you help users process items from a stream in parallel? Because you are already using iterables, users have some options available to them from the built-in [`itertools`](https://docs.python.org/3/library/itertools.html) library."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "One option is [`islice`](https://docs.python.org/3/library/itertools.html#itertools.islice), which behaves in a similar manner to Python [slice notation](https://docs.python.org/3/library/functions.html?highlight=slice#slice) (such as `xs[0:10]` to extract the first ten entries from a list `xs`). Users can use this function to extract items in batches and (1) pass each item in a batch to its own separate thread or (2) pass batches of items to separate threads. A basic batching method is presented below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from itertools import islice\n",
    "def batch(xs, size):\n",
    "    ys = list(islice(xs, 0, size))\n",
    "    while len(ys) > 0:\n",
    "        yield ys\n",
    "        ys = list(islice(xs, 0, size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Notice that this method inherits the graceful behavior of slice notation when the boundaries of the slices do not line up exactly with the number entries in the data structure instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[0, 2, 4], [6, 8, 10], [12, 14, 16], [18, 20]]"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "list(batch(skips(0,21), 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Can you define a generator that returns batches of batches (*e.g.*, at most `n` batches each of size at most `k`)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Another option is to use the [`tee`](https://docs.python.org/3/library/itertools.html#itertools.tee) function, which can duplicate a single iterable into multiple iterables. However, this function is really only simulating this effect by storing a large amount of auxiliary information from one of the iterables. Thus, it may use a significant amount of memory and is not safe to use with multiprocessing. It is best suited for situations in which the iterables are known to have a small number of items, as in the example below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0, 2, 4, 6, 8, 10], [0, 2, 4, 6, 8, 10])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from itertools import tee\n",
    "(a, b) = tee(skips(0,11), 2)\n",
    "(list(a), list(b))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "The example above is arguably implemented in a more clear and familiar way by simply wrapping the iterables using `list`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "([0, 2, 4, 6, 8, 10], [0, 2, 4, 6, 8, 10])"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ns = list(skips(0,11))\n",
    "(ns, ns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Further Reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "This article presents the relationships and distinctions between a number of related Python language constructs, and illustrates via the use case of data streams how these constructs can be leveraged. You can visit the Python Wiki if you are looking for additional discussion and examples of both [iterators](https://wiki.python.org/moin/Iterator) and [generators](https://wiki.python.org/moin/Generators). The Python documentation also contains a [Functional Programming HOWTO](https://docs.python.org/3/howto/functional.html) that discusses how iterators and generators offer new kinds of modularity and composability. Furthermore, the Python documentation entry for the built-in [itertools](https://docs.python.org/3/library/itertools.html) library contains many built-in functions and recommended patterns that are specialized to particular scenarios."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}