One of the main differences between Python 3.0 and Python 2.X is the stronger emphasis on iterators. In addition to the iterators associated with built-in types such as files and dictionaries, the dictionary methods keys, values and items return iterable objects in Python 3.0, as do some of the built-in functions such as zip and range.

The Zip Function

As an example, consider the zip function. The Python zip function returns an iterator of tuples, where the i-th tuple contains the i-th element from each of the argument sequences or iterables. For example:

>>> x = [1, 2, 3]
>>> y = [4, 5, 6]
>>> zipped = zip(x, y)
>>> list(zipped)
[(1, 4), (2, 5), (3, 6)]

Note that if there is a size mismatch, the iterator returned will only return as many tuples as the smallest sequence:

>>> z = [7, 8]
>>> zipped = zip(x, z)
>>> list(zipped)
[(1, 7), (2, 8)]

These examples involve combining two lists, but any iterable sequence can be used in a zip object. For example, imagine we want to generate a series of tuples containing one file line in each tuple as well as the file line number:

>>> x = range(1,5)
>>> f = open('simple.py')
>>> zipped = zip(x,f)
>>> list(zipped)
[(1, 'a = [1, 2, 3, 4]\n'), (2, 'for x in a:\n'), (3, '\tprint(x)\n'), (4, '\tprint(x*2)\n')]

We could have used a list for x, but in this case, it’s easier to use a range. All of our examples so far have involved two iterable sequences in the zip object, but there’s no reason we couldn’t have more:

>>> x = [1, 2, 3]
>>> y = [4, 5, 6]
>>> z = [7, 8, 9]
>>> zipped = zip(x,y,z)
>>> list(zipped)
[(1, 4, 7), (2, 5, 8), (3, 6, 9)]

Note that in earlier versions of Python (2.X), typing in the name of the zip object at the interactive prompt will return the contents of the zip object. This is because in Python 2.X, the zip object is a list. In Python 3.0, you need to convert the zip object into a list first. This means a little extra coding, but it is also an asset in larger programs. Delayed evaluation conserves memory and avoids pauses while large result lists are computed.

The Range Function

We demonstrated here that the range is also an iterator; in 3.0, it returns an iterator that generates numbers in the range on demand, instead of building the result list in memory. This subsumes the older 2.X xrange, and you must use list(range()) to force an actual range if one is needed. For example:

>>> myrange = range(10)
>>> myrange
range(0, 10)

>>> I = iter(myrange)
>>> next(I)
0
>>> next(I)
1
>>> next(I)
2

>>> list(range(10))
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

Unlike the list returned by such a call in 2.X, range objects in 3.0 support only iteration, indexing, and the len function. They do not support any other sequence operations. Thus, you should use list if you need more list tools.

>>> myrange = range(10)
>>> len(myrange)
10
>>> myrange[0]
0
>>> myrange[-1]
9

>>> next(I)
3
>>> I.__next__()
4

It should also be noted that Python 2.X has a built-in object called xrange, which is like range but produces items on demand instead of building a list of results in memory all at once. Since the iterator-based range does this in Python 3.0, xrange is no longer available and has been subsumed. You may still see it, however, in older Python code.

External Links:

Zip objects at doc.python.org
Python Iterators at Python Wiki
Python Iterator tutorial at bogotobogo.com

Now that we have completed our look at strings, we will begin to look at another useful object: Python iterators.

The for loop is often used when we have to iterate over a series of numbers; for example:

>>> for x in range(1,5):
	print(x)

1
2
3
4

The for loop can also work on any sequence type in Python, including lists, tuples and strings. For example:

>>> for x in [1, 2, 3, 4]:
	print(x)

1
2
3
4

>>> for x in 'quick':
	print(x * 2)

qq
uu
ii
cc
kk

Actually, the for loop turns out to be even more generic than this: it works on any iterable object. In fact, this is true of all iteration tools that scan objects from left to right in Python, including for loops, list comprehensions, membership tests, the map built-in function, and more.

Introduction to Python Iterators

The concept of iterable objects is relatively recent in Python, but it has come to permeate the language’s design.

We can get a better understanding of Python iterators and how they have become pervasive in the language’s design by looking at a built-in type such as the file. Open file objects have a method called readline, which reads one line of text from a file at a time. Each time we call the readline method, we advance to the next line. At the send of the file, an empty string is returned, which we can detect to break out of the loop:

>>> f = open('simple.py')
>>> f.readline()
'a = [1, 2, 3, 4]\n'
>>> f.readline()
'for x in a:\n'
>>> f.readline()
'	print(x)\n'
>>> f.readline()
'	print(x*2)\n'
>>> f.readline()
''

However, files also have a method name __next__ that has a nearly identical effect: it returns the next line from a file each time it is called. The only noticeable difference is that __next__ raises a built-in StopIteration exception at the end of file instead of returning an empty string:

>>> f = open('simple.py')
>>> f.__next__()
'a = [1, 2, 3, 4,]\n'
>>> f.__next__()
'for x in a:\n'
>>> f.__next__()
'\tprint(x)\n'
>>> f.__next__()
'\tprint(x*2)\n'
>>> f.__next__()
Traceback (most recent call last):
  File "<pyshell#31>", line 1, in 
    f.__next__()
StopIteration

This interface is exactly what we call the iteration protocol in Python. Any object with a __next__ method to advance to a next result, which raises StopIteration at the end of the series of results, is considered a Python iterator. Any such object may also be stepped through with a for loop or other iteration tool, because all Python iterators tools normally work internally by calling __next__ on each iteration and catching the StopIteration exception to determine when to exit.

Thus, the best way to reach a text file line by line is to allow the for loop to automatically call __next__ to advance to the next line on each iteration. The file object’s Python iterator will do the work of automatically loading lines as you go. For example, if we want to read in simple.py line by line, we can code it as follows:

>>> for line in open('simple.py'):
	print(line, end='')

a = [1, 2, 3, 4]
for x in a:
	print(x)
	print(x*2)

Almost as easily, we can iterate through the file and print it out in uppercase:

>>> for line in open('simple.py'):
	print(line.upper(), end='')

A = [1, 2, 3, 4]
FOR X IN A:
	PRINT(X)
	PRINT(X*2)

Notice that the print uses end=” here to suppress adding a \n, because line strings already have one. This is considered the best way to read text files line by line today for several reasons: it is the simplest to code, it might be the quickest to run, and is the best in terms of memory and usage. The older, original way to achieve the same effect with a for loop was to call the file readlines method to load the file’s content into memory as a list of line strings:

>>> for line in open('simple.py').readlines():
	print(line.upper(), end='')

This readlines technique still works, but it is not considered the best practice today and performs poorly in terms of memory usage. In fact, because this version really does load the entire file into memory all at once, it will not even work for files too big to fit into the memoery space available on your computer. Rather, because it reads one line at a time, the Python iterator-based version is not prone to such memory issues.

The iterator version might run quicker as well, though it can vary by release. It is also possible to read a file line by line with a while loop:

>>> f = open('simple.py')
>>> while True:
	line = f.readline()
	if not line: break
	print(line.upper(), end='')

	
A = [1, 2, 3, 4]
FOR X IN A:
	PRINT(X)
	PRINT(X*2)

However, this may run slower than the iterator-based for loop version, because Python iterators run at C language speed inside Python, whereas the while loop version run Python byte code through the Python virtual machine. Any time you swap Python code, for C code, speed tends to increase, though not in all cases.

External Links:

Python Iterators at Python Wiki

Python Iterator tutorial at bogotobogo.com