Blog

Imagine you’re waiting in line to get into a movie 🎥 theater.

When it’s finally your 🧑 turn to enter, the attendant 👮 checks your ticket and allows you to proceed.

Just like this, iterators allow you to traverse through a collection of elements, one at a time⌚. Each element is like a ticket🎫 that needs to be checked before you can move on to the next one.

This way, you can access the elements in a sequence, without having to load the entire collection into memory 💾 at once.

Before we further proceed let’s look at the solution to the previous post’s challenge.

Previous post’s challenge’s solution

PyPy and CPython are both ✌️ implementations of python.

The major difference ⚡️ between them is the way they collect garbage 🗑️.

CPython as we saw in the previous post uses reference counting for garbage 🗑️ collection and makes use of linked lists.

PyPy on the other hand uses “garbage collection by tracing🛰️”. This means it makes use of the concept of reachable and unreachable objects. And only deals with reachable ones.

For example:

a = [1,2,3]

Here, a is a reachable object. But, if we do:

a = "Hello"

Then, [1,2,3] would be unreachable.

This approach makes PyPy a better, faster🏎️, and more efficient implementation in many cases.

However, the choice between both depends highly on their use case.

What are Iterators?

Just like anything else they are objects in python. These objects have 2 ✌️ special methods:

• __iter__()
• __next__()

The __iter__() method returns self.

And the __next__() method returns the next ⏭️ element of the sequence.

With reading this, you would think 🤔 lists, tuples, dictionaries, etc📑📚 are all iterators. But it’s not❌ true. They can return an iterator but they are not iterators.

Here’s how you make an iterator object out of a list📑:

#Creating a list
my_list = ["The Godfather", "The Shawshank Redemption", "The Dark Knight", "Pulp Fiction", "Forrest Gump"]
#iterator object from my_list
my_i = iter(my_list)
print(next(my_i))
print(next(my_i))
print(next(my_i))
print(next(my_i))
print(next(my_i))

Output:

The Godfather
The Shawshank Redemption
The Dark Knight
Pulp Fiction
Forrest Gump

Note: here we have used the iter() function to get an iterator object from the list📑. the iter() function and the __iter__() method are two ✌️ different things.

iter() is a built-in function that returns an iterator. __iter__() method is a method inside an iterator and an iterable object.

Similarly, the __next__() method and the next() function are two ✌️ different things.

Thus, just like at the ticket 🎫 counter of the 🎥 theater, the tickets are checked one by one. The iterator object allows to access a sequence of elements one by one.

Let’s get this more clearly by watching 👀 Iterables.

Iterables

Now, the lists📑 and tuples that clicked in your mind🧠 above are iterables.

They are the objects that can be iterated over. These objects have an __iter__() method that returns an iterator.

There are several other built-in iterables in Python besides lists📑 and tuples, including strings🧵, dictionaries📚, sets, and more.

By including the __iter__() function, user-defined classes can also be made iterable.

While both iterables and iterators may seem similar, there is an important distinction⚡️. Let’s see what.

The difference between Iterators and Iterable

Iterables are objects that can be iterated over, whereas iterators are objects that do the actual iteration.

Iterators can save the current state till the next() ⏭️ method is called. Whereas, iterables do not🚫 have the ability to save the current state.

An iterable generates an iterator when passed to the built-in iter() function.

In Python🐍, it is important to note that while every iterator is iterable, the opposite is not true: not every iterable is an iterator.

For example, a list📑 is iterable but a list is not an iterator.

Hope you are clear about the difference it is very important.💡 Now let’s move on towards traversing through iterables.

This means checking the tickets 🎫 of viewers in the line…

Iterating

Traversing🔎 through iterables using iterators is iterating. Iterating is nothing but going through the iterable’s elements one by one.

In our analogy, it means that we are now going through each and every ticket one after another.

So, the first and foremost way is here:

The __next__() Method And The next Function

To use the next method we would need an iterator object.

To get an iterator from an iterable(in our case a list )we use the iter function:

#True means the viewer has a ticket and False means he doesn't have one
viewer_queue = [True, True, False, True, True]
my_i = iter(viewer_queue)

Now, to get values from this iterator we need to use our next method:

#True means the viewer has a ticket and False means he doesn't have one
viewer_queue = [True, True, False, True, True]
my_i = iter(viewer_queue)
print(my_i.__next__())
print(my_i.__next__())
print(my_i.__next__())
print(my_i.__next__())
print(my_i.__next__())

Output:

True
True
False
True
True

This is a good 👍 way to get values one after another from an iterator but here’s a better way:

#True means the viewer has a ticket and False means he doesn't have one
viewer_queue = [True, True, False, True, True]
my_i = iter(viewer_queue)
print(next(my_i))
print(next(my_i))
print(next(my_i))
print(next(my_i))
print(next(my_i))

Output:

True
True
False
True
True

Both are the correct ✔️ ways but the second one is more convenient. The next() function automatically calls the __next__() method of the iterator object, so you don’t have to call it explicitly.

This is all good 👍 but there is an even more convenient way to do so.

I mean would you stop each viewer and then access his value by writing the code?✋👨‍💻

I don’t think it would be convenient.👎

for loop (how does it do it)

A good way would be to use a for loop like this:

#True means the viewer has a ticket and False means he doesn't have one
viewer_queue = [True, True, False, True, True]
for viewer in viewer_queue:
    print(viewer)

Output:

True
True
False
True
True

See here you didn’t need to make an iterator out of the list📑. This is because for loop does that for you behind the scenes.

Here are the steps👣 followed by the for loop behind the scenes:

• 👉First, it creates an iterator object.
• 👉Then, it runs a while loop till the next function raises a StopIteration✋ error.

The code would look something like this:

viewer_queue = [True, True, False, True, True]
# get the iterator object
iterator = iter(viewer_queue)
# loop until StopIteration is raised
while True:
    try:
        # get the next item from the iterator
        item = next(iterator)
        # process the item
        print(item)
    except StopIteration:
        # handle the end of the iterable
        break

Now, without any worries😟, you can sit back🪑 and the for loop will check the viewer’s tickets🎫 for you unlike before when you had to manually call the next ⏭️ function for every viewer.

I know you are thinking💭 what is this StopIteration?

But before we get into that let’s first create our own iterator object…

Custom Iterator

So, we all know by now that to create any object we need a blueprint called a class.

Which means we will make a class.

This class would have two ✌️ methods(behaviors):

• __iter__()
• __next__()

Let’s go:

So, I won’t directly write code. Let’s first make a blueprint that we can understand, this would help🤝 you while making projects in the future.

First, we want an iterator that would take a sequence of tuples something like this:

viewers = [
    ("Alice", True),
    ("Bob", False),
    ("Charlie", True),
    ("Dave", True),
]

This means our class will have a list as its attribute. Along with this, we would need a counter to know where we have reached. The __init__() would look like this:

class TicketChecker:
    def __init__(self, line):
        self.line = line
        self.current = 0

Now, as we already making an iterator object so we will return the object itself inside our iterator method:

class TicketChecker:
    def __init__(self, line):
        self.line = line
        self.current = 0
    
    def __iter__(self):
        return self

To get to our next⏭️ element we would:

• first need to check if there are any more elements👍 in our sequence or not,👎
• if there are elements(viewers) left, then we would take the current tuple and will increase the counter ➕by 1.
• aftermath we will check if the viewer has a ticket.
  • if so we will print “{viewer} has a ticket🎫”
  • else we will say “{viewer} does not have a ticket🎫”

To check if the viewer has a ticket we will check the second✌️ part of our tuple and to print the viewer’s name we would use the first part of it.

Here’s the full code of TicketChecker class and its usage example:

class TicketChecker:
    def __init__(self, line):
        self.line = line
        self.current = 0
    
    def __iter__(self):
        return self
    
    def __next__(self):
        if self.current >= len(self.line):
            raise StopIteration
        viewer = self.line[self.current]
        self.current += 1
        if viewer[1]:
            return f"{viewer[0]} has a ticket"
        else:
            return f"{viewer[0]} does not have a ticket"
viewers = [
    ("Alice", True),
    ("Bob", False),
    ("Charlie", True),
    ("Dave", True),
]
ticket_checker = TicketChecker(viewers)
for result in ticket_checker:
    print(result)

Output:

Alice has a ticket
Bob does not have a ticket
Charlie has a ticket
Dave has a ticket

There you go… You just made your own cool😎 iterator.

Now, time to discuss StopIteration ✋ before you make me bald👨‍🦲 out of anger.

StopIteration

You stop your vehicle when you see a red signal right?🚦

Similarly, the iterator gets to know that there are no more elements left with the help of the StopIteration🛑 exception.

Python’s official doc. explicitly states that the StopIteration exception should be raised when there are no further items to yield from the iterator.📚

Here’s what happened in for loop when it printed the last element of our ticket_checker object:

The for loop first created an iterator object using the __iter__() method which returned an instance of our TicketChecker class as we returned self there.

Next, the for loop calls the __next__()⏭️ method until it raises StopIteration exception. Which means there are no more elements left in the sequence.

We saw a lot about iterators and iterables but where are they even used? Let’s explore…🤔

Why and When to Use Iterators

A small video to explain iterators’ efficiency. (It is a visual demonstration of all the points below)

Memory efficiency

Now imagine💭 the movie is super popular and we have over a million users👥👥.

If we used a list(data structure) instead of our iterator class(data structure), it would require a lot of memory because the entire list is loaded in memory at once.

However, if you use an iterator instead, only one viewer at a time is loaded into memory, which is much more memory efficient.

The lazy nature of iterators: Faster and small-sized

The very nature of iterators is lazy, like you and me. The data is processed only as it is needed, so you can start getting results faster and with less memory overhead.

This means iterators not only save space on your memory but also, make your program, run smooth and fast as melted butter🧈 on bread🍞.

These are just two✌️ benefits explained in detail. But there are many other benefits like Data abstraction and Modularity. Modularity means that you can make your code more usable by dividing it into modules.

So, iterators sound like the world🌎, fast execution with a little memory usage, data abstraction, and modularity.

Ditch💔 the list now?

No, both have their own use cases, they are equally important and useful in their own ways.

An actual application of iterator

One good application 🗔 of an iterator is to take multiple lines of input in python.

Here’s how the built-in the iter() function and for loop can make it happen:

lines = []
for line in iter(input, ''):
    lines.append(line)
print(lines)

Here the iter function is taking 2 arguments, the input function (as iterable) and a sentinel value(a special value that is used to mark the end of a sequence of data) ''.

This code will consider input as an iterable and '' as a stop value.

Thus, when the user will give blank input the loop will end.

Infinite Iterators

Applications like Netflix, Prime Video,📺 etc have an endless amount📈 of data to deal with which is ever-increasing. Be it for content or for user data analysis.💻📊

For such use cases, there is something like infinite♾️ iterators. These are the iterators that can generate endless stream of values.

The built-in functions that can make such endless travelers are present in the itertools library:

1. itertools.count,
2. itertools.cycle,
3. itertools.repeat, and a lot more.

I am not explaining all these in detail as using infinite iterators isn’t common.

Also, if you ever use them you need to be extra cautious as they can crash or hang your program if they go into an endless loop.⚠️🔄💻

Through my🙅‍♂️🔁 experience till now, I can safely say that they are not much used.

itertools module and its use in Iterators

Itertools is a vast library📚 and here are the major and widely used functions of this library:

1. accumulate(iterable[, func]):
   • Use: Returns a running total of the values in an iterable.
   • Example: itertools.accumulate([1, 2, 3, 4, 5])
   • Output: 1 3 6 10 15
   • Explanation: The function takes an iterable as input and returns an iterator that yields the running total of the values. In the example code, [1, 2, 3, 4, 5] is the iterable and accumulate() returns an iterator that yields the running total of these values as 1 3 6 10 15.
   • it should be mentioned that the second argument, func, is not used in the given example, but can be used to apply a specific operation to the elements.
2. chain🔗(*iterables):
   • Use: Combines multiple iterables into a single iterable.
   • Example: itertools.chain([1, 2], [3, 4], [5, 6])
   • Output: 1 2 3 4 5 6
   • Explanation: The function takes multiple iterables as input and returns an iterator that yields the elements from each iterable in order. In the example code, [1, 2], [3, 4], and [5, 6] are the iterables that are combined into a single iterable using chain(), which yields the elements from all three iterables in order as 1 2 3 4 5 6.
3. compress(data, selectors):
   • Use: Filters the elements from an iterable based on a corresponding selector iterable.
   • Example: itertools.compress(['a', 'b', 'c', 'd'], [True, False, True, False])
   • Output: a c
   • Explanation: The function takes two iterables as input – data and selectors, where selectors contains boolean values corresponding to each element in data. compress() returns an iterator that yields the elements from data whose corresponding element in selectors is True. In the example code, ['a', 'b', 'c', 'd'] is the data iterable and [True, False, True, False] is the selectors iterable, so compress() returns an iterator that yields 'a' and 'c'.
   • it should be clarified that the output is the elements from data whose corresponding element in selectors is True.
4. count(start=0, step=1):
   • Use: Returns an iterator that generates an infinite sequence of numbers starting from start with a step of step.
   • Example: itertools.count(1, 2)
   • Output: 1 3 5 7 9 ...
   • Explanation: The function takes two arguments – start and step, which are optional and default to 0 and 1, respectively. count() returns an iterator that generates an infinite sequence of numbers starting from start with a step of step. In the example code, count() generates an infinite sequence of odd numbers starting from 1 with a step of 2.
   • it should be mentioned that start and step can also be negative, which would generate a sequence of decreasing numbers.
5. cycle🚲(iterable):
   • Use: Returns an iterator that repeats the elements of an iterable indefinitely.
   • Example: itertools.cycle([1, 2, 3])
   • Output: 1 2 3 1 2 3 1 2 3 ...
   • Explanation: The function takes an iterable as input and returns an iterator that repeats the elements of the iterable indefinitely. In the example code, [1, 2, 3] is the iterable that is repeated indefinitely.

Here I haven’t explained all the possible functionalities of the itertools module. You can read the official doc here to learn even more functionalities.

What are the Limits of Iterator?

One-way traversal🚫➡️

Iterators can only be iterated once, in a forward➡️ direction. Once an iterator has been exhausted, it cannot be reused♻️ or restarted.

You might say that we can iterate over lists using a for loop multiple times. For loops do create an iterator to iterate over lists. Then how is it true?

Well, every time you use a for loop a new iterator is created. That is why you are able to go through that list again and again.

No length

Iterators do not have a length or size, and so it is not possible to determine how many elements an iterator contains without iterating over it.

It’s like you need to dive into an ocean🌊 to know its depth which is risky.

No index access

Unlike lists, iterators do not support random access by index. To access a specific element in an iterator, you must iterate over it until you reach the desired✨ element.

This makes the time⌚ taken to access individual elements higher. In lists this time is constant.

In terms of Big O: the time taken to access an element in an iterator would be O(n), whereas the time taken to access an element from a list would be O(1).

No slicing🔪

Iterators do not support slicing operations, which means you cannot extract a subset of elements from an iterator using syntax like [start:stop:step].

No modification

Iterators are read-only, meaning you cannot modify🛠️ the elements of an iterator directly. To modify the elements of an iterable, you must convert it to a list or other mutable data structure first.

So ditch💔 iterators?

No, as I said before both lists and iterators are equally important you use them as and when needed.

Conclusion

🎟️🍿In summary, we’ve learned a lot about iterators and iterables! We used the analogy of checking tickets in a movie theater line🎫 to help us understand the difference between the two.

We also learned that iterators are objects that actually iterate through iterables.

💡We then went on to create our own TicketChecker iterator and saw how the StopIteration exception is used.

We also explored the benefits and limitations of using iterators, as well as real-life examples of their applications.

🔍To further expand our knowledge, we looked at the itertools library and its major functions. Finally, we acknowledged that while iterators have their advantages, they also have some limitations.

Overall, we hope this post has helped you better understand the power and potential of iterators in Python!🐍

Challenge 🧗‍♀️

Write a program that takes a list of numbers as input and returns a new list with only the even numbers, without using the built-in filter() function. Instead, you should use an iterator to iterate over the list and check each number for evenness. The program should return the new list containing only the even numbers.

Here’s an example of what the input and output should look like:

Input: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] Output: [2, 4, 6, 8, 10]

Note that your program should work with any list of integers, not just the example input provided above.

Stay happy😄 and keep coding and do suggest any improvements if there.👍

Take care and have a great😊 time I’ll see you soon in the next post…Bye Bye👋