How to create an iterator in Python

Iterators in Python provide a way to access elements of a collection one by one. They are fundamental for efficient memory management with large datasets through protocols like __iter__() and __next__().

In this article, you'll explore techniques to build your own custom iterators. You'll discover practical tips, see real-world applications, and get essential debugging advice to master this powerful Python feature.

Basic iterator class implementation with __iter__() and __next__()

class Counter:
def __init__(self, limit):
self.limit = limit
self.count = 0

def __iter__(self):
return self

def __next__(self):
if self.count < self.limit:
self.count += 1
return self.count
raise StopIteration

# Usage
my_counter = Counter(3)
for num in my_counter:
print(num)--OUTPUT--1
2
3

The Counter class implements Python's iterator protocol using two essential dunder methods. This allows it to manage its own state and produce a sequence of values on demand.

• The __iter__() method is required for an object to be iterable. Here, it returns self, which is a common pattern that indicates the object is its own iterator.
• The __next__() method provides the logic for producing the next value. When the counter reaches its limit, it raises a StopIteration exception. This isn't an error—it's the standard signal that tells the for loop the iteration is complete.

Intermediate iterator approaches

Beyond the formal class structure, you can create iterators more concisely with generator functions that use yield or by leveraging Python's built-in iteration tools.

Using generator functions with yield

def simple_generator(limit):
count = 0
while count < limit:
count += 1
yield count

# Generator functions create iterators automatically
for num in simple_generator(3):
print(num)--OUTPUT--1
2
3

Generator functions provide a more concise way to create iterators. When you use the yield keyword in a function, Python automatically creates an iterator that handles the underlying protocol for you, saving you from writing a full class.

• The yield statement pauses the function, saves its state, and returns a value to the loop.
• On the next iteration, the function resumes exactly where it left off until it completes or another yield is hit.

Working with iter() and next() functions

# Create an iterator from a list
numbers = [1, 2, 3]
iterator = iter(numbers)

# Manual iteration
print(next(iterator))
print(next(iterator))
print(next(iterator))
# next(iterator) # Would raise StopIteration--OUTPUT--1
2
3

Python's built-in iter() function can create an iterator from any iterable sequence, like a list. This is the same mechanism for loops use behind the scenes. Once you have an iterator, you can manually pull values from it using the next() function.

• Each call to next() fetches the subsequent item and advances the iterator's internal state.
• If you call next() after the last item has been retrieved, it raises a StopIteration exception, signaling that the iteration is complete.

Creating iterators from existing collections

# Dictionary keys iteration
user = {"name": "Alice", "age": 30, "city": "New York"}
keys_iterator = iter(user.keys())
values_iterator = iter(user.values())

print(list(keys_iterator))
print(list(values_iterator))--OUTPUT--['name', 'age', 'city']
['Alice', 30, 'New York']

You aren't limited to creating iterators from scratch. Python's built-in collections, like dictionaries, work seamlessly with the iterator protocol. Methods like .keys() and .values() provide iterable views of a dictionary's contents.

• By passing these views to the iter() function, you get dedicated iterators for keys and values.
• This allows you to process them separately and efficiently, without needing to create new data structures in memory.

Advanced iterator techniques

Beyond simple sequences, you can create iterators for more advanced scenarios, including infinite loops, complex state tracking, and specialized patterns with the itertools module.

Creating infinite iterators

from itertools import count

# Create an infinite counter
counter = count(1)
print(next(counter))
print(next(counter))
print(next(counter))

# Use with caution - this would run forever:
# for num in counter:
# print(num)--OUTPUT--1
2
3

The itertools module is your go-to for specialized iterator patterns, including infinite ones. The count() function, for instance, creates an iterator that generates an endless sequence of numbers, starting from a specified value.

• Unlike other iterators, it never raises a StopIteration exception, so it doesn’t have a natural endpoint.

Because it runs indefinitely, you'll need to provide an explicit stopping condition when using it in a loop to avoid an infinite cycle.

Custom iterator with state management

class DatabaseIterator:
def __init__(self, data):
self.data = data
self.index = 0
print("Iterator initialized")

def __iter__(self):
return self

def __next__(self):
if self.index < len(self.data):
result = f"Record {self.index}: {self.data[self.index]}"
self.index += 1
return result
print("Iterator exhausted")
raise StopIteration

records = ["Alice", "Bob", "Charlie"]
db_iter = DatabaseIterator(records)
for record in db_iter:
print(record)--OUTPUT--Iterator initialized
Record 0: Alice
Record 1: Bob
Record 2: Charlie
Iterator exhausted

This example demonstrates how a custom iterator can manage its own internal state. The __init__ method prepares the iterator by storing the dataset and setting an index to 0, which acts as a bookmark for its current position.

• Each time __next__() is called, it uses the index to retrieve the next record and then increments it, effectively moving the bookmark forward.
• This pattern is ideal for iterators that need to track complex state, such as their position within a dataset or progress through a multi-step process.

Using itertools for complex iteration patterns

from itertools import cycle, islice, chain

# Combine multiple iterators
numbers = [1, 2, 3]
letters = ['a', 'b', 'c']
combined = chain(numbers, letters)
print(list(combined))

# Create a cycling iterator and limit it
colors = ['red', 'green', 'blue']
color_cycle = cycle(colors)
print(list(islice(color_cycle, 5)))--OUTPUT--[1, 2, 3, 'a', 'b', 'c']
['red', 'green', 'blue', 'red', 'green']

The itertools module offers powerful tools for creating complex iterator patterns. You can combine multiple sequences into one or create iterators that repeat indefinitely.

• The chain() function links iterables together, processing them sequentially. In the example, it first goes through all the numbers and then all the letters.
• cycle() repeats a sequence endlessly. To prevent an infinite loop, you can pair it with islice(), which takes a specific number of items from an iterator. Here, it grabs the first five colors from the repeating cycle.

Move faster with Replit

Replit is an AI-powered development platform that comes with all Python dependencies pre-installed, so you can skip setup and start coding instantly. Instead of piecing together individual techniques, you can use Agent 4 to build complete applications from a simple description.

• A log processor that uses chain() to merge multiple data streams into a single, sequential output for analysis.
• A task scheduler that uses cycle() to assign tasks to team members on a rotating basis, with islice() to limit assignments for a specific period.
• A data-fetching utility that implements a custom iterator to stream large datasets from a file, processing records one by one to conserve memory.

Simply describe your app, and Replit will write the code, test it, and fix issues automatically, all within your browser.

Common errors and challenges

Building custom iterators is powerful, but you'll likely run into a few common pitfalls along the way.

• Accidentally modifying a collection during for loop iteration. It's a classic mistake to change a list or dictionary while you're looping over it. Doing so can lead to unpredictable results, like skipping items, because the iterator loses track of its position within the changing collection.
• Handling one-time-use generator issues. Generators are exhaustible, meaning you can only iterate over them once. After a for loop consumes all the values, the generator is empty. Trying to loop over it again won't produce any values or errors—it will simply do nothing, which can be confusing to debug.
• Properly handling StopIteration in custom iterators. In a custom iterator class, you must raise StopIteration to signal that there are no more items. If you forget this step, your iterator will never end, and any for loop using it will run infinitely.

Accidentally modifying a collection during for loop iteration

Modifying a collection, such as a list, while iterating over it is a recipe for confusion. When you remove an item, you shift the positions of the remaining elements, causing the iterator to skip over what should be the next item.

For example, observe what happens in the following code when you try to remove all even numbers from a list during a for loop.

numbers = [1, 2, 3, 4, 5]
for num in numbers:
if num % 2 == 0:
numbers.remove(num) # Modifies the list while iterating!
print(numbers) # Result: [1, 3, 5]

When numbers.remove(2) runs, the list shrinks. The iterator then moves to the next position, which now contains 4, causing it to skip over 3 entirely. A safer approach is to iterate over a copy, as shown below.

numbers = [1, 2, 3, 4, 5]
to_remove = [num for num in numbers if num % 2 == 0]
for num in to_remove:
numbers.remove(num)
print(numbers) # Result: [1, 3, 5]

The solution works by separating the finding from the removing. By creating a new list of items to delete, like to_remove, you can iterate over it to safely modify the original list. This prevents the iterator from getting confused because the collection it's looping over isn't changing. It's a crucial pattern to remember anytime you need to filter a collection in place.

Handling one-time-use generator issues

Generators are one-time-use iterators. Once you've looped through all their values, they become exhausted. Trying to iterate over them again won't raise an error but will silently return nothing, as the following code demonstrates.

def numbers(n):
for i in range(1, n+1):
yield i

gen = numbers(3)
print(list(gen)) # [1, 2, 3]
print(list(gen)) # [] - generator is exhausted!

The first call to list(gen) pulls all the values, emptying the generator. When called again, it finds nothing left to process, which is why it returns an empty list. The following code demonstrates a common solution.

def numbers(n):
for i in range(1, n+1):
yield i

# Store the values if you need to use them multiple times
values = list(numbers(3))
print(values) # [1, 2, 3]
print(values) # [1, 2, 3] - still available!

If you need to use the generated values more than once, the solution is to store them in a collection. By calling list(numbers(3)), you consume the generator and save its output into a list. Now, you can access the values list repeatedly without issue. This pattern is crucial when you need to perform multiple passes over the same data, as the original generator can only be used a single time.

Properly handling StopIteration in custom iterators

In custom iterators, you must explicitly raise StopIteration to signal the end. Simply calling the exception like a function won't work and will cause an infinite loop. The following code demonstrates what happens when this crucial step is missed.

def custom_range(start, end):
current = start
while True:
if current >= end:
StopIteration() # Wrong! This doesn't stop iteration
yield current
current += 1

for num in custom_range(1, 4):
print(num) # Infinite loop: 1, 2, 3, 4, 5, ...

The function call StopIteration() creates an exception object but never actually signals the loop to stop. With no break condition, the while True: loop runs forever. The corrected version below shows how to properly halt the iteration.

def custom_range(start, end):
current = start
while current < end: # Condition handles termination
yield current
current += 1

for num in custom_range(1, 4):
print(num) # Prints: 1, 2, 3

The fix is to let the generator handle its own exit. By changing the loop to while current < end:, the function terminates naturally once the condition is false. You don't need to raise StopIteration yourself in a generator function; Python does it automatically when the function completes. This approach is cleaner and avoids the risk of creating an infinite loop by mistake.

Real-world applications

Now that you can navigate the common challenges, you can use iterators for powerful real-world tasks like file processing and data pipelines.

Processing text files with iter() and next()

You can efficiently process large files by creating an iterator with iter(), which lets you handle data line by line without loading the entire file into memory.

# Simulating reading a file line by line
sample_file_content = ["Header", "IMPORTANT: Critical info", "Regular text", "IMPORTANT: Note this"]
file_iterator = iter(sample_file_content)

for line_num, line in enumerate(file_iterator, 1):
if "IMPORTANT" in line:
print(f"Alert on line {line_num}: {line}")

This example demonstrates how to process a sequence item by item, simulating reading from a file. The iter() function converts the list into an iterator, which lets you pull data one piece at a time.

• The for loop uses enumerate() to track the line number, starting from 1, alongside the content of each line.
• Inside the loop, an if statement checks each line for the substring "IMPORTANT" and prints a formatted alert when it finds a match.

Building a data processing pipeline with yield

You can build efficient data processing pipelines by chaining generator functions with yield, allowing data to flow through multiple transformation steps without consuming large amounts of memory.

def get_data():
for i in range(1, 6):
yield i

def filter_even(numbers):
for num in numbers:
if num % 2 == 0:
yield num

def multiply_by_ten(numbers):
for num in numbers:
yield num * 10

# Create and execute the data pipeline
pipeline = multiply_by_ten(filter_even(get_data()))
for result in pipeline:
print(result)

This code creates a processing pipeline by nesting generator functions. The pipeline is formed by passing the output of get_data() into filter_even(), and its result into multiply_by_ten(). Nothing happens until the final for loop starts pulling values through.

• First, get_data() yields numbers from 1 to 5, one by one.
• Next, filter_even() receives each number but only passes on the even ones (2 and 4).
• Finally, multiply_by_ten() takes those results and yields them multiplied by 10, producing 20 and 40.

Get started with Replit

Put your new skills to work with Replit Agent. Just describe what you need: "a script that processes large files line-by-line" or "a data pipeline that merges two datasets and calculates a running average."

It writes the code, tests for errors, and deploys your application, handling the entire development cycle for you. Start building with Replit.