Python Generators – Yield, Syntax & Examples

Introduction

When working with large datasets in Python, storing all values in memory at once can be inefficient and sometimes impossible due to memory limitations. To solve this problem, Python provides a powerful feature called Generators.

Generators are special functions that allow you to generate values one at a time instead of creating and storing the entire sequence in memory. They are memory-efficient, faster for large datasets, and widely used in data processing, file handling, web scraping, and machine learning applications.

For example, instead of creating a list containing one million numbers, a generator can produce each number only when needed.

What are Python Generators?

A generator is a special type of function that returns an iterator and generates values lazily (on demand).

Instead of using return a generator uses yield.

Example

def numbers():
    yield 1
    yield 2
    yield 3

Usage:

num = numbers()
print(next(num))

Output:

The generator produces one value at a time.

Why Use Generators?

Generators provide several advantages.

1. Memory Efficient

They do not store all values in memory.

2. Faster Execution

Values are generated only when needed.

3. Lazy Evaluation

Computations occur on demand.

4. Suitable for Large Data

Ideal for processing large files and datasets.

5. Easy to Implement

Simpler than creating custom iterator classes.

Generator vs Normal Function

Normal Function

def show_numbers():
    return [1, 2, 3]

Usage:

print(show_numbers())

Output:

The entire list is created in memory.

Generator Function

def show_numbers():
    yield 1
    yield 2
    yield 3

Usage:

numbers = show_numbers()
print(next(numbers))

Output:

Values are generated one at a time.

Syntax

Basic syntax:

def generator_name():
    yield value

Example:

def fruits():
    yield "Apple"
    yield "Banana"
    yield "Mango"

Usage:

fruit = fruits()
print(next(fruit))

Output:

Understanding the yield Keyword

The yield keyword is the heart of generators.

When Python encounters yield:

1. It returns the value.
2. It pauses the function.
3. It saves the current state.
4. Execution resumes from the same point later.

Example:

def demo():
    print("First")
    yield 1
    print("Second")
    yield 2

Usage:

obj = demo()
print(next(obj))
print(next(obj))

Output:

Notice that the function resumes where it previously stopped.

Creating a Simple Generator

Example:

def count():
    yield 1
    yield 2
    yield 3
    yield 4
    yield 5

Usage:

counter = count()
for num in counter:
    print(num)

Output:

Using next() with Generators

Generators produce values one at a time.

Example:

def numbers():
    yield 10
    yield 20
    yield 30

Usage:

obj = numbers()
print(next(obj))
print(next(obj))
print(next(obj))

Output:

StopIteration Exception

When all values are exhausted, Python raises:

StopIteration

Example:

def numbers():
    yield 1

Usage:

obj = numbers()
print(next(obj))
print(next(obj))

Output:

Generator with Loops

Generators often use loops to generate values dynamically.

Example:

def countdown(n):
    while n > 0:
        yield n
        n -= 1

Usage:

for num in countdown(5):
    print(num)

Output:

Generator for Even Numbers

Example:

def even_numbers(limit):
    for num in range(limit):
        if num % 2 == 0:
            yield num

Usage:

for num in even_numbers(10):
    print(num)

Output:

Generator Expressions

Python supports generator expressions similar to list comprehensions.

List Comprehension

numbers = [x*x for x in range(5)]
print(numbers)

Output:

Generator Expression

numbers = (x*x for x in range(5))
print(numbers)

Output:

Usage:

for num in numbers:
    print(num)

Output:

Generator Expression vs List Comprehension

Real-Life Examples:

1. Reading Large Files

Without Generator:

file = open("data.txt")
lines = file.readlines()

Problem:

All lines are loaded into memory.

Using Generator:

def read_file(filename):
    with open(filename) as file:
        for line in file:
            yield line

Usage:

for line in read_file("data.txt"):
    print(line)

Benefits:

• Memory efficient
• Faster processing
• Handles huge files

2. Employee ID Generator

def employee_ids():
    emp_id = 1000
    while True:
        emp_id += 1
        yield emp_id

Usage:

ids = employee_ids()
print(next(ids))
print(next(ids))
print(next(ids))

Output:

Useful for generating unique IDs.

3. Fibonacci Series

def fibonacci(n):
    a, b = 0, 1
    for _ in range(n):
        yield a
        a, b = b, a + b

Usage:

for num in fibonacci(10):
    print(num)

Output:

Generators are Iterators

Every generator is automatically an iterator.

Example:

def numbers():
    yield 1
    yield 2

Check:

obj = numbers()
print(iter(obj) == obj)

Output:

Generators automatically implement:

__iter__()
__next__()

Advantages of Generators

Disadvantages of Generators

Generators vs Iterators

Example:

Generator:

def numbers():
    yield 1

Iterator:

class Numbers:
    def __iter__(self):
        return self
    def __next__(self):
        pass

Generators are usually easier to implement.

Common Mistakes

1. Using return Instead of yield

Incorrect:

def numbers():
    return 1
    return 2

Correct:

def numbers():
    yield 1
    yield 2

2. Forgetting That Generators Are Single-Use

Incorrect:

gen = numbers()
for x in gen:
    print(x)
for x in gen:
    print(x)

Second loop produces no output.

3. Converting Large Generators to Lists

Incorrect:

list(large_generator)

This defeats the memory-saving benefits.

4. Calling next() Too Many Times

Incorrect:

next(gen)
next(gen)

After exhaustion StopIteration is raised.

5. Using Generators When Random Access Is Needed

Generators do not support:

gen[0]

Use lists if indexing is required.

Best Practices

1. Use Generators for Large Data

Ideal for:

• Large files
• Data streams
• Database records

2. Prefer Generator Expressions

Instead of:

[x*x for x in range(1000000)]

Use:

(x*x for x in range(1000000))

for better memory efficiency.

3. Keep Generator Logic Simple

Avoid overly complex generator functions.

4. Use Meaningful Names

Good:

generate_ids()
read_file()
fibonacci()

Bad:

gen()
func()

5. Handle Exhaustion Properly

Always be aware that generators can only be consumed once.

Conclusion

Python Generators are a powerful feature that enables efficient and memory-friendly data processing. By using the yield keyword, generators produce values one at a time rather than storing entire datasets in memory.

They are ideal for large files, data streams, infinite sequences, and performance-critical applications.

Python Generators – Interview Questions

Python Generators – Objective Questions (MCQs)

def gen():
for i in range(3):
yield i
g = gen()
print(next(g))
print(next(g))

Related Python Tutorials