Marcell Ciszek Druzynski

Iterators in Go

Understanding Iterators in the Go Language and Their Benefits

August 08, 2024

We finally have iterators in the Go programming language, and I couldn't be more excited about it! If you're unfamiliar with what an iterator is, it's a technique that many other languages, such as JavaScript, Java, and Python, have been using for some time.

Since the introduction of generics in Go 1.18, it was only a matter of time before iterators made their way into the language. In this article, I'll explain the basics what iterators are, how they work, and why they're beneficial for Go developers.

What is an Iterator?

An iterator is an object that allows you to access elements of a collection sequentially, without exposing the underlying structure of that collection. This makes it easier to work with data in a clean and efficient manner.

Overall, the addition of iterators enhances the Go language, making it more versatile and aligned with modern programming practices.

A common use case for iterators is to loop over a collection of items, such as an array or a list, and perform some operation on each item. This can be done using a for loop or other constructs that support iteration.

Let's take an example from JavaScript. In JavaScript, iterators are objects that define a next() method, allowing you to access elements of a collection sequentially. Here’s how they work:

1. Iterator Protocol

An iterator must implement the iterator protocol by providing a next() method. This method returns an object with two key properties:

  • value: The next value in the sequence.
  • done: A boolean indicating whether the iteration is complete.

2. Iterable Protocol

For an object to be considered iterable, it must implement the iterable protocol by including a method with the key Symbol.iterator. This method should return an iterator object.

3. Usage of Iterators

Iterators are commonly used with several JavaScript features, including:

  • for...of loops: To iterate over iterable objects.
  • Spread operator (...): To expand iterable elements into individual elements.
  • Destructuring assignments: To unpack values from arrays or objects.

This overview provides insight into what you can do with iterators and their general functionality. While iterators are not yet fully integrated into the Go language, until we get Go version 1.23.

There has been considerable debate within the Go community regarding the introduction of iterators. Some developers express concerns that iterators may add unnecessary complexity to the language. This has led to mixed feelings about the decision.

I appreciate the simplicity and readability that Go offers. However, I can understand the criticism regarding the syntax for creating an iterator function, which can appear somewhat convoluted.

Example: Iterating a Slice Backwards To illustrate this point, consider the following function for iterating over a slice in reverse. This example is adapted from go.dev:

1package slices
2
3func Backward[E any](s []E) func(func(int, E) bool) {
4    return func(yield func(int, E) bool) {
5        for i := len(s)-1; i >= 0; i-- {
6            if !yield(i, s[i]) {
7                return
8            }
9        }
10    }
11}
1package slices
2
3func Backward[E any](s []E) func(func(int, E) bool) {
4    return func(yield func(int, E) bool) {
5        for i := len(s)-1; i >= 0; i-- {
6            if !yield(i, s[i]) {
7                return
8            }
9        }
10    }
11}

And invoking the Backward() function:

1s := []string{"hello", "world"}
2	for i, x := range Backward(s) {
3		fmt.Println(i, x)
4	}
1s := []string{"hello", "world"}
2	for i, x := range Backward(s) {
3		fmt.Println(i, x)
4	}

As expected we get the strings world and hello printed to the console.

By looking at the function definition of Backward() it does not look intimidating to write that kind of code. Imagine if you would have something more complex logic then going backwards of an slice.

In Go version 1.23 we will get a new package, named iter that will help us with simplifying the code and make it more readable to write own iterators.

The Go team has introduces to us two different kinds of iterators from the new iter package. We got, Single-value iterators iter.Seq and Two-value iterators: iter.Seq2. With these two new types it will be easier for us to describe what our iterator return type is. So for example using the iter.Seq:

1func Numbers(xs []int) iter.Seq[int] {
2	return func(yield func(int) bool) {
3		for _, v := range xs {
4			if !yield(v) {
5				return
6			}
7		}
8	}
9}
1func Numbers(xs []int) iter.Seq[int] {
2	return func(yield func(int) bool) {
3		for _, v := range xs {
4			if !yield(v) {
5				return
6			}
7		}
8	}
9}

Important to know is that the Numbers() is not the iterator here, it basically return an iterator of type iter.Seq[int]

And then loop over Numbers() to print the values.

1	xs := []int{10, 20, 30, 40, 50}
2	for x := range Numbers(xs) {
3		fmt.Println(x)
4	}
1	xs := []int{10, 20, 30, 40, 50}
2	for x := range Numbers(xs) {
3		fmt.Println(x)
4	}

Since we used iter.Seq[int] that only gives back one value at the time trying to get for example the index will not work. We would have to modify the Numbers() a bit so it returns iter.Seq2[int,int] instead.

1
2func Numbers(xs []int) iter.Seq2[int, int] {
3	return func(yield func(int, int) bool) {
4		for i, v := range xs {
5			if !yield(i, v) {
6				return
7			}
8		}
9	}
10}
1
2func Numbers(xs []int) iter.Seq2[int, int] {
3	return func(yield func(int, int) bool) {
4		for i, v := range xs {
5			if !yield(i, v) {
6				return
7			}
8		}
9	}
10}
1xs := []int{10, 20, 30, 40, 50}
2	for i, x := range Numbers(xs) {
3		fmt.Println(i, x)
4	}
1xs := []int{10, 20, 30, 40, 50}
2	for i, x := range Numbers(xs) {
3		fmt.Println(i, x)
4	}
Short summery about the iter.Seq and iter.Seq2
  • iter.Seq[V any]: This type is a function that takes a yield function as a parameter. The yield function accepts a value of type V and returns a boolean. If the yield function returns false, the iteration stops.
  • iter.Seq2[K, V any]: Similar to iter.Seq, but it yields two values: a key of type K and a value of type V.

Where Are the Map, Filter, and Reduce Functions?

You might be wondering where the functional higher-order functions like map, filter, and reduce are, which are commonly found in functional programming languages and are widely used in languages like JavaScript, Java, and Rust.

Currently, if we want to use map, for instance, we need to create our own implementations. However, with the introduction of iterators in Go, it becomes feasible to implement these functions more naturally.

A Defensive Approach to New Features

I appreciate how the Go team introduces new features with a cautious perspective. By allowing developers to create their own implementations initially, the community can evaluate whether there is a genuine demand for these features.

I believe that with the new iterators, developers will be able to use higher-order functions effectively, and it’s only a matter of time before these functions are officially integrated into the language.

While this is just my speculation, the potential for incorporating map, filter, and reduce into Go seems promising as the community adapts to the new iterator functionality.

Here’s an example of how to implement your own map and filter functions using iterators in Go. These functions will allow you to process collections in a functional style.

1package main
2
3import (
4	"fmt"
5	"iter"
6)
7
8func main() {
9	numbers := func(yield func(int) bool) {
10		for i := 1; i <= 10; i++ {
11			if !yield(i) {
12				return
13			}
14		}
15	}
16
17	// Filter even numbers and then square them using Map
18	evenSquares := Map(
19		Filter(numbers, func(n int) bool { return n%2 == 0 }),
20		func(n int) int { return n * n },
21	)
22
23	for v := range evenSquares {
24		fmt.Println(v)
25	}
26}
27
28func Filter[T any](input iter.Seq[T], predicate func(T) bool) iter.Seq[T] {
29	return func(yield func(T) bool) {
30		for v := range input {
31			if predicate(v) {
32				if !yield(v) {
33					return
34				}
35			}
36		}
37	}
38}
39
40func Map[T, U any](input iter.Seq[T], transform func(T) U) iter.Seq[U] {
41	return func(yield func(U) bool) {
42		for v := range input {
43			if !yield(transform(v)) {
44				return
45			}
46		}
47	}
48}
49
1package main
2
3import (
4	"fmt"
5	"iter"
6)
7
8func main() {
9	numbers := func(yield func(int) bool) {
10		for i := 1; i <= 10; i++ {
11			if !yield(i) {
12				return
13			}
14		}
15	}
16
17	// Filter even numbers and then square them using Map
18	evenSquares := Map(
19		Filter(numbers, func(n int) bool { return n%2 == 0 }),
20		func(n int) int { return n * n },
21	)
22
23	for v := range evenSquares {
24		fmt.Println(v)
25	}
26}
27
28func Filter[T any](input iter.Seq[T], predicate func(T) bool) iter.Seq[T] {
29	return func(yield func(T) bool) {
30		for v := range input {
31			if predicate(v) {
32				if !yield(v) {
33					return
34				}
35			}
36		}
37	}
38}
39
40func Map[T, U any](input iter.Seq[T], transform func(T) U) iter.Seq[U] {
41	return func(yield func(U) bool) {
42		for v := range input {
43			if !yield(transform(v)) {
44				return
45			}
46		}
47	}
48}
49

And we would get the result:

14
216
336
464
5100
14
216
336
464
5100

At the outset, it's likely that the Go community will implement custom solutions similar to what we see with libraries like Lodash in JavaScript. A third-party library could emerge to provide higher-order functions such as map, filter, and reduce.

If these functions gain traction and the Go community expresses a strong demand for them, it’s plausible that they will eventually become part of the Go language itself. This would enhance the language's capabilities and align it more closely with functional programming paradigms.

Benefits

What are the benefits with using iterators? I would say there are a lot of benefits of using iterators. We get improved memory efficiency for large datasets, where iterators help allow us to processing one item at a time instead of loading an entire slice into memory. Iterators generate values on-demand rather than pre-computing everything upfront, it is a pull operation and not a push operation. We can also use iterators to create infinite sequences, which is not possible with slices.

Right now there are arguments that the new iterators are not readable and brings a lot of complexity to the language and I am agree. But I do think when devs will get used to the iterators and we have a more common way to use them together with usable higher order functions it will get more readable and easier to grasp about.

The Go team believes iterators will simplify many common programming patterns and improve performance for certain use cases. I am excited to see how the Go community will adopt and utilize iterators in their projects.

Summary

Go 1.23 introduces new iterator functionality, including the iter.Seq and iter.Seq2 types, which provide a simpler way to define and use iterators. While some in the Go community are concerned that iterators add unnecessary complexity to the language, others argue they will simplify common programming patterns and improve performance for certain use cases.

The new iter package provides two main iterator types:

  • iter.Seq[T any]: A function that takes a yield function accepting a value of type T and returns a boolean. Iteration stops if yield returns false.
  • iter.Seq2[K, V any]: Similar to iter.Seq, but yields both a key of type K and a value of type V.

Using these iterator types, it becomes easier to implement common higher-order functions like Filter and Map that operate on iterators.

While these higher-order functions are not yet built into the language, they are expected to be provided by third-party libraries initially. If widely adopted, they may eventually be added to the Go standard library.

So, go out and try them out! The new iterator functionality in Go 1.23 opens up exciting possibilities for more expressive and efficient code. Happy coding!

Resources