Promises, Iterators, and for-await-of in js

In this post I will go through Async Generators and how we can use them. async functions have been embraced by the community, but unfortunately generators less so. I say unfortunately because I feel they offer us some valuable tools for creating everything from asynchronous flows, to state machines, to working with lists in a lazy fashion.

That they provide a way to work lazily with lists, or contiguous bits of data, is exactly the reason that they can be used with arrays of promises, and this is exactly what for-await-of is about: Taking one value at a time, and do something with that before asking for the next one.

This is fundamentally also what Node Streams and RxJS are about. I will return to that towards the end of the post.

I should say that a prerequisite for this post is that you know at least the basics of generators and iterators. If you don’t you can probably still follow a long, but I believe you will enjoy it more if you first introduce yourself to the basic concepts.

Before we get into the meat, let’s define a function that we’ll be using a lot:

function delay(ms) {
    return new Promise(resolve => {
        setTimeout(resolve, ms)
    })
}

Calling delay with a number of milliseconds will give us a promise back that resolves after ms number of milliseconds. Simple, and useful for illustrating asynchronous work.

You will see it in code like this:

async function() {
    console.log(1)
    await delay(1000)
    console.log(2)
}

Okay, let’s go.

Promises vs Iterators

Let’s first have a look at the conceptual difference between promises and iterators. They are interesting because they are both dealing with asynchronicity between two parties.

Alice asks Bob for some data. Bob doesn’t have it yet, so he gives Alice a promise that she can use instead. The caller (Alice) wants the value right now, but Bob is the one that can not “keep up” and instead have to use asynchronous tricks instead.

async function bob() {
    await delay(500)
    return 42
}

async function alice() {
    const n = await bob()
    console.log(n)
}

alice() // logs 42 after 500ms

What if Alice wants data from Bob (that he has readily available), but she does not want it all right now? A promise is no good. If Alice does not want all the data right now, then she certainly does not want a promise of the data either. Instead she gets an iterator where she can pull values out of as she needs them.

function* bob() {
    let c = 42
    while(true) {
        yield c
        c += 2
        if (c > 46) {
            break
        }
    }
}

async function alice() {
    const iterator = bob()
    for (const n of iterator) {
        console.log(n)
        await delay(1000)
    }
}

alice() // prints 42, 44, 46, with 1 sec delays

So this time Alice is the one needing asynchronous tools. In the example above, she is pulling out values in a loop with one second delays.

She can also do this manually, without the loop.

async function alice() {
    const iterator = bob()
    console.log(iterator.next()) // {value: 42, done: false}
    console.log(iterator.next()) // {value: 44, done: false}
    console.log(iterator.next()) // {value: 46, done: false}
    console.log(iterator.next()) // {value: undefined, done: true}
}

The for-of loop understands iterators, so it makes them fairly easy to work with.

What I want you to understand from this, is that in as certain light promises and iterators are similar tools. They are both tools for dealing with asynchronous code, depending on who (the caller, Alice, or the called, Bob) isn’t ready yet.

But what if both may or may not yet be ready? Alice wants an iterator, and Bob wants to be an async function. However, async functions returns promises. Well, we can just combine it all.

async function* bob() {
    let c = 42
    while(true) {
        await delay(500)
        yield c
        c += 2
        if (c > 46) {
            break
        }
    }
}

We now turned Bob into an AsyncGenerator (see the async keyword and the *). We can take our latest version of Alice and modify her slightly to deal with the new version of Bob.

async function alice() {
    const iterator = bob()
    iterator.next().then(console.log) // {value: 42, done: false}
    iterator.next().then(console.log) // {value: 44, done: false}
    iterator.next().then(console.log) // {value: 46, done: false}
    iterator.next().then(console.log) // {value: undefined, done: true}
}

However we can’t use for-of to loop over it.

async function alice() {
    const iterator = bob()
    for (const p of iterator) {
        p.then(console.log)
    }
}

alice() // TypeError: iterator is not iterable

Instead we use the new for-await-of:

async function alice() {
    const iterator = bob()
    for await (const n of iterator) {
        console.log(n)
    }
}

This works. And it is simple to understand. Maybe not easy, but considering how difficult it can be to deal with asynchronous code, this model seems simple and explicit.

Streams of data

If we take a step back, and look at what we are dealing with, we may realize that these tools deal with, in principle, the same as both Node Streams and RxJS: contiguous bits of data flowing through a set of functions.

What can we use this for? Here are some examples:

Reading and writing to big files
HTTP requests and responses on the client
HTTP requests and responses on the server
TCP sockets
child process stdout and stderr
stdin
Events in the browser (mouse clicks, mouse moves, etc.)

Let’s create something that we can use as an eventListener. Essentially what we need, is something with an interface that has two functions:

.read(): Returns an AsyncGenerator. It will yield promises to us
.write(value): A function that, whenever we call it, we (if anyone pulled out a promise from the reader) resolve the promise

We will call it copy, and here is the code:

function copy() {
    const resolves = []

    async function* read() {
        while(true) {
            const value = await new Promise(function(resolve) {
                resolves.push(resolve)
            })
            yield value
        }
    }

    function write(data) {
        for (const resolve of resolves) {
            resolve(data)
        }
        resolves.length = 0 // clear array
    }

    return {read, write}
}

The read function will yield promises, but instead of resolving them itself, it just pushes the resolve function onto an array that it shares with the write function. When write is called with a value, it resolves all the pending promises and then clears them out so it doesn’t call them again (nothing would happen if it did, but the array will grow in size quickly if we don’t).

We can use it like this:

const c = copy()

document.querySelector(".my-button").addEventListener("click", c.write)

async function alice() {
    const iterator = c.read()
    for async (const event of iterator) {
        console.log(event)
    }
}

alice()

This will log all clicks on .my-button. This in itself is not impressive (we could just have added console.log as the event listener), but it gives us a different interface to work with.

AsyncGenerator utils

Let’s write a simple map function for our async iterators:

async function* map(iterator, fn) {
    for await (const x of iterator) {
        yield fn(x)
    }
}

While we are at it, let’s do a filter also:

async function* filter(iterator, fn) {
    for await (const x of iterator) {
        if (fn(x)) {
            yield x
        }
    }
}

We can use them like this

async function alice() {
    const iterator = bob()
    const mappedIterator = map(iterator, n => n + 3)
    const filteredIterator = filter(mappedIterator, n => n !== 47)

    for await (const n of filteredIterator) {
        console.log(n) // logs 45, then 49
    }
}

We can see that the map “eats” the iterator and creates a new iterator. The filter also “eats” an iterator (our new mappedIterator) and creates its own iterator. We could write it like this instead:

async function alice() {
    const filteredIterator = filter(map(bob(), n => n + 3), n => n !== 47)

    for await (const n of filteredIterator) {
        console.log(n) // logs 45, then 49
    }
}

This looks like the sort of craziness you get into with functional code, and that is exactly what it is. So let’s get inspired and change map and filter a bit so it takes the function first, and then returns a function that takes an iterator:

const map = fn => async function* (iterator) {
    for await (const x of iterator) {
        yield fn(x)
    }
}

const filter = fn => async function* (iterator) {
    for await (const x of iterator) {
        if (fn(x)) {
            yield x
        }
    }
}

Now we can compose them more easily. Especially if we have a pipe function:

function pipe(iterator, writer) {
    return writer(iterator) // returns an iterator
}

The name writer here refers to a function that takes an iterator.

Since this is a reducer function (it takes two parameters and returns a value of the same type as the first parameter), let’s just go ahead and let pipe take an arbitrary number of functions:

function pipe(...fns) {
    return fns.reduce((iterator, writer) => writer(iterator))
}

Now we can compose our map and filter:

async function alice() {
    const iterator = pipe(
        bob(),                  // an iterator
        map(n => n + 3),        // a writer
        filter(n => n !== 47)   // a writer
    )

    for await (const n of iterator) {
        console.log(n) // logs 45, then 49
    }
}

Now it get’s easier to read. With the pipeline operator it becomes even more consize:

const iterator = bob()
    |> map(n => n + 3)
    |> filter(n => n !== 47)

We’ll create one more util function, throttle, and then hook it up with some events.

const throttle = ms => async function* (iterator) {
    let time
    for await (const x of iterator) {
        const diff = Date.now() - time
        if (time && diff < ms) {
            await delay(ms - diff)
        }
        yield x
        time = Date.now()
    }
}

And let’s put it all together:

const c = copy()
const box = document.querySelector(".box")
box.addEventListener("mousemove", c.write)

const iterator = pipe(
    c.read(),
    throttle(500),
    map(event => ({x: event.clientX, y: event.clientY})),
    filter(({x, y}) => x % 2 === 0 && y % 2 === 0)
)

(async function(iterator) {
    for await (const coords of iterator) {
        box.innerHTML = `The mouse is at ${x}, ${y}.`
    }
})(iterator)

I added the code to this blog post, where .box is the box below. If your browser is of a newer kind, then you should be able to see the coords of the mouse when hovering it… but it’ll only update every 500ms, and only if the coordinates are both even.

A library

I wrote a little library with some (most?) of the utility functions you’d expect ready for use. I will admit that it is perhaps a bit obscure, but it does solve a certain class of problems and gives you a unified interface to a whole host of other more common usecases.

Interesting technology anyway, and I am happy to see for-await-of land in the browsers and Node.

♘

Promises, Iterators, and for-await-of in js

Promises vs Iterators

Streams of data

AsyncGenerator utils

A library