Generators

Without generators Newton would not feel very pythonic at all, so I consider it a must-have for the first public release.

Example

Just like in Python, generators are collections that produce the contents on demand, one element at a time. Any function containing a yield statement is a generator:

def onetwothree():
    yield 1
    yield 2
    yield 3

You loop over them just like any other iterable:

# prints 1, 2, 3 on separate lines
for num in onetwothree():
    print(num)

A pitfall

In Python generators are stateful objects that are consumed when iterated. This has always annoyed me since it means you cannot reorder the code freely.

For instance, let’s say you want to calculate the sum of elements in gen but don’t get the expected result. To check if an element is missing you print the contents:

print(list(gen)) # prints some numbers such as [1, 2, 3]
print(sum(gen))  # prints 0!

Now the sum is zero because the first line consumed the generator, making it empty in the second line.

Implementing generators this way is easy and efficient, but since I’m not trying to be fully compatible anyway I can try to improve on it. Just got to be careful not to cause any performance issues.

Immutable generators

In Newton generators will be immutable. Sounds odd at first since you must be able to advance them somehow, but the way it will work is simply that each step returns a new generator with the remaining elements. This implies memory allocation at every iteration, which could kill performance. It is not as bad as it seems, though:

• Inlining will merge the caller and callee in most cases, turning the state into local variables that can be stored in registers.
• The generator state will in most cases be small and can be allocated on the stack.
• The stack frames are adjacent in memory and likely to be cached.

So for tight loops I expect the cost to be zero or close to it in practise, and for complex loops the additional overhead may not be that significant anyway. Eventually I may need to come up with a special solution for fat generators with a lot of state, but that can wait as long as I don’t lock myself into a particular implementation.

An unsolved problem

The net effect is that you can iterate over the same generator multiple times, pass it around and reorder the code freely, as long as it does not read from a file. Unfortunately I don’t have a solution for generators that perform IO.

Stefan