Python has a long history of async programming, notably through the twisted, gevent and Stackless Python projects. In Python 3.4 the asyncio module was introduced, bringing some of this work into the Python core. Python 3.5 in turn added new syntax support with the async def and await statements. Let's see how they work.
Asynchronous programming
Async is shorthand for cooperative multitasking, a style of multitasking where the currently running task does not get interrupted, but must yield control to other tasks. In the normal case, it may fire off a network or IO request and then sleep until the request comes back with data. During the time that it sleeps, other tasks may run and submit their requests in the same way.
This type of multitasking is simpler to write and less error prone than using threads, because tasks only start and stop at precisely defined points. It's not suitable for CPU-bound parallelism, but it's very suitable for issuing or responding to many network requests at once.
Let's talk about how this is done now in Python core.
Coroutines
Tasks that you execute are called coroutines, and they are scheduled by the event loop. The loop is responsible for deciding which coroutine to begin next. This might depend on network or filesystem events, or on timers that particular coroutines are waiting for.
Let's begin by making a totally normal function a coroutine. This function just returns twice its argument as a result:
def double(x):
return x * 2
In Python 3.4, we can make this a coroutine with the asyncio.coroutine decorator:
import asyncio
@asyncio.coroutine
def double(x):
return x * 2
In Python 3.5 we have the async def syntax instead:
async def double(x):
return x * 2
What remains is not a normal function that you can call as you did before, but one that returns a coroutine when called.
>>> double(6)
<coroutine object double at 0x115b59d58>
Because coroutines and generators are intimately related, we can sneakily get the result back by treating it like a special type of generator:
>>> double(6).send(None)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
StopIteration: 12
But this is not how coroutines are meant to be run. You're meant to use an event loop:
>>> loop = asyncio.get_event_loop()
>>> coro = double(6)
>>> loop.run_until_complete(coro)
12
Notice that the run_until_complete method passed through the return value for us. Through the event loop, you can schedule coroutines in a large variety of ways.
Chaining coroutines
In the previous example, we converted a normal function to an asynchronous form. In doing so, we lost the ability to call the function normally and didn't really gain any benefit. The benefit comes when coroutines are chained together.
Suppose we want to check the status code of a page, to see if it's responding properly. Here's a synchronous version, using requests:
import requests
def check_status(url):
resp = requests.get(url)
return resp.status_code
If we make this naively into a coroutine, it will not be a good one:
async def check_status(url):
resp = requests.get(url)
return resp.status_code
Whenever this is run, it will block whilst getting the requests, and hog the event loop, preventing other coroutines from running. Instead, we need to use another coroutine to fetch the url. We'll take it from the aiohttp library instead of requests:
import aiohttp
async def check_status(url):
resp = await aiohttp.request('GET', url)
return resp.status_code
Here the await syntax means "block until the coroutine finishes". But whilst it's blocking, control is returned to the event loop, so that other things can run.
Notice that await only works inside a coroutine. Outside that, syntax error!
>>> await asyncio.sleep(1)
File "<stdin>", line 1
await asyncio.sleep(1)
^
SyntaxError: invalid syntax
Closing thoughts
In my short time experimenting with programming in this async style, it's struck me how sticky it is. Everything you do that directly or indirectly depends on IO will be drawn into the async style. The async style limits reuse, so it only makes sense to write coroutines for things which need it. For everything else, keep using plain old ordinary functions.
Within the async world, you get to program in a style that's very similar to the old blocking style, with just a few new keywords and a change of library here and there. That's a pretty small cost for a large benefit; I think the Python core developers have done well.