Making 100 million requests with Python aiohttp

Andy Balaam, OpenMarket
artificialworlds.net/blog

Contents

What is async?

What is async?

*Felix Geisendörfer, "Understanding node.js"

What is async?

What is async?

What is async?

import asyncio async def mycoro(number): print("Starting %d" % number) # Runs now await asyncio.sleep(1) print("Finishing %d" % number) return str(number)

What is async?

import asyncio async def mycoro(number): print("Starting %d" % number) await asyncio.sleep(1) print("Finishing %d" % number) # Runs later return str(number)

async in Python

async in Python

async in Python

import asyncio async def mycoro(number): print("Starting %d" % number) await asyncio.sleep(1) print("Finishing %d" % number) return str(number)

async in Python

import asyncio async def mycoro(number): print("Starting %d" % number) await asyncio.sleep(1) print("Finishing %d" % number) return str(number)

async in Python

>>> mycoro(3) <coroutine object mycoro at 0x7ff70b91f0a0>

async in Python

c = mycoro(3) task = asyncio.ensure_future(c) # or, Python 3.7+: task = asyncio.create_task(c)

async in Python

$ python3 python-async.py Task was destroyed but it is pending! task: <Task pending coro=<mycoro() running at python-async.py:10>> sys:1: RuntimeWarning: coroutine 'mycoro' was never awaited

async in Python

task = asyncio.ensure_future(mycoro(1)) loop = asyncio.get_event_loop() loop.run_until_complete(task) loop.close()

async in Python

# or, Python 3.7+: asyncio.run(mycoro(1))

async in Python

$ python3 python-async.py Starting 1 Finishing 1

async in Python

many = asyncio.gather( mycoro(1), mycoro(2), mycoro(3) ) asyncio.run(many)

async in Python

$ python3 python-async.py Starting 3 Starting 1 Starting 2 Finishing 3 Finishing 1 Finishing 2 ['1', '2', '3']

async in Python

import asyncio async def f2(): print("start f2") await asyncio.sleep(1) print("stop f2") async def f1(): print("start f1") await f2() print("stop f1") asyncio.run(f1())

async in Python

$ python3 python-async.py start f1 start f2 stop f2 stop f1

async in Python

HTTP requests

from aiohttp import ClientSession import asyncio async def fetch(url): async with ClientSession() as s, s.get(url) as res: ret = await res.read() print(ret) return ret asyncio.run(fetch("http://example.com"))

100 million requests!

100 million requests!

async def print_when_done(tasks): for res in asyncio.as_completed(tasks): print(await res) coros = [ fetch("http://example.com") for i in range(100_000_000) ] asyncio.run(print_when_done(coros))

100 million requests!

100 million requests!

async def print_when_done(tasks): for res in limited_as_completed(tasks, 1000): print(await res) coros = ( fetch("http://example.com") for i in range(100_000_000) ) asyncio.run(print_when_done(coros))

100 million requests!

100 million requests!

def limited_as_completed(coros, limit): futures = [ asyncio.ensure_future(c) for c in islice(coros, 0, limit) ] async def first_to_finish(): # Wait until something finishes. # Remove it from futures. # Add a new task to futures. # Return the finished one. while len(futures) > 0: yield first_to_finish()

More info

OpenMarket is hiring in London!
openmarket.com/company/careers

Read the blog post:
artificialworlds.net/blog/2017/06/12/

(or search "100 million requests aiohttp")

More info

About Andy: