interrupt issues with async code

[mlucool]

Hi,

Aside from the now fixed #881, I still run into some async + interrupt issues. I'm not 100% sure if I need to do something differently or there is a way to do what we mean.

The following is not interrupted and always run to completion. If you sleep instead of httpx.get (or request.get), it works as expected.

import asyncio
import httpx
from tqdm.asyncio import tqdm as tqdm_async

semaphore = asyncio.Semaphore(1)

x = 0
async def one_at_a_time():
    global semaphore

    async with semaphore:
        global x
        # await asyncio.sleep(1)
        httpx.get('http://httpbin.org/delay/1')
        x += 1
        print(x)
    return x

await tqdm_async.gather(*[one_at_a_time() for _ in range(10)])

Ideally, I want these to be tasks so I can cancel them. Again, interrupt has no effect:

import asyncio
from time import sleep
from tqdm.asyncio import tqdm as tqdm_async

semaphore = asyncio.Semaphore(1)
x = 0
async def one_at_a_time():
    global semaphore

    async with semaphore:
        global x
        sleep(1)
        x += 1
        print(x)
    return x

async def outer_async():
    return await one_at_a_time()
    
await tqdm_async.gather(*[asyncio.create_task(one_at_a_time()) for _ in range(10)])

Any suggestions for how to fix my code to work better with ipykernel or how to fix ipykernel (or IPython) itself? Even better would be something that works on 6.x, but I think that may be challenging as even #881 is not fixed there.

[davidbrochart]

Your example doesn't really make sense if the GET request is not async. This seems to work fine:

import asyncio
import httpx
from tqdm.asyncio import tqdm as tqdm_async

semaphore = asyncio.Semaphore(1)

x = 0
async def one_at_a_time():
    global semaphore

    async with semaphore:
        global x
        # await asyncio.sleep(1)
        # httpx.get('http://httpbin.org/delay/1')
        async with httpx.AsyncClient() as client:
            await client.get('http://httpbin.org/delay/1')
        x += 1
        #print(x)
    return x

await tqdm_async.gather(*[one_at_a_time() for _ in range(10)])

[mlucool]

Thanks for the pointer @davidbrochart. I thought we'd get some parallelism with create_task - but clearly that is incorrect since the blocking code does not return. Either way, I would still expect interrupt to have an effect.

FWIW, the above example was contrived since some API I use calls get under the hood. I'll have to find a way to change that.

[minrk]

A much simpler repro for failure to interrupt async code is to make a blocking call:

import time

async def blocking_wait():
    time.sleep(10)

await blocking_wait() # < cannot interrupt

This is because sigint_handler effectively assumes that async code never blocks, relying on cancel which doesn't actually halt execution until it yields.

I think we need to always raise KeyboardInterrupt(), even when running async code. I know it's complicated because the async loop might be running at that time, but I think we need to handle it differently so that blocking code is still properly interrupted even when the outer call is async.

[davidbrochart]

This is because sigint_handler effectively assumes that async code never blocks, relying on cancel which doesn't actually halt execution until it yields.

Right, but note that with current ipykernel your code snippet cannot be interrupted either. Even an async wait cannot be interrupted currently:

import asyncio

await asyncio.sleep(10) # < cannot interrupt

While it can with main, which is an improvement (see #881).