connection hangs randomly

[princesscrypto1337]

Hi,

At random times during my notebook session, a sql statement like %sql select count(*) from table will just hang even though the calculation should be fast. This hang cannot be interrupted by jupyter, so I always have to restart the kernel. I can confirm that my postgresql server is still running and I can run the same command when I log in through a client, so this should not be a server side problem.

Wondering if anyone else has seen this and if there's a workaround or solution.

[idomic]

@princesscrypto1337 does it still happen to you (I know this issue is pretty old)

[earlev4]

Huge thanks and appreciation to the contributors of the ipython-sql project! Very grateful for the project!

Hi @idomic. This problem is occurring for me. As @princesscrypto1337 mentions, the problem is random. Usually occurs after a lot of interaction with the Jupyter notebook. Like the issue reported, I am also using %sql select count(*). It also occurs in %sql create or replace table.... The issue seems to happen if I run those statements a couple or few times. In my case, I am using DuckDB. At first, I was using JupySQL and I thought it was a DuckDB issue. However, after troubleshooting the issue, and eliminating %sql and using a direct con.execute() the issue was no longer persistent. Just for context, using con.execute() the select count(*) only takes around 40-50 seconds. Letting %sql select count(*) run, can run for hours until stopped. I uninstalled JupySQL and tried ipython-sql (version: 0.4.1) and the issue appeared again (once again random). When the progress bar from DuckDB is available, the progress bar usually indicates the hang occurs at different times (32%, 38%, 41%, 88%, 90%, etc.). When the progress bar is not available, I just get the * indicating the code cell is currently executing. In my situation, I can interrupt the kernel. After interrupting the kernel, I tried %reload_ext sql and the problem still persists. Restarting the kernel, I can usually use ipython-sql again for a %sql select * from table limit 5, but then the issue starts to happen again when I do %sql select count(*). For now, I have suspended use. Thanks for your interest!

[earlev4]

Hi. After exhaustive trial and error, the DuckDB problem previously mentioned does not appear to be related to the ipython-sql or JupySQL extensions. The issue with the cell hanging in Jupyter occurs when the DuckDB configuration is SET enable_progress_bar=true;. There appears to be an issue already already opened in the DuckDB repository - 5407.

Sorry to confuse the issue reported by @princesscrypto1337 for PostgreSQL. However, if someone is encountering cell hangs in Jupyter with DuckDB, try disabling the progress bar until the issue is resolved. Thanks!

[idomic]

@edublancas

[edublancas]

The main issue here is to reproduce the problem. However, I think it'd still be valuable to provide a workaround, perhaps a timeout? So that cells that take too long abort, and there isn't a need to restart the kernel.

Would this help? @princesscrypto1337