Fixes #4067

This PR implements downloads for tables using the following approach:

Querying data:

• Creates a database cursor that queries records
• Fetches records in chunks of 2000 rows

Transferring:

• Forms csv from the results of each fetch
• Streams the csv to the browser, which saves it as a file

Approaches tested

Querying data:

• COPY command:
  • Faster if we only need to export entire non-foreign table.
  • Can form CSV output directly on the database.
  • For our requirements, we would have more use for the syntax COPY (SELECT * FROM table) TO instead of COPY table TO (since we need filtering & sorting and support for explorations in the future). This syntax has negligible performance difference compared to reading from a cursor.
  • COPY is not supported inside a db function.
• Multiple queries with standard limit and incremental offsets:
  • Can use the existing list function.
  • Flexible for our needs.
  • Chances of data changing under the hood between queries, so the end result is unreliable.
• DB cursor:
  • Can fetch & transfer in chunks.
  • Flexible for our needs.
  • Data is reliable since querying is part of the same transaction.
  • Reasonably fast, and since it fetches data in chunks, we have more control over memory and cpu usage.

Transferring:

• Direct transfer:
  • Forms the entire file in the server and transfers it.
  • Error handling is easier.
  • Processing time is higher as we wait for the entire file to complete and the request is kept open until then.
  • Higher chances of network disconnects since webservers/proxies/load-balancers might cut off idle connections after a threshold.
  • Cannot update progress on the frontend.
• Streaming:
  • Streams chunk by chunk to the frontend.
  • Error handling is additional work.
  • Keeps the connection open and active, so webservers/proxies/load-balancers would not cut them off.
  • Can update progress on the frontend.
• Storing as a file and allowing download via link:
  • Stores the csv data as a file on the Mathesar server.
  • Allows the frontend to download it via a link.
  • Requires additional permission checks.
  • Makes it possible to view export history and audit access.
  • Ideal for long-term, however we cannot do that without figuring out storage flow for Mathesar.
  • Not feasible for v0.2.0.

What is not implemented in this PR:

• Error handling i.e. viewing exact cause for failure on the frontend:
  • We cannot return a different status code while a streaming request in underway.
    • Say a few chunks have been sent, and one fails due to an unforeseen error.
    • We cannot return a different http status code separately for that chunk.
  • The download is directly made by the browser via the a['download'] tag.
  • This lets the browser take over and download the streamed data. When there's a failure, the browser would show an error. However, it cannot show the actual cause since streaming requests do not return the error.
  • We can handle this by streaming as json with each chunk containing a status code and data, and later joining the data on the frontend.
  • This isn't implemented in this PR, since it's additional work with little benefit. The only major cause for streaming to fail is due to network failures, which the browser would display.
  • We could implement this as a nice-to-have post v0.2.0, or when users report issues.

UX changes from the design

• The UX is made more straight-forward since download is handled directly by the browser.
• Here is a video of the interaction:

Checklist

• My pull request has a descriptive title (not a vague title like Update index.md).
• My pull request targets the develop branch of the repository
• My commit messages follow best practices.
• My code follows the established code style of the repository.
• I added tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no
  visible errors.

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1

Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129

Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.


Developer's Certificate of Origin 1.1

By making a contribution to this project, I certify that:

(a) The contribution was created in whole or in part by me and I
    have the right to submit it under the open source license
    indicated in the file; or

(b) The contribution is based upon previous work that, to the best
    of my knowledge, is covered under an appropriate open source
    license and I have the right under that license to submit that
    work with modifications, whether created in whole or in part
    by me, under the same open source license (unless I am
    permitted to submit under a different license), as indicated
    in the file; or

(c) The contribution was provided directly to me by some other
    person who certified (a), (b) or (c) and I have not modified
    it.

(d) I understand and agree that this project and the contribution
    are public and that a record of the contribution (including all
    personal information I submit with it, including my sign-off) is
    maintained indefinitely and may be redistributed consistent with
    this project or the open source license(s) involved.