- Prevent bad task serialization in schedule from causing a batch of tasks to be lost, see #815..
- Ensure we catch ResultTimeout which may occur when used with Sentinel, #813.
- Remove junk SQS implementation I was testing out.
- More makework thanks to the ass-clowns running Python. Fix issue with
deprecation of
datetime.utcnow()in 3.12. - Add API for customizing the
TaskWrapperimplementation, suitably namedget_task_wrapper_class(). - Make the
revoke_all(),restore_all()andis_revoked()more robust for various input types. - Fix bug that could occur in the event of a SIGHUP followed by a SIGINT when using thread workers.
- Added new experimental contrib module for SQS queue and S3 result storage.
- Check to ensure the gevent monkeypatch was applied when running the consumer with greenlet workers, log warning if it is not.
- Explicitly clear the revoked flag on task instances after execution (#713). This will help reduce junk keys left in the storage if you attempt to revoke a task while it is executing.
- Add support for
delay=,eta=in Huey's.s()and.then()- this adds support for delaying or scheduling pipelines. - Add support for rescheduling callback pipelines when rescheduling a task.
This is enabled by default (
preserve_pipeline=True). - Add an
on_commit_task()decorator for Django extension that will enqueue the task after any database changes have been committed. This eliminates a common race condition where a task is enqueued and executed before the corresponding database changes have been committed. - Allow overriding the
delayandetawhen raising aRetryTaskexception. This provides finer-grained control over when a task should be retried. - Add a very simple
ResultGroup.as_completed()helper to provide a way to deal with multiple results as they become available. Refs #746. - Add an
asynciohelper for resolving task results asynchronously. Asyncio users can useawait aget_result(result)orawait aget_result_group(rg)to fetch a task result in non-blocking fashion. - Fix bug in SIGINT and SIGTERM behavior for gevent users.
- Include lock name when a task fails due to
TaskLockedexception (#757).
Improves propagation of errors in task results and includes fix for newer versions of pip.
- Add
is_locked(lock_name)to test whether lock is held. - Allow raising
CancelExecutionwithin a Task, and overrideretries. - Add a very basic redis-backed lock that can be acquired more than once (to provide a rudimentary semaphore).
- Add a
periodic_task()wrapper forMiniHueyclass.
- Fix compatibility with redis-py 4.0.0+.
- Fix implementation of schedule-pop Lua script so it works with Redis cluster.
- Ensure Django connections are closed before and after (previously they were
only closed after) task execution with
db_task()anddb_periodic_task(). - Allow additional lock-names to be specified when flushing locks.
- Attempt to reconnect to database if connection becomes unusable (e.g. due to
a server restart). See:
huey.contrib.sql_huey.SqlHuey. - Do not use a soft file-lock for
FileStorage- usefcntl.flock()instead.
- Task expiration: https://huey.readthedocs.io/en/latest/guide.html#task-expiration
- Add option to make
crontab()parsing strict, raising an error if an invalid interval specification is given. You probably want to enable this. - Fix bug in the FileStorage dequeue() method, which attempted to unlink an open file.
- Add hook (
Huey.build_error_result) for customizing the error result metadata. - Avoid crashing if another module already modified/set the multiprocessing start method.
- Add
SIGNAL_INTERRUPTEDto signal when a task is interrupted when a consumer exits abruptly. - Use the
Huey.create_consumer()API within the Django management command, to allow Django users to customize the creation of theConsumerinstance.
- Use monotonic clock for timing operations within the consumer.
- Ensure internal state is cleaned up on file-lock when released.
- Support passing around TaskException as a pickled value.
- Set the multiprocessing mode to "fork" on MacOS and Python 3.8 or newer.
- Added option to enforce FIFO behavior when using Sqlite as storage.
- Added the
on_shutdownhandler to djhuey namespace. - Ensure exception is set on AsyncResult in mini-huey.
- Fix task
repr(refs #460). - Adds task-id into metadata for task exceptions (refs #461).
- Ensure database connection is not closed when using the
call_localmethod of Django helper extensiondb_periodic_task(). - Allow pickle protocol to be explicitly configured in serializer parameters.
- Adds
FileHueyand fullFileStorageimplementation. - Add
shutdown()hook, which will be run in the context of the worker threads/processes during shutdown. This hook can be used to clean-up shared or global resources, for example. - Allow pipelines to be chained together. Additionally, support chaining task instances.
- Fix semantics of
SIGNAL_COMPLETEso that it is not sent until the result is ready. - Use classes for the specific Huey implementations (e.g.
RedisHuey) so that it is easier to subclass / extend. Previously we just used a partial application of the constructor, which could be confusing. - Fix shutdown logic in consumer when using multiprocess worker model. Previously the consumer would perform a "graceful" shutdown, even when an immediate shutdown was requested (SIGTERM). Also cleans up the signal-handling code and ensures that interrupted tasks log a warning properly to indicate they were interrupted.
- Allow
AsyncResultobject used inMiniHueyto support the__call__()method to block and resolve the task result. - When running the django
run_hueymanagement command, the huey loggers will not be configured if another logging handler is already registered to the huey namespace. - Added experimental contrib storage engine using
kyoto tycoon <http://fallabs.com/kyototycoon>_ which supports task priority and the option to do automatic result expiration. Requires theukt <https://github.com/coleifer/ukt>_ python package and a custom kyototycoon lua script. - Allow the Sqlite storage engine busy timeout to be configured when
instantiating
SqliteHuey.
- Ensure that
task()-decorated functions retain their docstrings. - Fix logger setup so that the consumer log configuration is only applied to
the
hueynamespace, rather than the root logger. - Expose
result,signalanddisconnect_signalin the Django huey extension. - Add
SignedSerializer, which signs and validates task messages. - Refactor the
SqliteStorageso that it can be more easily extended to support other databases.
- Added new contrib module
sql_huey, which usespeewee <https://github.com/coleifer/peewee>_ to provide storage layer using any of the supported databases (sqlite, mysql or postgresql). - Added
RedisExpireHuey, which modifies the usual Redis result storage logic to use an expire time for task result values. A consequence of this is that this storage implementation must keep all result keys at the top-level Redis keyspace. There are some small changes to the storage APIs as well, but will only possibly affect maintainers of alternative storage layers. - Also added a
PriorityRedisExpireHueywhich combines the priority-queue support fromPriorityRedisHueywith the result-store expiration mechanism ofRedisExpireHuey. - Fix gzip compatibility issue when using Python 2.x.
- Add option to
Hueyto usezlibas the compression method instead of gzip. - Added
FileStorageMethodsstorage mixin, which uses the filesystem for task result-store APIs (put, peek, pop). - The storage-specific
Hueyimplementations (e.g.RedisHuey) are no longer subclasses, but instead are partial applications of theHueyconstructor.
- Small fixes, fixed typo in Exception class being caught by scheduler.
This section describes the changes in the 2.0.0 release. A detailed list of changes can be found here: https://huey.readthedocs.io/en/latest/changes.html
Overview of changes:
always_eagermode has been renamed toimmediatemode. Unlike previous versions,immediatemode involves the same code paths used by the consumer process. This makes it easier to test features like task revocation and task scheduling without needing to run a dedicated consumer process. Immediate mode uses an in-memory storage layer by default, but can be configured to use "live" storage like Redis or Sqlite.- The events stream API has been removed in favor of simpler callback-driven signals APIs. These callbacks are executed synchronously within the huey consumer process.
- A new serialization format is used in 2.0.0, however consumers running 2.0 will continue to be able to read and deserialize messages enqueued by Huey version 1.11.0 for backwards compatibility.
- Support for task priorities.
- New
Serializerabstraction allows users to customize the serialization format used when reading and writing tasks. - Huey consumer and scheduler can be more easily run within the application process, if you prefer not to run a separate consumer process.
- Tasks can now specify an
on_errorhandler, in addition to the previously-supportedon_completehandler. - Task pipelines return a special
ResultGroupobject which simplifies reading the results of a sequence of task executions. SqliteHueyhas been promoted out ofcontrib, onto an equal footing withRedisHuey. To simplify deployment, the dependency on peewee was removed and the Sqlite storage engine uses the Pythonsqlite3driver directly.
Backwards-incompatible changes
Previously, it was possible for certain tasks to be silently ignored if a task with that name already existed in the registry. To fix this, I have made two changes:
- The task-name, when serialized, now consists of the task module and the name of the decorated function. So, "queue_task_foo" becomes "myapp.tasks.foo".
- An exception will be raised when attempting to register a task function with the same module + name.
Together, these changes are intended to fix problems described in #386.
Because these changes will impact the serialization (and deserialization) of messages, it is important that you consume all tasks (including scheduled tasks) before upgrading.
Always-eager mode changes
In order to provide a more consistent API, tasks enqueued using always_eager
mode will now return a dummy TaskResultWrapper implementation that wraps the
return value of the task. This change is designed to provide the same API for
reading task result values, regardless of whether you are using always-eager
mode or not.
Previously, tasks executed with always_eager would return the Python value
directly from the task. When using Huey with the consumer, though, task results
are not available immediately, so a special wrapper TaskResultWrapper is
returned, which provides helper methods for retrieving the return value of the
task. Going forward, always_eager tasks will return EagerTaskResultWrapper,
which implements the same get() API that is typically used to retrieve task
return values.
- Compatibility with redis-py 3.0, updated requirements / dependencies.
- Add pre-/post- hooks into the djhuey namespace.
- Log time taken to execute tasks at default log level.
- Fix missing import in SQLite storage backend.
- Small refactoring in Redis storage backend to make it easier to override the driver / client implementation.
- Fix failing tests for simpledb storage backend.
- Fixed regression where in always eager mode exceptions within tasks were being swallowed instead of raised.
- Added an API for registering hooks to run when each worker process starts-up. This simplifies creating global/process-wide shared resources, such as a connection pool or database client. Documentation.
- More granular "extras" installation options.
- Remove call to SimpleDB Client.connect(), as the
simpledbAPIs have changed and no longer use this method. - Ensure that pre- and post-execute hooks are run when using Huey in "always_eager" mode.
- Gracefully stop Huey consumer when SIGINT is received.
- Improved continuous integration, now testing on Python 3.7 as well.
- Ensure that the default SIGINT handler is registered. This fixes an edge-case that arises when the consumer is run without job control, which causes interrupt signals to be ignored.
- Restarts (SIGHUP) are now graceful by default.
- Ensure the scheduler loop does not drift (fixes #304).
- Add
TaskResultWrapper.reset()to enable resetting the results of tasks that failed and are subsequently being retried. - Allow task-decorated functions to be also decorated as periodic tasks.
Due to problems with the django patch that added support for multiple huey instances, I've decided to rollback those changes.
Django integration in Huey 1.9.0 will work the same as it had previously in 1.7.x and earlier.
Apologies, I should have reviewed the patch more thoroughly and insisted on better test coverage.
NOTE: These changes were remove in 1.9.0
In 1.8.0, support for multiple huey instances was added (with thanks to @Sebubu and @MarcoGlauser for the patches). Although existing Django/Huey apps should continue to work, there is a new configuration format available and I'd recommend that you take a look at the docs and switch over to it:
Django integration documentation
Previous versions of huey would store the traceback and associated metadata for
a failed task within the result_store, regardless of whether store_errors
was true or not. As of 1.7.0, task exceptions will only be stored in the result
store if store_errors is True. See #290 for discussion.
- Add backwards-compatibility to queue serialization protocol so that 1.6 consumers can continue to work with tasks enqueued by huey versions 1.5 and lower.
- Support for task pipelining and task function partials (which is not compatible with 1.5's task serialization format see note below).
- Support for triggering task retries using
RetryTaskexception. - Support for task locking, restricting concurrency of a given task.
- Getting result of task that failed with an exception results in a
TaskExceptionbeing raised. - Updated health check to ensure the task scheduler is always running.
- Refactor implementation of
task()andperiodic_task()decorators, which should have the added benefit of making them easier to extend. - Refactored result-store APIs to simplify serialization / deserialization logic.
- Fixed bug in serialization of task exceptions.
- Added simple client/server implementation for testing locally. Blog post on the subject.
In v1.6.0, the serialization format of tasks has changed to accomodate an extra piece of metadata. As a result, tasks enqueued with huey versions previous to 1.6 will not be able to be consumed by the 1.6 consumer.
At present there is a workaround available in 1.6.1, but it will be removed when 1.7.0 is released later.
- Allow arbitrary settings to be specified in
task()decorators. - New task name format includes function module as part of task name.
- Fix for operating systems that do not implement SIGHUP.
- Fix bug in
contrib.minimaltask scheduler timing.
- Implemented pre-execute and post-execute hooks.
- Implemented task cancellation mechanism as part of pre-execute hooks.
- Implemented atomic "set if not exists" for Redis and SQLite, which is used by the locking APIs.
- Includes addition of
TaskLockandHuey.lock_task()helpers. - Extend
HueyAPI to add method for creating the consumer.
- Added support for gracefully restarting the consumer using SIGHUP.
- Fixed a bug where periodic tasks were not being given unique task IDs when executed by the consumer. Periodic tasks now receive a unique ID each time they are invoked.
Added support for specifying a retry and retry_delay on periodic tasks.
Simply pass the desired values into the periodic_task() decorator after the
validation function, as keyword arguments.
- Allow all instances of a task to be revoked/restored by adding the
revoke(),restore()andis_revoked()methods to all decorated tasks (where previously they were only available on periodic tasks). - Periodic task instances now have a unique identifier.
- Added documentation on how to correctly use the Django consumer management
command with the
geventworker model. - Logging will lazily resolve log messages.
- Bug was fixed that prevented the local (non-global) task registry from working as intended. This is now fixed.
- Docstrings added to the
BaseStorageAPIs.
Thanks to @mindojo-victor and @nachtmaar for help with some of the above items.
- Support using
7to represent Sunday when doing day-of-week calculations in thecrontabhelper. - Fix bug #243, wherein Django interpreted boolean CLI arguments as having a boolean default value.
Fixed a subtle bug in the way Huey calculated when to run the periodic task scheduler. If you had configured the consumer to check the schedule at an interval that was not a factor of 60, then there is a chance that periodic tasks may be scheduled at incorrect intervals from one minute to the next. This is fixed in 1.4.0.
Added better signal handling in order to support graceful shutdown. Graceful
shutdown involves letting workers finish executing any tasks they may be
processing at the time the shutdown signal is received. The default behavior is
to interrupt the workers mid-task. Huey uses SIGTERM to shutdown the
consumer immediately, and SIGINT to gracefully shutdown.
Added support for using either a global task registry, or a registry bound to
a particular Huey instance. The default behavior is to use a global registry
(backwards-compatible). To bind the registry to a single Huey instance, pass
global_registry=False when initializing your Huey object.
Added a reschedule() method to the TaskResultWrapper.
Documentation clean-ups and additions, particularly around the logic used to handle datetime conversion. Also added docs on shutdown modes for huey consumer.
Smarter conversion between datetimes, so that huey will correctly interpret naive or timezone-aware datetimes and properly convert to UTC when configured to do so. Previously, huey only operated on naive datetimes. Many thanks to @Antoine for this patch-set.
Documentation clean-ups and additions.
Adds flag to preserve result-store value in certain circumstances. Contains yet more hacking at the consumer configuration options, specifically hard-coded defaults are removed from the option definitions.
The run_huey management command was simplified as we are dropping support for
older (officially unsupported) versions of Django.
Added a sqlitedb contrib module that uses a local SQLite database instead of
Redis for Queue persistence, task scheduling and result-storage.
Contains an attempt at fixing the django management command handling of the
default option.
Contains small bugfix for an earlier bugfix meant to prevent time.sleep() from being called with a negative time interval.
Removed the metadata APIs added in 1.1.0, as they seemed poorly-designed and altogether a decent idea terribly implemented. Perhaps something I'll revisit, but which should be easy to implement as a third-party library using the events APIs.
AsyncDatais renamed toTaskResultWrapper.Huey.result()is a new method that provides the result of a task, given a task ID.- Fixed a handful of bugs related to the error serialization.
- Change the default consumer log handler from RotatingFileHandler to the vanilla FileHandler class.
I've added a new API for fetching a task's result given on the task's ID. You
can now call huey.result(task_id) and retrieve the result if the task has
finished executing. Additionally, the Huey.result
method accepts the same parameters as AsyncData.get,
allowing you to block for results, specify a timeout, etc.
There is also a new parameter on the above methods, preserve=False. By
default, the result store will delete a task result once it has been read. Specifying
preserve=True ensures the data is not removed.
This is a small release with a couple minor bugfixes.
- Fixed task metadata serialization bug. #140
- Small cleanup to event iterator storage implementation.
- Updated getting started documentation to reflect changes in the 1.x APIs.
- Big changes to simplify the way
Hueyis instantiated. No changes should be necessary if already usingRedisHuey. - Refactored the storage APIs and simplified the public interface. There is now a single object, whereas before there were 4 components (queue, result store, scheduler and event emitter).
- Added methods for retrieving and introspecting the pending task queue, the schedule, results, and errors.
- Errors can now be stored, in addition to regular task results.
- Added metadata methods for tracking task execution, errors, task duration, and more. These will be the building blocks for tools to provide some insight into the inner-workings of your consumers and producers.
- Many new events are emitted by the consumer, and some have parameters. These are documented here.
What follows is a description of the changes between 0.4.9 and 1.0.0. There are some backwards-incompatible changes to be aware of as well as new options for the consumer. Most APIs are the same, however.
Backwards incompatible changes:
huey.djhueymoved tohuey.contrib.djhuey. You will need to update any import statements as well as your DjangoINSTALLED_APPSsetting to reflect the new module path.- Redis backend is now the only one available, and the corresponding code moved
from
huey.backends.redis_backendtohuey.storage. - Removed the "RabbitMQ" and "SQLite" queue backends.
- Removed the
-tand--threadsoption from the consumer. You should now use-wor--workers. - Removed the
-pand--periodicno-op options from the consumer. These are enabled by default so the option had no meaning. - The
scheduler-intervaloption is configured using-swhen previously it was-S. Furthermore, this must be a value between 1 and 60. - Removed the
peewee_helpersmodule.
New features:
- The queue consumer now supports multi-process or multi-greenlet execution models (in addition to multi-threaded, which previously was the only option).
- Added
pending(),scheduled()andall_results()methods to theHueyclass to allow introspection of the Queue's state at the current moment in time.