Fix defusing race between Tenant::shutdown and offload_timeline #10150
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There is a race condition between
Tenant::shutdown
'sdefuse_for_drop
loop andoffload_timeline
, where timeline offloading can insert into a tenant that is in the process of shutting down, in fact so far progressed that thedefuse_for_drop
has already been called.This prevents warn log lines of the form:
The solution piggybacks on the
offloaded_timelines
lock: both the defuse loop and the offloaded timeline insertion need to acquire the lock, and we know that the defuse loop only runs after the tenant has set itsTenantState
toStopping
.So if we hold the
offloaded_timelines
lock, and know that theTenantState
is notStopping
, then we know that the defuse loop has not ran yet, and holding the lock ensures that it doesn't start running while we are inserting the offloaded timeline.Fixes #10070