Skip to content

Timer keeps running after workflow cancellation raising an error #3156

@jahzzeel

Description

@jahzzeel

Describe the bug
Consider this:

@workflow.run
    async def run():
        try:
                status = await workflow.execute_activity(
                start_activity,
                args=[request_id, team_id, template_id],
                start_to_close_timeout=timedelta(seconds=10),
                retry_policy=retry_policy,
            )

            while True:
                 if status == X:
                     break
                 await workflow.sleep(30)
        except asyncio.CancelledError:
             status = await workflow.execute_activity(
                cleanup_activity,
                args=[request_id, team_id, template_id],
                start_to_close_timeout=timedelta(seconds=10),
                retry_policy=retry_policy,
            )
            if status == X:
                     raise
            await workflow.sleep(30)

The problem I'm dealing with is that when doing await client.get_workflow_handle("your_workflow_id").cancel()
even though the except code is running fine, the first sleep in the try block keeps running and finally showing an error, even though this error do not interrupt the normal flow, in the Temporal UI the clock keeps ticking until it finishes, causing it to display incorrect information since it should have been stoped on cancellation.

Note: I have also tried asyncio.shield() but the error remains

To Reproduce
Steps to reproduce the behavior:

  1. do await client.get_workflow_handle("your_workflow_id").cancel()
  2. have a sleep in the try block and another in the except asyncio.CancelledError block, the second sleep should have enough time to allow the first one to finish and show the error

Part of the Error:
ERROR temporalio.worker._workflow_instance Exception in callback _WorkflowInstanceImpl.workflow_sleep..() at
temporal/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 1727, in
lambda: fut.set_result(None),
^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.InvalidStateError: invalid state

Expected behavior
every await task should stop in the try block including the sleeps when the workflow is cancelled

Screenshots

Image

as seen in the image the first sleep kept running for about 10 more seconds after the cancellation request
Desktop (please complete the following information):

  • OS: ubuntu 23
  • Browser chrome
  • Version 2.39.0

Additional context
Add any other context about the problem here.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions