Prefect automation not being triggered

[samlawler-entain]

Bug summary

I have a basic automation that has a trigger of:
When any flow run stays in Suspended, Paused, Pending, Running, Retrying for 10 hours
The action changes the flow run state to be Failed - likely irrelevant to the issue.

When this automation was first made (~4 months ago), this had worked a few times. However since then I had discovered that it had not been triggering despite having flow runs that met the conditions of the automation.

I have attached 3 screenshots (in order):

• The automation trigger settings
• An example of a flow run that was in a running state for almost 2 days which should have been triggered the automation. I had manually cancelled this.
• The events of that automation covering the timeframe of the mentioned example. Showing that the automation was not triggered during that timeframe.

Automation Id: 1fa3621d-0371-4e19-862a-dbf9969cce94
Example flow run id: 49beefa3-5e14-4baa-a5ad-1242ddce9724

Therefore, the issue being raised is the automation is not being triggered, despite having multiple flow runs that would have triggered this automation if working correctly.

It should also be noted that i have other automations running, which execute as expected. It's only this automation with these conditions that i am experiencing this issue with.

Version info

Version:             2.15.0
API version:         0.8.4
Python version:      3.11.9
Git commit:          e90804b8
Built:               Thu, Feb 15, 2024 2:54 PM
OS/Arch:             win32/AMD64
Profile:             default
Server type:         cloud

Additional context

No response

[jeanluciano]

Hi @samlawler-entain, reproduced the issue. Looking further into it.

[jeanluciano]

Okay after further looking into it, it seems like UI could be misleading here. It creates an event triggers that would trigger if the events happen in the within timeframe. In this case:

• PENDING event is seen, we open a 10 hour bucket
• RUNNING event is seen soon after, it counts towards the 10 hour bucket (because both are defined in after)
• We now close the bucket cause we've seen another event (when we're looking to NOT see another event to detect a stuck run)

If a user needs an automation to trigger for any state, what is needed is a composite_trigger. i.e:

{
    "type": "compound",
    "require": "any",
    "triggers": [
        {
            "type": "event",
            "after": ["prefect.flow-run.Running"],  # Only start watching after RUNNING
            "expect": ["prefect.flow-run.*"],       # Watch for ANY state change
            "within": 36000.0.0,                    # Look for 10 hours
            "posture": "Proactive",
            "threshold": 1,                         # Fire if no events seen
            "for_each": ["prefect.resource.id"],
            "match": {
                "prefect.resource.id": "prefect.flow-run.*"
            }
        },
        {
            "type": "event", 
            "after": ["prefect.flow-run.Pending"],  # Only start watching after PENDING
            "expect": ["prefect.flow-run.*"],       # Watch for ANY state change
            "within": 36000.0.0,                    # Look for 10 hours
            "posture": "Proactive",
            "threshold": 1,                         # Fire if no events seen
            "for_each": ["prefect.resource.id"],
            "match": {
                "prefect.resource.id": "prefect.flow-run.*"
            }
        },
  {
            "type": "event", 
            "after": ["prefect.flow-run.Suspended"],  # Only start watching after SUSPENDED
            "expect": ["prefect.flow-run.*"],       # Watch for ANY state change
            "within": 36000.0.0,                    # Look for 10 hours
            "posture": "Proactive",
            "threshold": 1,                         # Fire if no events seen
            "for_each": ["prefect.resource.id"],
            "match": {
                "prefect.resource.id": "prefect.flow-run.*"
            }
        },
  {
            "type": "event", 
            "after": ["prefect.flow-run.Retrying"],  # Only start watching after RETRYING
            "expect": ["prefect.flow-run.*"],       # Watch for ANY state change
            "within": 36000.0.0,                    # Look for 10 hours
            "posture": "Proactive",
            "threshold": 1,                         # Fire if no events seen
            "for_each": ["prefect.resource.id"],
            "match": {
                "prefect.resource.id": "prefect.flow-run.*"
            }
        },

    ]
}