Handle system time move backward case in timer task processing

[yycptt]

What changed?

• Handle system time move backward case in timer task processing by using max(now(), task visibility timestamp)

Why?

• Monotonic time is used in time.Time comparison only when both operands have monotonic time value. In our case, the timestamp stored/derived in/from mutable state doesn't have monotonic, thus the comparison logic will use wall clock time to decide if a timer task should be processed. If wall clock move backwards (after we verifies that now() > task visibility timestamp when submitting the task to the task scheduler), the timer task will be dropped and cause workflow to stuck.

How did you test it?

• Unit tests

Potential risks

• In worst case, we will execute a timer task logic earlier than expected.

Documentation

Is hotfix candidate?

• Could be. But the bug should be very rare as the use millisecond precision when doing time comparison in timer task processing.