runtime: fix ticker not stopping when Stop races with its callback (#5487)

* runtime: fix ticker not stopping when Stop races with its callback

On the threads and cores schedulers, timer callbacks run concurrently
with user goroutines.

If Stop (or Reset) was called in the window after the node was popped
but before its callback re-added it, removeTimer would not find the
timer in the queue, so it would be re-added to timer anyway.

Track timers whose callback is currently running in a firing list, and
have removeTimer mark a firing timer as stopped so its callback does not
re-add it.

Also make the timers test drain robust: allow a possible in-flight tick
delivered concurrently with Stop to settle before draining the channel.

Signed-off-by: deadprogram <ron@hybridgroup.com>

* runtime: fix Stop/Reset semantics when racing a firing timer callback

Address review feedback on the ticker Stop-race fix. On the threads and
cores schedulers a timer callback runs concurrently with user goroutines,
which left several problems:

- removeTimer reported a firing timer as successfully removed, so
  Stop/Reset could return true even though the callback had already
  started (wrong semantics, notably for AfterFunc). firingTimerStop now
  returns a bool and removeTimer no longer hands the still-firing node
  back to resetTimer.

- The periodic advance (when += period) ran in timerCallback outside the
  scheduler timer lock. Move it into each scheduler's reAddTimer, under
  the lock and after the stopped check, so a concurrent Reset can't have
  its freshly-queued deadline corrupted.

- resetTimer now sets when/period after removeTimer for the same reason.

Add testdata/timer_stop_reset_race.go and TestTimerStopResetRace, which
reproduce the stop-while-firing and reset-while-firing races via the
runtime timer linkname hooks.

Signed-off-by: deadprogram <ron@hybridgroup.com>

* runtime: fix timer Stop/Reset race with firing periodic timers

Signed-off-by: deadprogram <ron@hybridgroup.com>

---------

Signed-off-by: deadprogram <ron@hybridgroup.com>

src/runtime/time.go

@@ -53,13 +53,13 @@ func stopTimer(tim *timeTimer) bool {
 
 //go:linkname resetTimer time.resetTimer
 func resetTimer(t *timeTimer, when, period int64) bool {
-	t.timer.when = when
-	t.timer.period = period
 	n := removeTimer(&t.timer)
 	removed := n != nil
 	if n == nil {
 		n = new(timerNode)
 	}
+	t.timer.when = when
+	t.timer.period = period
 	n.timer = &t.timer
 	n.callback = timerCallback
 	addTimer(n)
@@ -84,6 +84,19 @@ type timerNode struct {
 	next     *timerNode
 	timer    *timer
 	callback func(node *timerNode, delta int64)
+
+	// The following fields are only used by schedulers that run timer
+	// callbacks concurrently with user goroutines (the threads and cores
+	// schedulers). They make it possible to stop or reset a periodic timer (a
+	// ticker) while its callback is running, without the callback re-adding the
+	// timer to the queue afterwards. They are protected by the scheduler's
+	// timer lock.
+	//
+	// firingNext links nodes whose callback is currently running into the
+	// firingTimers list. stopped is set when the timer was stopped or reset
+	// while its callback was running, so that timerCallback does not re-add it.
+	firingNext *timerNode
+	stopped    bool
 }
 
 // whenTicks returns the (absolute) time when this timer should trigger next.
@@ -108,8 +121,7 @@ func timerCallback(tn *timerNode, delta int64) {
 
 	// If this is a periodic timer (a ticker), re-add it to the queue.
 	if tn.timer.period != 0 {
-		tn.timer.when += tn.timer.period
-		addTimer(tn)
+		reAddTimer(tn)
 	}
 }