Merge #153277

153277: roachtest: indefinite block on task events r=srosenberg a=herkolategan

Previously, we could end up with dangling goroutines for tasks if some tasks were trying to send events, but the consumer has already stopped consuming events. This would cause an indefinite block in the task goroutine when trying to send the event.

This change adds a `select` when trying to send the events, but if the context has been cancelled, it will break out. There's a small chance that the event could end up being sent even if the context has been canceled, but on the consumer side we also check for a canceled context.

Epic: None
Release note: None

Co-authored-by: Herko Lategan <herko@cockroachlabs.com>

Author: craig[bot]

pkg/cmd/roachtest/roachtestutil/task/manager.go

@@ -229,7 +229,18 @@ func (t *group) GoWithCancel(fn Func, opts ...Option) context.CancelFunc {
 			return nil
 		}
 		if !opt.DisableReporting {
-			t.manager.events <- event
+			// There's a slight chance that the context is canceled between the check
+			// above and this select. In which case Go might probabilistically select
+			// sending the event. This should not cause issues as the consumer of the
+			// events should also check if the parent context is canceled. But we
+			// require the select here to avoid blocking if the parent context is
+			// canceled, and the consumer is no longer consuming events.
+			select {
+			case t.manager.events <- event:
+			case <-t.manager.ctx.Done():
+				//  do not send the event if the parent context is canceled
+				return nil
+			}
 		}
 		return err
 	})