roachtest: jobs/mixed-versions failed

[cockroach-teamcity]

roachtest.jobs/mixed-versions failed with artifacts on release-25.4 @ 147fd35c9958d933cc5b1dcaf97c50803e35b01b:

(test_monitor.go:35).func1: test monitor: unexpected node event: n3: cockroach process for virtual cluster "mixed-version-tenant-0hxb6", instance 0 died (exit code 1)
(mixedversion.go:891).Run: preparing to run step 41: failed to get cluster version for node 3 (system): context canceled
test artifacts and logs in: /artifacts/jobs/mixed-versions/run_1

Parameters:

• arch=amd64
• cloud=gce
• coverageBuild=false
• cpu=4
• encrypted=false
• fs=ext4
• localSSD=true
• mvtDeploymentMode=separate-process
• mvtVersions=v25.1.10 → v25.2.8 → release-25.4
• runtimeAssertionsBuild=false
• ssd=0

Help

See: roachtest README

See: How To Investigate (internal)

See: Grafana

/cc @cockroachdb/disaster-recovery

This test on roachdash | Improve this report!

Jira issue: CRDB-56318

[msbutler]

* panic: expected panic from panicNodeMutator
* (1) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
*   | 	pkg/sql/conn_executor.go:1032
*   | [...repeated from below...]
* Wraps: (2) while executing: SELECT crdb_internal.force_panic(_)
* Wraps: (3) attached stack trace
*   -- stack trace:
*   | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn.func1
*   | 	pkg/sql/conn_executor.go:1032
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:791
*   | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
*   | 	pkg/sql/colexecerror/error.go:137
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:791
*   | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError.func1
*   | 	pkg/sql/colexecerror/error.go:137
*   | runtime.gopanic
*   | 	GOROOT/src/runtime/panic.go:791
*   | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.NonCatchablePanic
*   | 	pkg/sql/colexecerror/error.go:317
*   | github.com/cockroachdb/cockroach/pkg/sql/sem/builtins.init.func275
*   | 	pkg/sql/sem/builtins/builtins.go:5942
*   | github.com/cockroachdb/cockroach/pkg/sql/sem/eval.(*evaluator).EvalFuncExpr
*   | 	pkg/sql/sem/eval/expr.go:503
*   | github.com/cockroachdb/cockroach/pkg/sql/sem/tree.(*FuncExpr).Eval
*   | 	bazel-out/k8-opt/bin/pkg/sql/sem/tree/eval_expr_generated.go:287
*   | github.com/cockroachdb/cockroach/pkg/sql/sem/eval.Expr
*   | 	pkg/sql/sem/eval/expr.go:21
*   | github.com/cockroachdb/cockroach/pkg/sql.(*valuesNode).startExec
*   | 	pkg/sql/values.go:86
*   | github.com/cockroachdb/cockroach/pkg/sql.startExec
*   | 	pkg/sql/plan.go:578
*   | github.com/cockroachdb/cockroach/pkg/sql.(*planNodeToRowSource).Start
*   | 	pkg/sql/plan_node_to_row_source.go:181
*   | github.com/cockroachdb/cockroach/pkg/sql/colexec.(*Columnarizer).Init
*   | 	pkg/sql/colexec/columnarizer.go:178
*   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).init
*   | 	pkg/sql/colflow/stats.go:96
*   | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
*   | 	pkg/sql/colexecerror/error.go:162
*   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*batchInfoCollector).Init
*   | 	pkg/sql/colflow/stats.go:105
*   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run.(*BatchFlowCoordinator).init.func2
*   | 	pkg/sql/colflow/flow_coordinator.go:236
*   | github.com/cockroachdb/cockroach/pkg/sql/colexecerror.CatchVectorizedRuntimeError
*   | 	pkg/sql/colexecerror/error.go:162
*   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).init
*   | 	pkg/sql/colflow/flow_coordinator.go:235
*   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*BatchFlowCoordinator).Run
*   | 	pkg/sql/colflow/flow_coordinator.go:269
*   | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
*   | 	pkg/sql/colflow/vectorized_flow.go:316
*   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
*   | 	pkg/sql/distsql_running.go:991
*   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
*   | 	pkg/sql/distsql_running.go:2094
*   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll.func3
*   | 	pkg/sql/distsql_running.go:1798
*   | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
*   | 	pkg/sql/distsql_running.go:1801
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
*   | 	pkg/sql/conn_executor_exec.go:3519
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
*   | 	pkg/sql/conn_executor_exec.go:3034
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
*   | 	pkg/sql/conn_executor_exec.go:1087
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func2
*   | 	pkg/sql/conn_executor_exec.go:172
*   | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
*   | 	pkg/sql/conn_executor_exec.go:4534
* Wraps: (4) panic: expected panic from panicNodeMutator
* Error types: (1) *withstack.withStack (2) *safedetails.withSafeDetails (3) *withstack.withStack (4) *errutil.leafError
*
c

[msbutler]

oh wait, this is probably some mixed version test flake. the test driver is panicking nodes. if this begins to flake regularly, i'll look further into this.