Sentry: fetcher.go:1353: executing declarative schema change PostCommitPhase stage 2 of 7 with 1 BackfillType op (rollback=false) for CREATE INDEX: failed to construct index entries during backfill: N...

[cockroach-sentry]

This issue was auto filed by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry Link: https://cockroach-labs.sentry.io/issues/7006920505/?referrer=webhooks_plugin

Panic Message:

fetcher.go:1353: executing declarative schema change PostCommitPhase stage 2 of 7 with 1 BackfillType op (rollback=false) for CREATE INDEX: failed to construct index entries during backfill: Non-nullable column "×:×" with no value! Index scanned was × with the index key columns (×) and the values (×)
(1) forced error mark
  | ×
  | github.com/cockroachdb/errors/withstack/*withstack.withStack::
Wraps: (2)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scerrors.EventLogger.HandlePanicAndLogError
  | 	pkg/sql/schemachanger/scerrors/errors.go:82
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scrun.executeStage
  | 	pkg/sql/schemachanger/scrun/scrun.go:211
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scrun.RunSchemaChangesInJob.func1
  | 	pkg/sql/schemachanger/scrun/scrun.go:140
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scdeps.(*jobExecutionDeps).WithTxnInJob.func1
  | 	pkg/sql/schemachanger/scdeps/run_deps.go:146
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).DescsTxn.func1
  | 	pkg/sql/internal.go:1944
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn.func4
  | 	pkg/sql/internal.go:2041
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
  | 	pkg/kv/txn.go:1132
  | github.com/cockroachdb/cockroach/pkg/kv.runTxn
  | 	pkg/kv/db.go:1075
  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
  | 	pkg/kv/db.go:1038
  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).Txn
  | 	pkg/kv/db.go:1013
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn
  | 	pkg/sql/internal.go:2028
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).DescsTxn
  | 	pkg/sql/internal.go:1942
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scdeps.(*jobExecutionDeps).WithTxnInJob
  | 	pkg/sql/schemachanger/scdeps/run_deps.go:111
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scrun.RunSchemaChangesInJob
  | 	pkg/sql/schemachanger/scrun/scrun.go:136
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scjob.(*newSchemaChangeResumer).run
  | 	pkg/sql/schemachanger/scjob/job.go:145
  | github.com/cockroachdb/cockroach/pkg/sql/schemachanger/scjob.(*newSchemaChangeResumer).Resume
  | 	pkg/sql/schemachanger/scjob/job.go:57
  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine.func2
  | 	pkg/jobs/registry.go:1720
  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).stepThroughStateMachine
  | 	pkg/jobs/registry.go:1721
  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).runJob
  | 	pkg/jobs/adopt.go:410
  | github.com/cockroachdb/cockroach/pkg/jobs.(*Registry).resumeJob.func1
  | 	pkg/jobs/adopt.go:263
  | github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2
  | 	pkg/util/stop/stopper.go:500
  | runtime.goexit
  | 	src/runtime/asm_amd64.s:1700
Wraps: (3) executing declarative schema change PostCommitPhase stage 2 of 7 with 1 BackfillType op (rollback=false) for CREATE INDEX
Wraps: (4)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).runBackfill.func1
  | 	pkg/sql/rowexec/indexbackfiller.go:427
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).runBackfill.Group.GoCtx.func3
  | 	pkg/util/ctxgroup/ctxgroup.go:199
  | golang.org/x/sync/errgroup.(*Group).Go.func1
  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
  | runtime.goexit
  | 	src/runtime/asm_amd64.s:1700
Wraps: (5) failed to construct index entries during backfill
Wraps: (6) assertion failure
Wraps: (7)
  | (opaque error wrapper)
  | type name: github.com/cockroachdb/errors/withstack/*withstack.withStack
  | reportable 0:
  |
  | github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).finalizeRow
  | 	pkg/sql/row/fetcher.go:1353
  | github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).NextRow
  | 	pkg/sql/row/fetcher.go:1187
  | github.com/cockroachdb/cockroach/pkg/sql/row.(*Fetcher).NextRowDecodedInto
  | 	pkg/sql/row/fetcher.go:1258
  | github.com/cockroachdb/cockroach/pkg/sql/backfill.(*IndexBackfiller).BuildIndexEntriesChunk
  | 	pkg/sql/backfill/backfill.go:1078
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).buildIndexEntryBatch.func2
  | 	pkg/sql/rowexec/indexbackfiller.go:566
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).Txn.func1
  | 	pkg/sql/internal.go:1954
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn.func4
  | 	pkg/sql/internal.go:2041
  | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).exec
  | 	pkg/kv/txn.go:1132
  | github.com/cockroachdb/cockroach/pkg/kv.runTxn
  | 	pkg/kv/db.go:1075
  | github.com/cockroachdb/cockroach/pkg/kv.(*DB).TxnWithAdmissionControl
  | 	pkg/kv/db.go:1038
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn.func2
  | 	pkg/sql/internal.go:2014
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).txn
  | 	pkg/sql/internal.go:2028
  | github.com/cockroachdb/cockroach/pkg/sql.(*InternalDB).Txn
  | 	pkg/sql/internal.go:1955
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBatchRetry).buildBatchWithRetry
  | 	pkg/sql/rowexec/indexbackfiller.go:607
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).buildIndexEntryBatch
  | 	pkg/sql/rowexec/indexbackfiller.go:575
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).constructIndexEntries
  | 	pkg/sql/rowexec/indexbackfiller.go:148
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).runBackfill.func1
  | 	pkg/sql/rowexec/indexbackfiller.go:425
  | github.com/cockroachdb/cockroach/pkg/sql/rowexec.(*indexBackfiller).runBackfill.Group.GoCtx.func3
  | 	pkg/util/ctxgroup/ctxgroup.go:199
  | golang.org/x/sync/errgroup.(*Group).Go.func1
  | 	external/org_golang_x_sync/errgroup/errgroup.go:78
  | runtime.goexit
  | 	src/runtime/asm_amd64.s:1700
Wraps: (8) Non-nullable column "×:×" with no value! Index scanned was × with the index key columns (×) and the values (×)
Error types: (1) *markers.withMark (2) *errbase.opaqueWrapper (3) *errutil.withPrefix (4) *errbase.opaqueWrapper (5) *errutil.withPrefix (6) *assert.withAssertionFailure (7) *errbase.opaqueWrapper (8) *errutil.leafError
-- report composition:
*errutil.leafError: Non-nullable column "×:×" with no value! Index scanned was × with the index key columns (×) and the values (×)
fetcher.go:1353: *withstack.withStack (top exception)
*assert.withAssertionFailure
*errutil.withPrefix: failed to construct index entries during backfill
indexbackfiller.go:427: *withstack.withStack (1)
*errutil.withPrefix: executing declarative schema change PostCommitPhase stage 2 of 7 with 1 BackfillType op (rollback=false) for CREATE INDEX
errors.go:82: *withstack.withStack (2)
*markers.withMark
(check the extra data payloads)

Stacktrace (expand for inline code snippets):

src/runtime/asm_amd64.s#L1699-L1701
pkg/util/stop/stopper.go#L499-L501
pkg/jobs/adopt.go#L262-L264
pkg/jobs/adopt.go#L409-L411
pkg/jobs/registry.go#L1720-L1722
pkg/jobs/registry.go#L1719-L1721
pkg/sql/schemachanger/scjob/job.go#L56-L58
pkg/sql/schemachanger/scjob/job.go#L144-L146
pkg/sql/schemachanger/scrun/scrun.go#L135-L137
pkg/sql/schemachanger/scdeps/run_deps.go#L110-L112
pkg/sql/internal.go#L1941-L1943
pkg/sql/internal.go#L2027-L2029
pkg/kv/db.go#L1012-L1014
pkg/kv/db.go#L1037-L1039
pkg/kv/db.go#L1074-L1076
pkg/kv/txn.go#L1131-L1133
pkg/sql/internal.go#L2040-L2042
pkg/sql/internal.go#L1943-L1945
pkg/sql/schemachanger/scdeps/run_deps.go#L145-L147
pkg/sql/schemachanger/scrun/scrun.go#L139-L141
pkg/sql/schemachanger/scrun/scrun.go#L210-L212
pkg/sql/schemachanger/scerrors/errors.go#L81-L83
src/runtime/asm_amd64.s#L1699-L1701
external/org_golang_x_sync/errgroup/errgroup.go#L77-L79
pkg/util/ctxgroup/ctxgroup.go#L198-L200
pkg/sql/rowexec/indexbackfiller.go#L426-L428
src/runtime/asm_amd64.s#L1699-L1701
external/org_golang_x_sync/errgroup/errgroup.go#L77-L79
pkg/util/ctxgroup/ctxgroup.go#L198-L200
pkg/sql/rowexec/indexbackfiller.go#L424-L426
pkg/sql/rowexec/indexbackfiller.go#L147-L149
pkg/sql/rowexec/indexbackfiller.go#L574-L576
pkg/sql/rowexec/indexbackfiller.go#L606-L608
pkg/sql/internal.go#L1954-L1956
pkg/sql/internal.go#L2027-L2029
pkg/sql/internal.go#L2013-L2015
pkg/kv/db.go#L1037-L1039
pkg/kv/db.go#L1074-L1076
pkg/kv/txn.go#L1131-L1133
pkg/sql/internal.go#L2040-L2042
pkg/sql/internal.go#L1953-L1955
pkg/sql/rowexec/indexbackfiller.go#L565-L567
pkg/sql/backfill/backfill.go#L1077-L1079
pkg/sql/row/fetcher.go#L1257-L1259
pkg/sql/row/fetcher.go#L1186-L1188
pkg/sql/row/fetcher.go#L1352-L1354

src/runtime/asm_amd64.s in runtime.goexit at line 1700
pkg/util/stop/stopper.go in pkg/util/stop.(*Stopper).RunAsyncTaskEx.func2 at line 500
pkg/jobs/adopt.go in pkg/jobs.(*Registry).resumeJob.func1 at line 263
pkg/jobs/adopt.go in pkg/jobs.(*Registry).runJob at line 410
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine at line 1721
pkg/jobs/registry.go in pkg/jobs.(*Registry).stepThroughStateMachine.func2 at line 1720
pkg/sql/schemachanger/scjob/job.go in pkg/sql/schemachanger/scjob.(*newSchemaChangeResumer).Resume at line 57
pkg/sql/schemachanger/scjob/job.go in pkg/sql/schemachanger/scjob.(*newSchemaChangeResumer).run at line 145
pkg/sql/schemachanger/scrun/scrun.go in pkg/sql/schemachanger/scrun.RunSchemaChangesInJob at line 136
pkg/sql/schemachanger/scdeps/run_deps.go in pkg/sql/schemachanger/scdeps.(*jobExecutionDeps).WithTxnInJob at line 111
pkg/sql/internal.go in pkg/sql.(*InternalDB).DescsTxn at line 1942
pkg/sql/internal.go in pkg/sql.(*InternalDB).txn at line 2028
pkg/kv/db.go in pkg/kv.(*DB).Txn at line 1013
pkg/kv/db.go in pkg/kv.(*DB).TxnWithAdmissionControl at line 1038
pkg/kv/db.go in pkg/kv.runTxn at line 1075
pkg/kv/txn.go in pkg/kv.(*Txn).exec at line 1132
pkg/sql/internal.go in pkg/sql.(*InternalDB).txn.func4 at line 2041
pkg/sql/internal.go in pkg/sql.(*InternalDB).DescsTxn.func1 at line 1944
pkg/sql/schemachanger/scdeps/run_deps.go in pkg/sql/schemachanger/scdeps.(*jobExecutionDeps).WithTxnInJob.func1 at line 146
pkg/sql/schemachanger/scrun/scrun.go in pkg/sql/schemachanger/scrun.RunSchemaChangesInJob.func1 at line 140
pkg/sql/schemachanger/scrun/scrun.go in pkg/sql/schemachanger/scrun.executeStage at line 211
pkg/sql/schemachanger/scerrors/errors.go in pkg/sql/schemachanger/scerrors.EventLogger.HandlePanicAndLogError at line 82
src/runtime/asm_amd64.s in runtime.goexit at line 1700
external/org_golang_x_sync/errgroup/errgroup.go in golang.org/x/sync/errgroup.(*Group).Go.func1 at line 78
pkg/util/ctxgroup/ctxgroup.go in pkg/sql/rowexec.(*indexBackfiller).runBackfill.Group.GoCtx.func3 at line 199
pkg/sql/rowexec/indexbackfiller.go in pkg/sql/rowexec.(*indexBackfiller).runBackfill.func1 at line 427
src/runtime/asm_amd64.s in runtime.goexit at line 1700
external/org_golang_x_sync/errgroup/errgroup.go in golang.org/x/sync/errgroup.(*Group).Go.func1 at line 78
pkg/util/ctxgroup/ctxgroup.go in pkg/sql/rowexec.(*indexBackfiller).runBackfill.Group.GoCtx.func3 at line 199
pkg/sql/rowexec/indexbackfiller.go in pkg/sql/rowexec.(*indexBackfiller).runBackfill.func1 at line 425
pkg/sql/rowexec/indexbackfiller.go in pkg/sql/rowexec.(*indexBackfiller).constructIndexEntries at line 148
pkg/sql/rowexec/indexbackfiller.go in pkg/sql/rowexec.(*indexBackfiller).buildIndexEntryBatch at line 575
pkg/sql/rowexec/indexbackfiller.go in pkg/sql/rowexec.(*indexBatchRetry).buildBatchWithRetry at line 607
pkg/sql/internal.go in pkg/sql.(*InternalDB).Txn at line 1955
pkg/sql/internal.go in pkg/sql.(*InternalDB).txn at line 2028
pkg/sql/internal.go in pkg/sql.(*InternalDB).txn.func2 at line 2014
pkg/kv/db.go in pkg/kv.(*DB).TxnWithAdmissionControl at line 1038
pkg/kv/db.go in pkg/kv.runTxn at line 1075
pkg/kv/txn.go in pkg/kv.(*Txn).exec at line 1132
pkg/sql/internal.go in pkg/sql.(*InternalDB).txn.func4 at line 2041
pkg/sql/internal.go in pkg/sql.(*InternalDB).Txn.func1 at line 1954
pkg/sql/rowexec/indexbackfiller.go in pkg/sql/rowexec.(*indexBackfiller).buildIndexEntryBatch.func2 at line 566
pkg/sql/backfill/backfill.go in pkg/sql/backfill.(*IndexBackfiller).BuildIndexEntriesChunk at line 1078
pkg/sql/row/fetcher.go in pkg/sql/row.(*Fetcher).NextRowDecodedInto at line 1258
pkg/sql/row/fetcher.go in pkg/sql/row.(*Fetcher).NextRow at line 1187
pkg/sql/row/fetcher.go in pkg/sql/row.(*Fetcher).finalizeRow at line 1353

Tags

Tag	Value
Command	mt start-sql
Environment	v25.2.6
Go Version	go1.23.12 X:nocoverageredesign
Platform	linux amd64
Distribution	CCL
Cockroach Release	v25.2.6
Cockroach SHA	e9b11cf
# of CPUs	32
# of Goroutines	1177

Jira issue: CRDB-56305

[blathers-crl]

CC'ing via the CODEOWNERS-based sentry heuristic:

• @cockroachdb/sql-queries

Sentry issue cause: pkg/sql/row/fetcher.go

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

[spilchen]

Not actionable without an escalation.