resolves #3344: multi phase planning (#3328)

This PR introduces multiphase planning in the `CascadesPlanner`. A
planner phase transitions between planner stages:
```
stages: INITIAL ---------------> CANONICAL ---------------> PLANNED
phases:            REWRITING                   PLANNING
```
                 
The current planner's work is now realized as a planner phase called
`PLANNING` that transitions from planner stage `CANONICAL` to planner
stage `PLANNED`.
`Reference`s maintain the planner stages of the phase that created them
and can be modified by calling `Reference.advancePlannerStage()` which
is currently done during `ExploreGroup`. `CascadesPlanner.Task`s use a
planner phase to resolve the rule set and the cost model that belong to
the phase.

This PR also introduces the `REWRITING` phase which is to be filled out
in subsequent PRs. Currently the ruleset of that phase only contains
`FinalizeExpressionsRule` which is required to make progress and to
allow us to transition to the next planner stage and in turn next
planner stage.

When any phase is initiated in the planner, we will explore the
expression DAG in much the same way as before using the corresponding
planner rule set whose rules then yield new expressions. A rule is
either an `ExplorationRule` or an `ImplementationRule`. When an
`ImplementationRule` yields an expression it is considered a final
expression and is treated exactly the same way as
previously`RecordQueryPlan`s. That means we optimize that final
expressions' inputs using the planner phase's cost model. For
`REWRITING` we use `RewritingCostModel` which consistently picks the DAG
with smallest semantic hash. In that way we construct a DAG bottom up
consisting of the cheapest expression DAG according to
`RewritingCostModel`. The first task to be pushed and the last task in
the task stack will eventually initiate `PlannerPhase.PLANNING` which
then kicks off another round of regular planning.

There are other changes in this PR as well that are necessary in the
same unit of work:
- `References` distinguish between _exploratory_ and _final_ members.
Exploration rules yield and memoize exploratory expressions while
implementation rules yield and memoize final expressions.
- Final expression members are treated in the same way as previously
`RecordQueryPlan`s. An exploratory expression cannot be a
`RecordQueryPlan`
- `Memoizer` has been refactored to be more safe and hopefully easier to
be used. There were a few problems in the code not calling the right
methods for memoization that were fixed.
- All phases explore and implement `RelationalExpression` except the
last `PLANNING` phase which produces `RecordQueryPlan`s. There is a
non-trivial amount of code that makes that somewhat less visible and to
some degree polymorphic.
- most rules now extend `ExplorationCascadesRule` or
`ImplementationCascadesRule` which also determines which yields and
memoization API are available to the rule. Did I mention that I wanted
to make this more safe?
- During `OptimizeGroup.execute()` when we prune the group we previously
would just pick a `RelationalExpression` as the sole survivor if there
were no plans. That logic is changed to only affect final expressions,
i.e. exploratory expressions are kept in the group even after pruning.
If there are no final expressions in the reference before pruning, there
will not be a final expression in the reference after pruning.
- improved Javadocs

Author: normen662

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/metadata/expressions/FieldKeyExpression.java

@@ -221,7 +221,7 @@ public Quantifier.ForEach explodeField(@Nonnull Quantifier.ForEach baseQuantifie
                 .build();
         switch (fanType) {
             case FanOut:
-                return Quantifier.forEach(Reference.of(
+                return Quantifier.forEach(Reference.initialOf(
                         ExplodeExpression.explodeField(baseQuantifier, fieldNames)));
             default:
                 throw new RecordCoreException("unrecognized fan type");

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/provider/foundationdb/FDBDatabaseFactory.java

@@ -340,7 +340,7 @@ public long getInitialDelayMillis() {
      */
     public void setInitialDelayMillis(long initialDelayMillis) {
         if (initialDelayMillis < 0) {
-            throw new RecordCoreException("Cannot set initial delay milleseconds to less than zero");
+            throw new RecordCoreException("Cannot set initial delay milliseconds to less than zero");
         } else if (initialDelayMillis > maxDelayMillis) {
             throw new RecordCoreException("Cannot set initial delay to greater than maximum delay");
         }

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/query/expressions/OneOfThemWithComparison.java

@@ -107,14 +107,14 @@ public GraphExpansion expand(@Nonnull final Quantifier.ForEach baseQuantifier,
                 .addAll(fieldNamePrefix)
                 .add(getFieldName())
                 .build();
-        final Quantifier.ForEach childBase = Quantifier.forEach(Reference.of(ExplodeExpression.explodeField(baseQuantifier, fieldNames)));
+        final Quantifier.ForEach childBase = Quantifier.forEach(Reference.initialOf(ExplodeExpression.explodeField(baseQuantifier, fieldNames)));
         final SelectExpression selectExpression =
                 GraphExpansion.builder()
                         .addQuantifier(childBase)
                         .addPredicate(childBase.getFlowedObjectValue().withComparison(comparison))
                         .build()
                         .buildSimpleSelectOverQuantifier(childBase);
-        final Quantifier.Existential childQuantifier = Quantifier.existential(Reference.of(selectExpression));
+        final Quantifier.Existential childQuantifier = Quantifier.existential(Reference.initialOf(selectExpression));
 
         // create a query component that creates a path to this prefix and then applies this to it
         // this is needed for reapplication of the component if the sub query cannot be matched or only matched with

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/query/expressions/OneOfThemWithComponent.java

@@ -108,13 +108,13 @@ public GraphExpansion expand(@Nonnull final Quantifier.ForEach baseQuantifier,
                 .addAll(fieldNamePrefix)
                 .add(getFieldName())
                 .build();
-        final Quantifier.ForEach childBase = Quantifier.forEach(Reference.of(ExplodeExpression.explodeField(baseQuantifier, fieldNames)));
+        final Quantifier.ForEach childBase = Quantifier.forEach(Reference.initialOf(ExplodeExpression.explodeField(baseQuantifier, fieldNames)));
         final GraphExpansion graphExpansion = getChild().expand(childBase, outerQuantifierSupplier, Collections.emptyList());
         final SelectExpression selectExpression =
                 GraphExpansion.ofOthers(GraphExpansion.builder().addQuantifier(childBase).build(), graphExpansion)
                         .buildSimpleSelectOverQuantifier(childBase);
 
-        Quantifier.Existential childQuantifier = Quantifier.existential(Reference.of(selectExpression));
+        Quantifier.Existential childQuantifier = Quantifier.existential(Reference.initialOf(selectExpression));
 
         // create a query component that creates a path to this prefix and then applies this to it
         // this is needed for reapplication of the component if the sub query cannot be matched or only matched with

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/query/expressions/QueryRecordFunctionWithComparison.java

@@ -149,7 +149,7 @@ public GraphExpansion expand(@Nonnull final Quantifier.ForEach baseQuantifier,
                             .addResultValue(rankValue)
                             .build();
             final var rankSelectExpression = rankExpansion.buildSelect();
-            final var rankQuantifier = Quantifier.forEach(Reference.of(rankSelectExpression));
+            final var rankQuantifier = Quantifier.forEach(Reference.initialOf(rankSelectExpression));
 
             //
             // Construct another select expression that applies the predicate on the rank value as well as adds a join
@@ -174,7 +174,7 @@ public GraphExpansion expand(@Nonnull final Quantifier.ForEach baseQuantifier,
             final var rankAndJoiningPredicateSelectExpression =
                     GraphExpansion.ofOthers(rankComparisonExpansion, selfJoinPredicateExpansion).buildSelect();
             final var rankComparisonQuantifier =
-                    Quantifier.existential(Reference.of(rankAndJoiningPredicateSelectExpression));
+                    Quantifier.existential(Reference.initialOf(rankAndJoiningPredicateSelectExpression));
 
             // create a query component that creates a path to this prefix and then applies this to it
             // this is needed for reapplication of the component if the sub query cannot be matched or only matched with

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/query/plan/HeuristicPlanner.java

@@ -0,0 +1,36 @@
+/*
+ * HeuristicPlanner.java
+ *
+ * This source file is part of the FoundationDB open source project
+ *
+ * Copyright 2015-2025 Apple Inc. and the FoundationDB project authors
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.apple.foundationdb.record.query.plan;
+
+import java.lang.annotation.Documented;
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * An annotation used on public types, fields, and methods to indicate their usage by the heuristic query planner.
+ */
+@Target({ElementType.TYPE, ElementType.METHOD, ElementType.CONSTRUCTOR, ElementType.FIELD})
+@Retention(RetentionPolicy.CLASS)
+@Documented
+public @interface HeuristicPlanner {
+}

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/query/plan/RecordQueryPlannerConfiguration.java

@@ -25,6 +25,7 @@
 import com.apple.foundationdb.record.RecordPlannerConfigurationProto;
 import com.apple.foundationdb.record.query.RecordQuery;
 import com.apple.foundationdb.record.query.plan.cascades.CascadesRule;
+import com.apple.foundationdb.record.query.plan.cascades.PlannerRule;
 import com.apple.foundationdb.record.query.plan.cascades.PlanningRuleSet;
 import com.apple.foundationdb.record.query.plan.cascades.rules.PredicateToLogicalUnionRule;
 import com.apple.foundationdb.record.query.plan.plans.QueryPlan;
@@ -274,7 +275,7 @@ public RecordQueryPlannerSortConfiguration getSortConfiguration() {
      * @param rule in question
      * @return {@code true} is enabled, {@code false} otherwise
      */
-    public boolean isRuleEnabled(@Nonnull CascadesRule<?> rule) {
+    public boolean isRuleEnabled(@Nonnull PlannerRule<?, ?> rule) {
         return !disabledTransformationRules.contains(rule.getClass().getSimpleName());
     }
 

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/query/plan/bitmap/ComposedBitmapIndexQueryPlan.java

@@ -192,21 +192,18 @@ public Set<CorrelationIdentifier> getCorrelatedTo() {
     public ComposedBitmapIndexQueryPlan translateCorrelations(@Nonnull final TranslationMap translationMap,
                                                               final boolean shouldSimplifyValues,
                                                               @Nonnull final List<? extends Quantifier> translatedQuantifiers) {
+        if (translationMap.definesOnlyIdentities()) {
+            return this;
+        }
+
         final var translatedIndexPlansBuilder = ImmutableList.<RecordQueryCoveringIndexPlan>builder();
-        boolean allAreSame = true;
         for (final var indexPlan : indexPlans) {
             final var translatedIndexPlan =
                     indexPlan.translateCorrelations(translationMap, shouldSimplifyValues, translatedQuantifiers);
-            if (translatedIndexPlan != indexPlan) {
-                allAreSame = false;
-            }
             translatedIndexPlansBuilder.add(translatedIndexPlan);
         }
 
-        if (!allAreSame) {
-            return new ComposedBitmapIndexQueryPlan(translatedIndexPlansBuilder.build(), composer);
-        }
-        return this;
+        return new ComposedBitmapIndexQueryPlan(translatedIndexPlansBuilder.build(), composer);
     }
 
     @Nonnull

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/query/plan/cascades/AggregateIndexExpansionVisitor.java

@@ -144,8 +144,8 @@ public MatchCandidate expand(@Nonnull final Supplier<Quantifier.ForEach> baseQua
 
         // 4. add sort on top, if necessary, this will be absorbed later on as an ordering property of the match candidate.
         final var maybeWithSort = placeHolderAliases.isEmpty()
-                ? Reference.of(selectHaving) // single group, sort by constant
-                : Reference.of(new MatchableSortExpression(placeHolderAliases, isReverse, selectHaving));
+                ? Reference.initialOf(selectHaving) // single group, sort by constant
+                : Reference.initialOf(new MatchableSortExpression(placeHolderAliases, isReverse, selectHaving));
 
         final var traversal = Traversal.withRoot(maybeWithSort);
         return new AggregateIndexMatchCandidate(index,
@@ -213,7 +213,7 @@ private NonnullPair<Quantifier, List<Placeholder>> constructSelectWhereAndPlaceh
         builder.addAllQuantifiers(baseExpansion.getQuantifiers());
         allExpansionsBuilder.add(builder.build());
 
-        return NonnullPair.of(Quantifier.forEach(Reference.of(GraphExpansion.ofOthers(allExpansionsBuilder.build()).buildSelect())), baseExpansion.getPlaceholders());
+        return NonnullPair.of(Quantifier.forEach(Reference.initialOf(GraphExpansion.ofOthers(allExpansionsBuilder.build()).buildSelect())), baseExpansion.getPlaceholders());
     }
 
     @Nonnull
@@ -268,7 +268,7 @@ protected NonnullPair<Quantifier, List<Placeholder>> constructGroupBy(@Nonnull f
                 groupingColsValue.getResultType().getFields().isEmpty() ? null : groupingColsValue,
                 RecordConstructorValue.ofUnnamed(ImmutableList.of(aggregateValue)),
                 GroupByExpression::nestedResults, selectWhereQun);
-        final var groupByReference = Reference.of(groupByExpression);
+        final var groupByReference = Reference.initialOf(groupByExpression);
         final var groupByQuantifier = Quantifier.forEach(groupByReference);
         return NonnullPair.of(groupByQuantifier, ImmutableList.of());
     }

fdb-record-layer-core/src/main/java/com/apple/foundationdb/record/query/plan/cascades/BitmapAggregateIndexExpansionVisitor.java

@@ -118,7 +118,7 @@ protected NonnullPair<Quantifier, List<Placeholder>> constructGroupBy(@Nonnull f
 
         final var pulledUpGroupingValues = ImmutableList.<Value>builder().addAll(explicitPulledUpGroupingValues).add(implicitGroupingValue).build();
         final var groupingColsValue = RecordConstructorValue.ofUnnamed(pulledUpGroupingValues);
-        return NonnullPair.of(Quantifier.forEach(Reference.of(
+        return NonnullPair.of(Quantifier.forEach(Reference.initialOf(
                 new GroupByExpression(groupingColsValue, RecordConstructorValue.ofUnnamed(ImmutableList.of(aggregateValue)),
                         GroupByExpression::nestedResults, selectWhereQun))), ImmutableList.of(placeHolder));
     }