[SPARK-54289][SQL][FOLLOW-UP] Make Merge Into update assignment by field default for UPDATE SET * and align configs

### What changes were proposed in this pull request?
Follow up of: #53149

1. Make the update assignment by field the Spark 4.1 behavior.  For context, the case to allow assignment key and value to be different struct for MERGE INTO is new in Spark 4.1 so we have a chance to define the behavior.  In Spark, nested fields are usually treated as top level column so it should follow the behavior: see #53149 (comment)
2. Rename existing config to control the struct type compatibility check in assignment.  We do not need to mention 'source' as actually the assignment can be to anything, not necessarily to source table.

### Why are the changes needed?
See above

### Does this PR introduce _any_ user-facing change?
No, this feature is unreleased (allowing assignment source to be of different struct type as target)

### How was this patch tested?
Existing unit test

### Was this patch authored or co-authored using generative AI tooling?
No

Closes #53199 from szehon-ho/merge_schema_evolution_update_nested_follow.

Authored-by: Szehon Ho <szehon.apache@gmail.com>
Signed-off-by: Dongjoon Hyun <dongjoon@apache.org>

Author: szehon-ho

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/AssignmentUtils.scala

@@ -29,7 +29,6 @@ import org.apache.spark.sql.catalyst.util.CharVarcharUtils
 import org.apache.spark.sql.catalyst.util.ResolveDefaultColumns.getDefaultValueExprOrNullLit
 import org.apache.spark.sql.connector.catalog.CatalogV2Implicits._
 import org.apache.spark.sql.errors.QueryCompilationErrors
-import org.apache.spark.sql.internal.SQLConf
 import org.apache.spark.sql.types.{DataType, StructType}
 import org.apache.spark.util.ArrayImplicits._
 
@@ -183,7 +182,7 @@ object AssignmentUtils extends SQLConfHelper with CastSupport {
     } else if (exactAssignments.isEmpty && fieldAssignments.isEmpty) {
       TableOutputResolver.checkNullability(colExpr, col, conf, colPath)
     } else if (exactAssignments.nonEmpty) {
-     if (SQLConf.get.mergeUpdateStructsByField && updateStar) {
+     if (updateStar) {
         val value = exactAssignments.head.value
         col.dataType match {
           case structType: StructType =>

sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveRowLevelCommandAssignments.scala

@@ -53,7 +53,7 @@ object ResolveRowLevelCommandAssignments extends Rule[LogicalPlan] {
     case m: MergeIntoTable if !m.skipSchemaResolution && m.resolved && m.rewritable && !m.aligned &&
       !m.needSchemaEvolution =>
       validateStoreAssignmentPolicy()
-      val coerceNestedTypes = SQLConf.get.mergeCoerceNestedTypes
+      val coerceNestedTypes = SQLConf.get.coerceMergeNestedTypes
       m.copy(
         targetTable = cleanAttrMetadata(m.targetTable),
         matchedActions = alignActions(m.targetTable.output, m.matchedActions,

sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala

@@ -6704,18 +6704,6 @@ object SQLConf {
       .booleanConf
       .createWithDefault(true)
 
-  val MERGE_INTO_NESTED_TYPE_UPDATE_BY_FIELD =
-    buildConf("spark.sql.merge.nested.type.assign.by.field")
-      .internal()
-      .doc("If enabled and spark.sql.merge.source.nested.type.coercion.enabled is true," +
-        "allow MERGE INTO with UPDATE SET * action to set nested structs field by field. " +
-        "In updated rows, target structs will preserve the original value for fields missing " +
-        "in the the source struct. If disabled, the entire target struct will be replaced, " +
-        "and fields missing in the source struct will be null.")
-      .version("4.1.0")
-      .booleanConf
-      .createWithDefault(true)
-
   /**
    * Holds information about keys that have been deprecated.
    *
@@ -7915,12 +7903,9 @@ class SQLConf extends Serializable with Logging with SqlApiConf {
   def legacyXMLParserEnabled: Boolean =
     getConf(SQLConf.LEGACY_XML_PARSER_ENABLED)
 
-  def mergeCoerceNestedTypes: Boolean =
+  def coerceMergeNestedTypes: Boolean =
     getConf(SQLConf.MERGE_INTO_NESTED_TYPE_COERCION_ENABLED)
 
-  def mergeUpdateStructsByField: Boolean =
-    getConf(SQLConf.MERGE_INTO_NESTED_TYPE_UPDATE_BY_FIELD)
-
   /** ********************** SQLConf functionality methods ************ */
 
   /** Set Spark SQL configuration properties. */