Cut the Gordian Knot: Don't widen unions to transparent (#15642)

The idea is that some unions usually make more sense than others. For
instance,
if `Apply` and `Ident` are case classes that extend `Tree`, it makes
sense to
widen `Apply | Ident` to `Tree`. But it makes less sense to widen
`String | Int`
to `Matchable`.

Making sense means: (1) Matches our intuitive understanding, and (2)
choosing not to
widen would usually not cause errors.

To explain (2): In the `Tree` case it might well be that we define a
given for `Inv[Tree]` for
invariant class `Inv`, and then we would not find that given for
`Inv[Apply | Ident]`.
But it's much less likely that we are looking for a given of type
`Inv[Any]`.

This commit does two things:

- add logic not to widen a union if the result is a product of only
transparent traits or classes.
 - treat `Any`, `AnyVal`, `Object`, and `Matchable` as transparent.

Author: odersky

community-build/community-projects/scalaz

@@ -550,35 +550,39 @@ trait ConstraintHandling {
         inst
   end approximation
 
+  private def isTransparent(tp: Type)(using Context): Boolean = tp match
+    case AndType(tp1, tp2) => isTransparent(tp1) && isTransparent(tp2)
+    case _ => tp.typeSymbol.isTransparentClass && !tp.isLambdaSub
+
   /** If `tp` is an intersection such that some operands are transparent trait instances
    *  and others are not, replace as many transparent trait instances as possible with Any
    *  as long as the result is still a subtype of `bound`. But fall back to the
    *  original type if the resulting widened type is a supertype of all dropped
    *  types (since in this case the type was not a true intersection of transparent traits
    *  and other types to start with).
    */
-  def dropTransparentTraits(tp: Type, bound: Type)(using Context): Type =
+  def dropTransparentClasses(tp: Type, bound: Type)(using Context): Type =
     var kept: Set[Type] = Set()      // types to keep since otherwise bound would not fit
     var dropped: List[Type] = List() // the types dropped so far, last one on top
 
-    def dropOneTransparentTrait(tp: Type): Type =
+    def dropOneTransparentClass(tp: Type): Type =
       val tpd = tp.dealias
-      if tpd.typeSymbol.isTransparentTrait && !tpd.isLambdaSub && !kept.contains(tpd) then
+      if isTransparent(tpd) && !kept.contains(tpd) then
         dropped = tpd :: dropped
         defn.AnyType
       else tpd match
         case AndType(tp1, tp2) =>
-          val tp1w = dropOneTransparentTrait(tp1)
+          val tp1w = dropOneTransparentClass(tp1)
           if tp1w ne tp1 then tp1w & tp2
           else
-            val tp2w = dropOneTransparentTrait(tp2)
+            val tp2w = dropOneTransparentClass(tp2)
             if tp2w ne tp2 then tp1 & tp2w
             else tpd
         case _ =>
           tp
 
     def recur(tp: Type): Type =
-      val tpw = dropOneTransparentTrait(tp)
+      val tpw = dropOneTransparentClass(tp)
       if tpw eq tp then tp
       else if tpw <:< bound then recur(tpw)
       else
@@ -595,7 +599,7 @@ trait ConstraintHandling {
       tp
     else
       tpw
-  end dropTransparentTraits
+  end dropTransparentClasses
 
   /** If `tp` is an applied match type alias which is also an unreducible application
    *  of a higher-kinded type to a wildcard argument, widen to the match type's bound,
@@ -621,7 +625,7 @@ trait ConstraintHandling {
    *      union type (except for unions | Null, which are kept in the state they were).
    *   3. Widen some irreducible applications of higher-kinded types to wildcard arguments
    *      (see @widenIrreducible).
-   *   4. Drop transparent traits from intersections (see @dropTransparentTraits).
+   *   4. Drop transparent traits from intersections (see @dropTransparentClasses).
    *
    *  Don't do these widenings if `bound` is a subtype of `scala.Singleton`.
    *  Also, if the result of these widenings is a TypeRef to a module class,
@@ -648,7 +652,16 @@ trait ConstraintHandling {
 
     val wideInst =
       if isSingleton(bound) then inst
-      else dropTransparentTraits(widenIrreducible(widenOr(widenSingle(inst))), bound)
+      else
+        val widenedFromSingle = widenSingle(inst)
+        val widenedFromUnion = widenOr(widenedFromSingle)
+        val widened =
+          if (widenedFromUnion ne widenedFromSingle) && isTransparent(widenedFromUnion) then
+            widenedFromSingle
+          else
+            dropTransparentClasses(widenedFromUnion, bound)
+        widenIrreducible(widened)
+
     wideInst match
       case wideInst: TypeRef if wideInst.symbol.is(Module) =>
         TermRef(wideInst.prefix, wideInst.symbol.sourceModule)

compiler/src/dotty/tools/dotc/core/ConstraintHandling.scala

@@ -1827,20 +1827,53 @@ class Definitions {
   def isInfix(sym: Symbol)(using Context): Boolean =
     (sym eq Object_eq) || (sym eq Object_ne)
 
-  @tu lazy val assumedTransparentTraits =
-    Set[Symbol](ComparableClass, ProductClass, SerializableClass,
-      // add these for now, until we had a chance to retrofit 2.13 stdlib
-      // we should do a more through sweep through it then.
-      requiredClass("scala.collection.SortedOps"),
-      requiredClass("scala.collection.StrictOptimizedSortedSetOps"),
-      requiredClass("scala.collection.generic.DefaultSerializable"),
-      requiredClass("scala.collection.generic.IsIterable"),
-      requiredClass("scala.collection.generic.IsIterableOnce"),
-      requiredClass("scala.collection.generic.IsMap"),
-      requiredClass("scala.collection.generic.IsSeq"),
-      requiredClass("scala.collection.generic.Subtractable"),
-      requiredClass("scala.collection.immutable.StrictOptimizedSeqOps")
-    )
+  @tu lazy val assumedTransparentNames: Map[Name, Set[Symbol]] =
+    // add these for now, until we had a chance to retrofit 2.13 stdlib
+    // we should do a more through sweep through it then.
+    val strs = Map(
+      "Any" -> Set("scala"),
+      "AnyVal" -> Set("scala"),
+      "Matchable" -> Set("scala"),
+      "Product" -> Set("scala"),
+      "Object" -> Set("java.lang"),
+      "Comparable" -> Set("java.lang"),
+      "Serializable" -> Set("java.io"),
+      "BitSetOps" -> Set("scala.collection"),
+      "IndexedSeqOps" -> Set("scala.collection", "scala.collection.mutable", "scala.collection.immutable"),
+      "IterableOnceOps" -> Set("scala.collection"),
+      "IterableOps" -> Set("scala.collection"),
+      "LinearSeqOps" -> Set("scala.collection", "scala.collection.immutable"),
+      "MapOps" -> Set("scala.collection", "scala.collection.mutable", "scala.collection.immutable"),
+      "SeqOps" -> Set("scala.collection", "scala.collection.mutable", "scala.collection.immutable"),
+      "SetOps" -> Set("scala.collection", "scala.collection.mutable", "scala.collection.immutable"),
+      "SortedMapOps" -> Set("scala.collection", "scala.collection.mutable", "scala.collection.immutable"),
+      "SortedOps" -> Set("scala.collection"),
+      "SortedSetOps" -> Set("scala.collection", "scala.collection.mutable", "scala.collection.immutable"),
+      "StrictOptimizedIterableOps" -> Set("scala.collection"),
+      "StrictOptimizedLinearSeqOps" -> Set("scala.collection"),
+      "StrictOptimizedMapOps" -> Set("scala.collection", "scala.collection.immutable"),
+      "StrictOptimizedSeqOps" -> Set("scala.collection", "scala.collection.immutable"),
+      "StrictOptimizedSetOps" -> Set("scala.collection", "scala.collection.immutable"),
+      "StrictOptimizedSortedMapOps" -> Set("scala.collection", "scala.collection.immutable"),
+      "StrictOptimizedSortedSetOps" -> Set("scala.collection", "scala.collection.immutable"),
+      "ArrayDequeOps" -> Set("scala.collection.mutable"),
+      "DefaultSerializable" -> Set("scala.collection.generic"),
+      "IsIterable" -> Set("scala.collection.generic"),
+      "IsIterableLowPriority" -> Set("scala.collection.generic"),
+      "IsIterableOnce" -> Set("scala.collection.generic"),
+      "IsIterableOnceLowPriority" -> Set("scala.collection.generic"),
+      "IsMap" -> Set("scala.collection.generic"),
+      "IsSeq" -> Set("scala.collection.generic"))
+    strs.map { case (simple, pkgs) => (
+        simple.toTypeName,
+        pkgs.map(pkg => staticRef(pkg.toTermName, isPackage = true).symbol.moduleClass)
+      )
+    }
+
+  def isAssumedTransparent(sym: Symbol): Boolean =
+    assumedTransparentNames.get(sym.name) match
+      case Some(pkgs) => pkgs.contains(sym.owner)
+      case none => false
 
   // ----- primitive value class machinery ------------------------------------------
 

compiler/src/dotty/tools/dotc/core/Definitions.scala

@@ -357,7 +357,7 @@ object Flags {
   val (_, DefaultMethod @ _, _) = newFlags(38, "<defaultmethod>")
 
   /** Symbol is a transparent inline method or trait */
-  val (Transparent @ _, _, _) = newFlags(39, "transparent")
+  val (Transparent @ _, _, TransparentType @ _) = newFlags(39, "transparent")
 
   /** Symbol is an enum class or enum case (if used with case) */
   val (Enum @ _, EnumVal @ _, _) = newFlags(40, "enum")
@@ -609,5 +609,4 @@ object Flags {
   val SyntheticParam: FlagSet                = Synthetic | Param
   val SyntheticTermParam: FlagSet            = Synthetic | TermParam
   val SyntheticTypeParam: FlagSet            = Synthetic | TypeParam
-  val TransparentTrait: FlagSet              = Trait | Transparent
 }

compiler/src/dotty/tools/dotc/core/Flags.scala

@@ -1151,9 +1151,9 @@ object SymDenotations {
     final def isEffectivelySealed(using Context): Boolean =
       isOneOf(FinalOrSealed) || isClass && !isOneOf(EffectivelyOpenFlags)
 
-    final def isTransparentTrait(using Context): Boolean =
-      isAllOf(TransparentTrait)
-      || defn.assumedTransparentTraits.contains(symbol)
+    final def isTransparentClass(using Context): Boolean =
+      is(TransparentType)
+      || defn.isAssumedTransparent(symbol)
       || isClass && hasAnnotation(defn.TransparentTraitAnnot)
 
     /** The class containing this denotation which has the given effective name. */

compiler/src/dotty/tools/dotc/core/SymDenotations.scala

@@ -3007,8 +3007,8 @@ object TypeComparer {
   def widenInferred(inst: Type, bound: Type, widenUnions: Boolean)(using Context): Type =
     comparing(_.widenInferred(inst, bound, widenUnions))
 
-  def dropTransparentTraits(tp: Type, bound: Type)(using Context): Type =
-    comparing(_.dropTransparentTraits(tp, bound))
+  def dropTransparentClasses(tp: Type, bound: Type)(using Context): Type =
+    comparing(_.dropTransparentClasses(tp, bound))
 
   def constrainPatternType(pat: Type, scrut: Type, forceInvariantRefinement: Boolean = false)(using Context): Boolean =
     comparing(_.constrainPatternType(pat, scrut, forceInvariantRefinement))

compiler/src/dotty/tools/dotc/core/TypeComparer.scala

@@ -1210,7 +1210,7 @@ trait Applications extends Compatibility {
        && tree.tpe.classSymbol.isEnumCase
        && tree.tpe.widen.isValueType
     then
-      val widened = TypeComparer.dropTransparentTraits(
+      val widened = TypeComparer.dropTransparentClasses(
         tree.tpe.parents.reduceLeft(TypeComparer.andType(_, _)),
         pt)
       if widened <:< pt then Typed(tree, TypeTree(widened))

compiler/src/dotty/tools/dotc/typer/Applications.scala

@@ -495,7 +495,7 @@ object Checking {
     }
     if sym.is(Transparent) then
       if sym.isType then
-        if !sym.is(Trait) then fail(em"`transparent` can only be used for traits".toMessage)
+        if !sym.isExtensibleClass then fail(em"`transparent` can only be used for extensible classes and traits".toMessage)
       else
         if !sym.isInlineMethod then fail(em"`transparent` can only be used for inline methods".toMessage)
     if (!sym.isClass && sym.is(Abstract))

compiler/src/dotty/tools/dotc/typer/Checking.scala

@@ -122,7 +122,7 @@ class ShadowingTests extends ReplTest(options = ShadowingTests.options):
          |val y: String = foo
          |
          |scala> if (true) x else y
-         |val res0: Matchable = 42
+         |val res0: Int | String = 42
          |""".stripMargin.linesIterator.toList
     )
 

compiler/test/dotty/tools/repl/ShadowingTests.scala

@@ -72,6 +72,10 @@ a non-union type, for this purpose we define the _join_ of a union type `T1 |
 `T1`,...,`Tn`. Note that union types might still appear as type arguments in the
 resulting type, this guarantees that the join is always finite.
 
+The _visible join_ of a union type is its join where all operands of the intersection that
+are instances of [transparent](../other-new-features/transparent-traits.md) traits or classes are removed.
+
+
 ### Example
 
 Given
@@ -80,31 +84,50 @@ Given
 trait C[+T]
 trait D
 trait E
-class A extends C[A] with D
-class B extends C[B] with D with E
+transparent trait X
+class A extends C[A], D, X
+class B extends C[B], D, E, X
 ```
 
-The join of `A | B` is `C[A | B] & D`
+The join of `A | B` is `C[A | B] & D & X` and the visible join of `A | B` is `C[A | B] & D`.
+
+## Hard and Soft Union Types
+
+We distinguish between hard and soft union types. A _hard_ union type is a union type that's explicitly
+written in the source. For instance, in
+```scala
+val x: Int | String = ...
+```
+`Int | String` would be a hard union type. A _soft_ union type is a type that arises from type checking
+an alternative of expressions. For instance, the type of the expression
+```scala
+val x = 1
+val y = "abc"
+if cond then x else y
+```
+is the soft unon type `Int | String`. Similarly for match expressions. The type of
+```scala
+x match
+  case 1 => x
+  case 2 => "abc"
+  case 3 => List(1, 2, 3)
+```
+is the soft union type `Int | "abc" | List[Int]`.
+
 
 ## Type inference
 
 When inferring the result type of a definition (`val`, `var`, or `def`) and the
-type we are about to infer is a union type, then we replace it by its join.
+type we are about to infer is a soft union type, then we replace it by its visible join,
+provided it is not empty.
 Similarly, when instantiating a type argument, if the corresponding type
 parameter is not upper-bounded by a union type and the type we are about to
-instantiate is a union type, we replace it by its join. This mirrors the
+instantiate is a soft union type, we replace it by its visible join, provided it is not empty.
+This mirrors the
 treatment of singleton types which are also widened to their underlying type
 unless explicitly specified. The motivation is the same: inferring types
 which are "too precise" can lead to unintuitive typechecking issues later on.
 
-**Note:** Since this behavior limits the usability of union types, it might
-be changed in the future. For example by not widening unions that have been
-explicitly written down by the user and not inferred, or by not widening a type
-argument when the corresponding type parameter is covariant.
-
-See [PR #2330](https://github.com/lampepfl/dotty/pull/2330) and
-[Issue #4867](https://github.com/lampepfl/dotty/issues/4867) for further discussions.
-
 ### Example
 
 ```scala