Fix guess_rec_per_key to not calculate negative values

Summary:
The function `guess_rec_per_key` tries to interpolate index cardinality by multiplying by (current keypart number) / (total keyparts). However, the denominator uses `key->user_defined_key_parts` which does not include extended key parts, while the current keypart being passed could be referencing extended keyparts. This means that the multiplicative factor could be > 1 which breaks some assumptions in this function.

This bug mainly affects myrocks, since innodb never needs to guess `rec_per_key` as it is always populated in `::info`.

Reviewed By: luqun

Differential Revision: D43013207

fbshipit-source-id: 9060eaa

Author: Manuel Ung

sql/opt_statistics.cc

@@ -101,9 +101,11 @@ rec_per_key_t guess_rec_per_key(const TABLE *const table, const KEY *const key,
     if (key->user_defined_key_parts > 1) {
       // See formula above
       rec_per_key =
-          rec_per_key_first - (rec_per_key_t(used_keyparts - 1) /
-                               (key->user_defined_key_parts - 1)) *
-                                  (rec_per_key_first - rec_per_key_all);
+          rec_per_key_first -
+          (rec_per_key_t(std::min(used_keyparts, key->user_defined_key_parts) -
+                         1) /
+           (key->user_defined_key_parts - 1)) *
+              (rec_per_key_first - rec_per_key_all);
     } else {
       // Single column index
       if (key->actual_flags & HA_NOSAME)