Tokenizer/PHP: bug fix - "yield" vs function return by ref (PHP 5.4)

On PHP 5.4, where PHP doesn't natively have the `T_YIELD` token yet, a `yield` function name for a function declared to return by reference would be tokenized as `T_YIELD`, instead of `T_STRING` due to the `T_YIELD` polyfill happening _after_ the context sensitive keyword check has already run.

By moving the check for a `T_STRING` "yield" keyword up to above the check for context sensitive keywords, this bug is fixed.

Includes additional test.

Author: jrfnl

src/Tokenizers/PHP.php

@@ -605,6 +605,27 @@ protected function tokenize($string)
                 echo PHP_EOL;
             }
 
+            /*
+                Before PHP 5.5, the yield keyword was tokenized as
+                T_STRING. So look for and change this token in
+                earlier versions.
+            */
+
+            if (PHP_VERSION_ID < 50500
+                && $tokenIsArray === true
+                && $token[0] === T_STRING
+                && strtolower($token[1]) === 'yield'
+                && isset($this->tstringContexts[$finalTokens[$lastNotEmptyToken]['code']]) === false
+            ) {
+                // Could still be a context sensitive keyword or "yield from" and potentially multi-line,
+                // so adjust the token stack in place.
+                $token[0] = T_YIELD;
+
+                if (PHP_CODESNIFFER_VERBOSITY > 1) {
+                    echo "\t\t* token $stackPtr changed from T_STRING to T_YIELD".PHP_EOL;
+                }
+            }
+
             /*
                 Tokenize context sensitive keyword as string when it should be string.
             */
@@ -1492,26 +1513,6 @@ protected function tokenize($string)
                 continue;
             }//end if
 
-            /*
-                Before PHP 5.5, the yield keyword was tokenized as
-                T_STRING. So look for and change this token in
-                earlier versions.
-            */
-
-            if (PHP_VERSION_ID < 50500
-                && $tokenIsArray === true
-                && $token[0] === T_STRING
-                && strtolower($token[1]) === 'yield'
-                && isset($this->tstringContexts[$finalTokens[$lastNotEmptyToken]['code']]) === false
-            ) {
-                // Could still be "yield from" and potentially multi-line, so adjust the token stack.
-                $token[0] = T_YIELD;
-
-                if (PHP_CODESNIFFER_VERBOSITY > 1) {
-                    echo "\t\t* token $stackPtr changed from T_STRING to T_YIELD".PHP_EOL;
-                }
-            }
-
             /*
                 Before PHP 7.0, the "yield from" was tokenized as
                 T_YIELD, T_WHITESPACE and T_STRING. So look for

tests/Core/Tokenizers/PHP/YieldTest.inc

@@ -42,6 +42,9 @@ class Yield {
         /* testFromUsedForClassConstantAccess1 */
         echo MyClass::FROM;
     }
+
+    /* testYieldUsedAsMethodNameReturnByRef */
+    public function &yield() {}
 }
 
 function myGen() {

tests/Core/Tokenizers/PHP/YieldTest.php

@@ -232,6 +232,7 @@ public static function dataYieldNonKeyword()
             'yield used as property access 2'     => ['/* testYieldUsedAsPropertyName2 */'],
             'yield used as class constant access' => ['/* testYieldUsedForClassConstantAccess1 */'],
             'from used as class constant access'  => ['/* testFromUsedForClassConstantAccess1 */'],
+            'yield used as method name with ref'  => ['/* testYieldUsedAsMethodNameReturnByRef */'],
         ];
 
     }//end dataYieldNonKeyword()