Take charge of parsing and evaluating inputs

[lionel-]

Progress towards posit-dev/positron#1766
Progress towards posit-dev/positron#3078
Progress towards posit-dev/positron#9156
Addresses posit-dev/positron#6553
Closes #598
Closes #722
Closes #840

This PR does two things:

1. It takes charge of Reading/Parsing and Evaluation of R code. (First commit)
2. Refactors, simplifies, and fixes a bunch of issues. (All other commits)

Doing parsing and evaluation in Ark is a significant departure of how read_console(), the heart of the kernel, is currently structured. This handler is hooked to the R frontend method ReadConsole, which takes user inputs as strings. The R REPL gets these strings, parses them, evaluates them, prints the result, then loops back. With this PR, we take control of both parsing and evaluation.

There is no functional changes here except for a few bugfixes. In the future though, this will allow us to:

• Parse all inputs sent from editors with source references and injected breakpoints.
• When debugging, evaluate code in the frame selected in the IDE.

New Features

• Selections of code are now parsed in a single pass. A minor consequence is that all functions evaluated as part of the same selection share a common virtual document when step-debugging through them.

• recover() is now an alias for browser(). The R version of recover() was buggy (options(error = recover) seems broken positron#9156) and it was hard to conciliate its behaviour with the requirements of the Jupyter protocol. By aliasing it to browser(), we can leverage the IDE's call stack pane to fulfill the same functionality as the original recover() function (R: Frame-aware custom debug REPL positron#3078).

Fixes

• Minor: We no longer have any size limitation on lines of inputs. They were previously limited to 4096 bytes, the maximum size of the ReadConsole input buffer.

• Restart during debug sessions now works as expected: Restart the R kernel during debugging session leads to bad state positron#6553

• We now interrupt ongoing evaluations before restarting. This shouldn't have any impact on Positron, which sends an interrupt before restarting, but should make restarts behave better with other frontends. This is also a nice defensive measure even with Positron.

• We now flush autoprint output when an error is thrown by a print method. I was quite confused by this bug because it also swallowed debugging output!

• The set of syntax errors that throw an R error instead of returning an error code no longer cause a leaked backtrace in the console (Sending a pair of empty backticks to the kernel results in an internal error showing up in the console #598, Weird error with "\s" #722)

These fixes (and more) are covered by integration tests.

Approach

• We parse inputs into a vector of complete expressions. These expressions are stored in the read-console state in a new PendingInputs struct that implements a stack interface, with a pop method. This stack of pending inputs replaces the stack pending lines we implemented in Split inputs by newlines #536.

• When read-console is called by R for a new input, it is evaluated and the result is stored in base::.ark_last_value. We store it in the base environment for robust access from (almost) any evaluation environment (global on top level but the debugger can evaluate in any environment). We only require the presence of :: so that we can reach into base.

• Read-console returns to R with the string input "base::.ark_last_value" if evaluation returned visibly, or "base::invisible(base::.ark_last_value)" if it returned invisibly. The R REPL then parses this expression and evaluates it.

• Because we are still hooked into the R REPL, it has a chance to do top-level stuff like printing warnings, running task callbacks, updating .Last.value, etc.

While this approach works, I hit two main issues that we need to workaround:

• When an error happens, the R_EvalDepth variable is not reset before the next evaluation (it's normally reset here: https://github.com/r-devel/r-svn/blob/811080fb/src/main/main.c#L260). This could be bad especially if that error is an exceeded evaluation depth error and we're looping back to evaluate a new expression without having reset it.

• When a nested debug REPL returns to a parent REPL, the R_ConsoleIob parse buffer is normally reset after returning from eval() (here: https://github.com/r-devel/r-svn/blob/811080fb/src/main/main.c#L279). With this approach, it doesn't have a chance to reset before entering read-console again.

To solve the first issue, we evaluate a dummy input ("base::.Last.value" to keep the last value stable, this is tested) after an error occurs. This gives R a chance to fully reset state. To solve the second issue, we send a dummy parse causing a PARSE_NULL everytime we detect a nested read-console has returned. This causes R to reset the parse buffer here: https://github.com/r-devel/r-svn/blob/811080fb/src/main/main.c#L238.

These workarounds are unfortunate but have been mostly contained in the r_read_console() method, which keeps track of all these details and other state.

Reorganisation

• Read-console is both an event loop (when we're waiting for input from the frontend) and a state machine (to manage execution of inputs). The event loop part is now extracted in a new method run_event_loop(). We no longer wait for both input replies (when we're asking the frontend to reply to a readline() or menu() question) and execution requests (when we're waiting for the frontend to send us the next bit of code to execute) at the same time, it's either one or the other, which simplifies the code. The handler for input requests now invokes the event loop itself, instead of returning to the top level of read-console to fall through the event loop.

• The state machine part of read-console has been simplified in other ways. We now inspect the states from the top level and pass it down through arguments, which makes it easier to reason about. See in particular the new take_exception() and take_result() methods called from top-level and passed down to the handler for replying to an active request when there are no inputs left to evaluate. This allowed simplifying things quite a bit.

• We now keep track of the number of nested consoles (due to active debug sessions) on the stack. Not used for anything but it seems good information to keep track of. To support this (and other state bookkeeping), we use a new wrapper around R_ExecWithCleanup().

• The handling of debugger commands has been consolidated into a single location.

• A benefit of parsing expressions ourselves is that it is no longer possible for R to send us an incomplete prompt. This has been completely removed from the possible states read-console can be in, which is another nice simplification.

• The DummyFrontend for integration tests has now a few more methods like execute_request() that capture common patterns of messaging. This removes quite a lot of boilerplate.

[DavisVaughan]

Looks good! I've been running this for a few days and have not had any major issues.

In addition to my comments below, two other things

---

I still think Option would be useful around srcrefs to show to the reader that they are optional. I confused myself about them again today. It doesn't seem too hard to do this #961

---

There is one regression I noted with debugonce() when inside a package that you have done load_all() with:

devtools::load_all()
debugonce(vec_count)
vec_count(1:5)

If you take a step and then run 1:2 in the Console, then with this PR you get jumped to a fallback view of vec_count, which is pretty jarring and isn't right compared with the current behavior of staying in the deparsed srcref file we created for vctrs.

This seems worth fixing before merging.

This PR:

Screen.Recording.2025-11-11.at.9.15.22.AM.mov

Current main:

Screen.Recording.2025-11-11.at.9.16.28.AM.mov