Daemon crash caused by `importlib.util` missing from `import_map`

[sterliakov]

Crash Report

I was investigating #9366 (good news: looks like the original problem there is resolved, but I'm not sure yet) and observed that the pytest repro in #9366 (comment) is now crashing for different reason.

Traceback

Traceback (most recent call last):
  File "/home/stas/Documents/Work/mypy/mypy/dmypy/client.py", line 281, in main
    args.action(args)
  File "/home/stas/Documents/Work/mypy/mypy/dmypy/client.py", line 639, in do_daemon
    Server(options, args.status_file, timeout=args.timeout).serve()
  File "/home/stas/Documents/Work/mypy/mypy/dmypy_server.py", line 237, in serve
    resp = self.run_command(command, data)
  File "/home/stas/Documents/Work/mypy/mypy/dmypy_server.py", line 286, in run_command
    ret = method(self, **data)
  File "/home/stas/Documents/Work/mypy/mypy/dmypy_server.py", line 354, in cmd_run
    return self.check(sources, export_types, is_tty, terminal_width)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/dmypy_server.py", line 433, in check
    messages = self.fine_grained_increment_follow_imports(
        sources, explicit_export_types=export_types
    )
  File "/home/stas/Documents/Work/mypy/mypy/dmypy_server.py", line 683, in fine_grained_increment_follow_imports
    messages = fine_grained_manager.update(new_unsuppressed, [], followed=True)
  File "/home/stas/Documents/Work/mypy/mypy/server/update.py", line 265, in update
    result = self.update_one(
        changed_modules, initial_set, removed_set, blocking_error, followed
    )
  File "/home/stas/Documents/Work/mypy/mypy/server/update.py", line 367, in update_one
    result = self.update_module(next_id, next_path, next_id in removed_set, followed)
  File "/home/stas/Documents/Work/mypy/mypy/server/update.py", line 432, in update_module
    result = update_module_isolated(
        module, path, manager, previous_modules, graph, force_removed, followed
    )
  File "/home/stas/Documents/Work/mypy/mypy/server/update.py", line 655, in update_module_isolated
    semantic_analysis_for_scc(graph, [state.id], manager.errors)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/semanal_main.py", line 96, in semantic_analysis_for_scc
    process_functions(graph, scc, patches)
  File "/home/stas/Documents/Work/mypy/mypy/semanal_main.py", line 305, in process_functions
    process_top_level_function(
  File "/home/stas/Documents/Work/mypy/mypy/semanal_main.py", line 344, in process_top_level_function
    deferred, incomplete, progress = semantic_analyze_target(
  File "/home/stas/Documents/Work/mypy/mypy/semanal_main.py", line 410, in semantic_analyze_target
    analyzer.refresh_partial(
  File "/home/stas/Documents/Work/mypy/mypy/semanal.py", line 704, in refresh_partial
    self.accept(node)
    ~~~~~~~~~~~^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/semanal.py", line 7566, in accept
    report_internal_error(err, self.errors.file, node.line, self.errors, self.options)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/errors.py", line 1334, in report_internal_error
    raise err
  File "/home/stas/Documents/Work/mypy/mypy/semanal.py", line 7564, in accept
    node.accept(self)
    ~~~~~~~~~~~^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/nodes.py", line 994, in accept
    return visitor.visit_func_def(self)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/semanal.py", line 978, in visit_func_def
    self.analyze_func_def(defn)
    ~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/semanal.py", line 994, in analyze_func_def
    has_self_type = self.update_function_type_variables(defn.type, defn)
  File "/home/stas/Documents/Work/mypy/mypy/semanal.py", line 1216, in update_function_type_variables
    fun_type.variables, has_self_type = a.bind_function_type_variables(fun_type, defn)
                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/typeanal.py", line 1858, in bind_function_type_variables
    typevars, has_self_type = self.infer_type_variables(fun_type)
                              ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/typeanal.py", line 1836, in infer_type_variables
    type.ret_type.accept(visitor)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/types.py", line 3404, in accept
    return visitor.visit_union_type(self)
           ~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/typeanal.py", line 2667, in visit_union_type
    self.process_types(t.items)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/typeanal.py", line 2689, in process_types
    t.accept(self)
    ~~~~~~~~^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/types.py", line 1095, in accept
    return visitor.visit_unbound_type(self)
           ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/typeanal.py", line 2583, in visit_unbound_type
    node = self.api.lookup_qualified(name, t)
  File "/home/stas/Documents/Work/mypy/mypy/semanal.py", line 6650, in lookup_qualified
    nextsym = self.get_module_symbol(node, part)
  File "/home/stas/Documents/Work/mypy/mypy/semanal.py", line 6702, in get_module_symbol
    if fullname in self.modules and self.is_visible_import(module, fullname):
                                    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
  File "/home/stas/Documents/Work/mypy/mypy/semanal.py", line 6722, in is_visible_import
    if id in self.import_map[self.cur_mod_id]:
             ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'importlib.util'

To Reproduce

This is reproducible in a testcase, but I haven't been able to reproduce without importlib yet. The big pytest-based repro crashes missing _pytest.assertion.rewrite next time if I ignore importlib.util.

[case testDaemonImportMapRefresh]
$ dmypy start
Daemon started
$ dmypy check test.py
Success: no issues found in 1 source file
$ {python} -c 'print("import importlib.util")' > test.py
$ dmypy check test.py
Success: no issues found in 1 source file
[file test.py]

Rebuilding manager.import_map in mypy.server.update::ensure_trees_loaded unconditionally helps (for id in graph: manager.import_map[id] = set(graph[id].dependencies + graph[id].suppressed)), adding that makes the original repro in #9366 pass without crashes or false positive diagnostics, but I hope there's a saner way than that?

Your Environment

• Mypy version used: current master
• Mypy command-line flags: none
• Mypy configuration options from mypy.ini (and other config files): none
• Python version used: 3.13
• Operating system and version: Ubuntu 24.04 LTS, probably irrelevant

[sterliakov]

And @ilevkivskyi sorry for the ping, but import_map must be still fresh in your memory - you just added it in #20179. Does this click?

[ilevkivskyi]

but I hope there's a saner way than that?

I am not sure. I definitely remember a lot of problems with daemon when adding import_map. This is why there are all those if ... in self.import_map, they exist only because of the daemon. But TBH module currently being processed not in the import map is the whole new level of WAT.

Have you tried limiting the loop to only to_process (and/or newly added modules)? I am worried that recomputing full import map on each update will result in big performance penalty for large code bases (and people are more likely to use daemon with large code bases). If this doesn't work, another option would be to add something like:

if self.cur_mod_id not in self.import_map:
    return False

before the crash. This may cause an error, but daemon has logic to reprocess targets with errors, so it may ultimately resolve.

[sterliakov]

I considered ignoring this KeyError right there, but it feels so... wrong? self.cur_mod_id in self.import_map is our invariant, I don't feel good about silencing that.

Have you tried limiting the loop to only to_process (and/or newly added modules)?

I tried to_process and it still crashed, but it was a wrong thing to do - in the testcase we have to check importlib.util directly, it's never in to_process - we only request its deps from ensure_deps_loaded. Just recalculating it for current update batch seems to do the trick:

diff --git a/mypy/server/update.py b/mypy/server/update.py
index 839090ca4..86ccb57c2 100644
--- a/mypy/server/update.py
+++ b/mypy/server/update.py
@@ -631,6 +631,8 @@ def update_module_isolated(
 
     # Find any other modules brought in by imports.
     changed_modules = [(st.id, st.xpath) for st in new_modules]
+    for m in new_modules:
+        manager.import_map[m.id] = set(m.dependencies + m.suppressed)
 
     # If there are multiple modules to process, only process one of them and return
     # the remaining ones to the caller.

I have no idea whether there are other affected places, but this is at least well-scoped and should have zero perf impact. Would be nice to understand the real reproducer structure.

will result in big performance penalty

Yeah, that's why I hope for a better approach...

[ilevkivskyi]

Just recalculating it for current update batch seems to do the trick

OK, I think this should be the fix then.