Skip to content

error RuntimeError: Set changed size during iteration #1498

@zukka77

Description

@zukka77

Describe the bug

Launching a job to retrieve files on 715 host gives this error:

 RuntimeError: Set changed size during iteration
[snip].venv/lib/python3.13/site-packages/pyinfra_cli/util.py", line 221, in _parallel_load_hosts
    state.pool.spawn(load_file, host): host for host in state.inventory.iter_active_hosts()


To Reproduce

I'm using ssh connector on 715 hosts (not all hosts are reachable) , the inventory is dynamic:
from my_password import password

def make_hosts():
    with open("hostlist.csv", "r") as f:
        hosts = list(map(lambda x: x.strip().lower(), f.readlines()))
    return {
        "hosts": (
            hosts,
            {
                "ssh_user": "user",
                "ssh_password": password,
                "ssh_allow_agent": False,
                "ssh_known_hosts_file": "/dev/null",
                "_parallel": 10,
                "_sudo": True,
            },
        )
    }

I've tried with or without _parallel
these are the operations:

from pyinfra import host
from pyinfra.facts import server, files as files_facts
from pyinfra.operations import files
from pathlib import Path

hostname: str = host.get_fact(server.Hostname)
output_path = Path(f"sudoers/{hostname}")
(output_path / "sudoers.d").mkdir(parents=True, exist_ok=True)

files.get(
    src="/etc/sudoers",
    dest=f"sudoers/{hostname}/sudoers",
)

sudoers_d = host.get_fact(
    files_facts.FindFiles,
    "/etc/sudoers.d",
    quote_path=True,
)

for f in sudoers_d:
    files.get(
        src=f,
        dest=f"sudoers/{hostname}/sudoers.d/{Path(f).name}",
    )

Expected behavior

I'd like to have the files on my machine 😄

Meta

  • Include output of pyinfra --support.
    System: Linux
    Platform: Linux-6.17.7-arch1-1-x86_64-with-glibc2.42
    Release: 6.17.7-arch1-1
    Machine: x86_64
    pyinfra: v3.5.1
    click: v8.3.0
    distro: v1.9.0
    gevent: v25.9.1
    jinja2: v3.1.6
    packaging: v25.0
    paramiko: v3.5.1
    python-dateutil: v2.9.0.post0
    typeguard: v4.4.4
    typing-extensions: v4.15.0
    Executable: [snip]/.venv/bin/pyinfra
    Python: 3.13.2 (CPython, Clang 19.1.6 )

  • How was pyinfra installed (source/pip)?
    uv pip

  • Include pyinfra-debug.log (if one was created)

File " [snip]/.venv/lib/python3.13/site-packages/pyinfra_cli/cli.py", line 243, in cli
    _main(*args, **kwargs)
    ~~~~~^^^^^^^^^^^^^^^^^
  File " [snip]/.venv/lib/python3.13/site-packages/pyinfra_cli/cli.py", line 378, in _main
    can_diff, state, config = _handle_commands(
                              ~~~~~~~~~~~~~~~~^
        state, config, command, original_operations, operations
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File " [snip]/.venv/lib/python3.13/site-packages/pyinfra_cli/cli.py", line 705, in _handle_commands
    state, config, operations = _prepare_deploy_operations(state, config, operations)
                                ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File " [snip]/.venv/lib/python3.13/site-packages/pyinfra_cli/cli.py", line 766, in _prepare_deploy_operations
    load_deploy_file(state, filename)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File " [snip]/.venv/lib/python3.13/site-packages/pyinfra_cli/util.py", line 235, in load_deploy_file
    _parallel_load_hosts(state, lambda: exec_file(filename), filename)
    ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File " [snip]/.venv/lib/python3.13/site-packages/pyinfra_cli/util.py", line 221, in _parallel_load_hosts
    state.pool.spawn(load_file, host): host for host in state.inventory.iter_active_hosts()
                                                        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
RuntimeError: Set changed size during iteration
  • Consider including output with -vv and --debug.
    no other relevant informations seems added

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions