feat: epmdless clustering #205

doorgan · 2025-11-10T22:12:56Z

Alternative to #191 without introducing dependencies.

TODO:

Figure out why the engine node thinks Forge.EPMD does not exist.

mhanberg · 2025-11-11T15:06:00Z

apps/expert/rel/remote.vm.args.eex

+## Enable deployment without epmd
+## (requires changing both vm.args and remote.vm.args)
+-epmd_module Elixir.XPForge.EPMD
+-start_epmd false -dist_listen false


I don't think we need the -dist_listen false anymore

Yeah, this is a leftover of me splitting hairs

mhanberg · 2025-11-11T15:08:11Z

Figure out why the engine node thinks Forge.EPMD does not exist.

Are you seeing this in tests or when you actually put it up?

doorgan · 2025-11-11T15:14:10Z

@mhanberg both in tests and when building a release, I'm getting this error:

Kernel pid terminated (application_controller) ("{application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,'Elixir.Forge.EPMD',{'EXIT',{undef,[{'Elixir.Forge.EPMD',start_link,[],[]},{supervisor,do_start_child_i,3,[{file,\"supervisor.erl\"},{line,959}]},{supervisor,do_start_child,3,[{file,\"supervisor.erl\"},{line,945}]},{supervisor,'-start_children/2-fun-0-',3,[{file,\"supervisor.erl\"},{line,929}]},{supervisor,children_map,4,[{file,\"supervisor.erl\"},{line,1820}]},{supervisor,init_children,2,[{file,\"supervisor.erl\"},{line,889}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,2229}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,2184}]}]}}}}}},{kernel,start,[normal,[]]}}}")

mhanberg · 2025-11-11T15:25:10Z

@mhanberg both in tests and when building a release, I'm getting this error:

Kernel pid terminated (application_controller) ("{application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,'Elixir.Forge.EPMD',{'EXIT',{undef,[{'Elixir.Forge.EPMD',start_link,[],[]},{supervisor,do_start_child_i,3,[{file,\"supervisor.erl\"},{line,959}]},{supervisor,do_start_child,3,[{file,\"supervisor.erl\"},{line,945}]},{supervisor,'-start_children/2-fun-0-',3,[{file,\"supervisor.erl\"},{line,929}]},{supervisor,children_map,4,[{file,\"supervisor.erl\"},{line,1820}]},{supervisor,init_children,2,[{file,\"supervisor.erl\"},{line,889}]},{gen_server,init_it,2,[{file,\"gen_server.erl\"},{line,2229}]},{gen_server,init_it,6,[{file,\"gen_server.erl\"},{line,2184}]}]}}}}}},{kernel,start,[normal,[]]}}}")

What stands out to me is that the module name is not namespaced.

mhanberg · 2025-11-11T15:25:42Z

apps/engine/lib/engine/bootstrap.ex

  require Logger

  def init(%Project{} = project, document_store_entropy, app_configs) do
+    Application.put_env(:kernel, :epmd_module, Forge.EPMD, persistent: true)


Do you know why this is necessary?

I't not or it should not be, I'm trying out things from the Livebook codebase
It seems like the vm tries to find the EPMD module before loading the code paths provided via -pa
Here in particular I was trying to not specify the epmd module with erl flags and instead try to change the epmd module after the engine node starts. This line wasn't reached because the rpc command to bootstrap didn't get through, but this in the -e flag doesn't help either

apps/expert/lib/expert/engine_node.ex

doorgan · 2025-11-11T15:32:50Z

What stands out to me is that the module name is not namespaced.

Ah, that log is from running tests, where namespacing doesn't run
If I build a release I get this in the lsp logs:

[ERROR][2025-11-10 19:03:47] ...p/_transport.lua:36     "rpc"   "/Users/dorgan/dev/expert/apps/expert/burrito_out/expert_darwin_arm64"  "stderr"        "plication_controller) (\"{application_start_failure,kernel,{{shutdown,{failed_to_start_child,net_sup,{shutdown,{failed_to_start_child,'Elixir.XPForge.EPMD',{'EXIT',{undef,[{'Elixir.XPForge.EPMD',start_link,[],[]},{supervisor,do_start_child_i,3,[{file,\\\"supervisor.erl\\\"},{line,420}]},{supervisor,do_start_child,2,[{file,\\\"supervisor.erl\\\"},{line,406}]},{supervis"

doorgan · 2025-11-11T22:28:43Z

I can reproduce the same issues on a barebones mix project: https://github.com/doorgan/epmdless

It does partially work if I follow @josevalim's example and run elixirc on a file containing both modules(although I get the requested disconnect from node in order to prevent overlapping partitions warning and children can't connect to each other), but it breaks inside a mix project

mhanberg · 2025-11-12T15:25:07Z

I haven't researched further to validate and find a solution, but I believe the reason this is happening in the engine but not in expert is because expert is started as a release, which i believe by default loads all modules before the app starts.

Normal mix run or mix phx.server or iex -S mix invocations will lazily load modules as they are started, which is why it isn't present when it tries to utilize the -epmd_module. Which it seems to not load it when it tries to invoke it i guess.

Relevant section from the release docs at https://hexdocs.pm/elixir/releases.html#operating-system-scripts

# # Set the release to load code on demand (interactive) instead of preloading (embedded).
# export RELEASE_MODE=interactive

josevalim · 2025-11-12T16:31:15Z

I can take a look later but in Livebook we don't start the node in the release, we explicitly call :net_kernel.start or similar to start them. It may be for the reasons above. i will investigate later.

doorgan · 2025-11-14T15:20:46Z

@josevalim thanks for all the help! This PR is now ready for testing and review

doorgan · 2025-11-14T15:26:11Z

justfile

+              (cd "apps/$proj" && elixir --erl "-start_epmd false -epmd_module Elixir.Forge.EPMD" -S mix {{args}})
+            ;;
+            engine)
+              (cd "apps/$proj" && elixir --erl "-start_epmd false -epmd_module Elixir.Forge.EPMD" -S mix {{args}})


I think it's still useful to keep them separate even if they're the same right now but I don't feel strongly about it

josevalim · 2025-11-15T16:23:19Z

apps/expert/lib/expert/engine_node.ex

+            you must set the environment variable ELIXIR_ERL_OPTIONS="-epmd_module #{Forge.EPMD}"
+            """)
+          end
+      end


Should we do this when expert boots rather than when we try to start a child node?

josevalim · 2025-11-15T16:24:56Z

apps/expert/rel/remote.vm.args.eex

+
+## Enable deployment without epmd
+## (requires changing both vm.args and remote.vm.args)
+-epmd_module Elixir.XPForge.EPMD


Please double check this is indeed necessary. You may be fine using epmd for remote debugging (or maybe remote debugging is pointless).

josevalim

Looks good to me! I have added some comments but none of them are stoppers (and they might be wrong :D)

mhanberg

Works on my corporate macbook and on my personal linux desktop

doorgan force-pushed the doorgan/epmdless-2 branch 2 times, most recently from fccf784 to 4bffe9b Compare November 11, 2025 02:29

mhanberg reviewed Nov 11, 2025

View reviewed changes

apps/expert/lib/expert/engine_node.ex Show resolved Hide resolved

doorgan force-pushed the doorgan/epmdless-2 branch from 4bffe9b to bb5c93b Compare November 11, 2025 19:14

mhanberg mentioned this pull request Nov 12, 2025

feat: cluster without epmd (epmdless) #191

Closed

doorgan force-pushed the doorgan/epmdless-2 branch from a4d4585 to d3db3c4 Compare November 14, 2025 15:07

feat: epmdless clustering

ab64607

doorgan force-pushed the doorgan/epmdless-2 branch from d3db3c4 to ab64607 Compare November 14, 2025 15:14

doorgan marked this pull request as ready for review November 14, 2025 15:20

doorgan commented Nov 14, 2025

View reviewed changes

docs: update epmd/distribution docs

bd3839a

josevalim reviewed Nov 15, 2025

View reviewed changes

josevalim approved these changes Nov 15, 2025

View reviewed changes

mhanberg approved these changes Nov 15, 2025

View reviewed changes

fix: check for epmd on expert startup

c705958

doorgan force-pushed the doorgan/epmdless-2 branch from 4a6b4d7 to c705958 Compare November 15, 2025 17:25

doorgan merged commit 488d3a9 into main Nov 15, 2025
36 checks passed

doorgan deleted the doorgan/epmdless-2 branch November 15, 2025 17:31

doorgan mentioned this pull request Nov 15, 2025

Don't use EPMD. #108

Closed

github-actions bot mentioned this pull request Nov 12, 2025

chore(main): release 0.1.0 #38

Open

This was referenced Nov 15, 2025

** System NOT running to use fully qualified hostnames ** ** Hostname 127.0.0.1 is illegal ** #176

Closed

Error using LSP: Erlang error: {:erpc, :noconnection} #59

Closed

FunctionClauseError on XPExpert.EngineApi.format/2 #124

Closed

This was referenced Nov 15, 2025

textDocument/hover from Phoenix Deps doesn't show #183

Closed

Fixing a warning is not removing the curly line #212

Closed

Uh oh!

feat: epmdless clustering #205

feat: epmdless clustering #205

Conversation

doorgan commented Nov 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mhanberg Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

doorgan Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

mhanberg commented Nov 11, 2025

Uh oh!

doorgan commented Nov 11, 2025

Uh oh!

mhanberg commented Nov 11, 2025

Uh oh!

mhanberg Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

doorgan Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

doorgan commented Nov 11, 2025

Uh oh!

doorgan commented Nov 11, 2025

Uh oh!

mhanberg commented Nov 12, 2025

Uh oh!

josevalim commented Nov 12, 2025

Uh oh!

doorgan commented Nov 14, 2025

Uh oh!

doorgan Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

josevalim Nov 15, 2025

Choose a reason for hiding this comment

Uh oh!

josevalim Nov 15, 2025

Choose a reason for hiding this comment

Uh oh!

josevalim left a comment

Choose a reason for hiding this comment

Uh oh!

mhanberg left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

4 participants

doorgan commented Nov 10, 2025 •

edited

Loading