Don't use EPMD.

[mhanberg]

We shouldn't need to use EPMD for clustering as we are always aware of the nodes.

@josevalim has had similar problems with livebook related to EPMD and has fixed them with a epmdless style approach.

We should be able to leverage that approach in expert.

Should help with problems like the sub issues

[josevalim]

Perhaps the steps here will be enough. You start the main one with -start_epmd false -erl_epmd_port 6789 and each individual instance you start with -start_epmd false -erl_epmd_port 6789 -dist_listen false:

https://hexdocs.pm/mix/Mix.Tasks.Release.html#module-epmd-less-deployment

[mhanberg]

@josevalim do you know if this should work when many instances of the application? My intuition is telling me that the erl_epmd_port might need to be unique for each instance

[josevalim]

Yes, it should. The parent node is listening on port 6789 and each child node connects to port 6789 but does not listen to any (-dist_listen false). This requires, however, for the child node to be the one connecting to the parent.

[josevalim]

Oh, sorry, I believe I have misunderstood your question. You want to run multiple Expert servers, not multiple clients, then you are right!

In order to do that, you can write this code:

defmodule Expert.EPMD do
  def dist_port do
    :persistent_term.get(:expert_dist_port, nil)
  end

  # EPMD callbacks

  def register_node(name, port), do: register_node(name, port, :inet)

  def register_node(name, port, family) do
    :persistent_term.put(:expert_dist_port, port)

    # We don't care if EPMD is not running
    case :erl_epmd.register_node(name, port, family) do
      {:error, _} -> {:ok, -1}
      {:ok, _} = ok -> ok
    end
  end

  defdelegate start_link(), to: :erl_epmd
  defdelegate port_please(name, host), to: :erl_epmd
  defdelegate port_please(name, host, timeout), to: :erl_epmd
  defdelegate listen_port_please(name, host), to: :erl_epmd
  defdelegate address_please(name, host, family), to: :erl_epmd
  defdelegate names(host_name), to: :erl_epmd
end

This is basically a wrapper around EPMD so you get to know which port it is running on. Now, in your vm.args, you will set this: -epmd_module Elixir.Expert.EPMD.

Finally, when starting the clients, you will do this:

if port = Expert.EPMD.dist_port() do
  start_client("-start_epmd false -erl_epmd_port #{port} -dist_listen false")
else
  # start as today
end

[mhanberg]

@josevalim do you know if the method you described here (which uses the :erl_epmd module) is still "epmd-less" ?

I want to confirm that the :erl_epmd module does not need epmd itself.

[josevalim]

The clients no longer talk to the server via epmd, they connect directly to the port. I think the server still tries to register itself in epmd (but I think it still runs if it can’t register).

[josevalim]

That was a lie. I had to push one small change to the code above (now edited). Now it works. Here are the receipts.

Save the Expert.EPMD above to epmdless.ex and run this command:

$ killall epmd
$ elixirc epmdless.ex
$ iex --erl "-start_epmd false -epmd_module Elixir.Expert.EPMD" --sname foo
Erlang/OTP 27 [erts-15.1.2] [source] [64-bit] [smp:10:10] [ds:10:10:10] [async-threads:1] [jit]

Interactive Elixir (1.20.0-dev) - press Ctrl+C to exit (type h() ENTER for help)
iex(foo@macstudio)1> Expert.EPMD.dist_port
65378
iex(foo@macstudio)2>

Keep the node running but take note of the port above (you will get a different number). Now in another terminal, using the PORT from IEx above:

$ iex  --erl "-start_epmd false -erl_epmd_port PORT -dist_listen false" --sname bar

They should connect! To prove it doesn't use epmd, in another terminal:

$ epmd -names
epmd: Cannot connect to local epmd

[josevalim]

In practice, you only need to start the Expert server passing -epmd_module Elixir.Expert.EPMD in vm.args. The current implementation tries to register itself in epmd but, if it cannot find anything, it simply does not care. But you can also use "-start_epmd false -epmd_module Elixir.Expert.EPMD".

But the client node always needs the three flags above: -start_epmd false -erl_epmd_port PORT -dist_listen false.

I will be glad to jump into a call if you folks need me to.

[mhanberg]

Thanks José, I'll try this out.