How can I better detect and handle server errors

[stormyweathers]

Hey Jan,

First off, thanks for your work in maintaining this library. The code and extensive discussions around it have been very helpful to me over the last year.

[Pymodbus 3.8.6]

Lately, I've been developing a modbus server device, and encountered issues when trying to use it on a multi-drop bus. I encountered Issue #2556, where you mentioned this is not supported behavior. Ok, I think I understand why this would be hard to support, and I have since moved my server to a dedicated network to avoid this issue.

However, this error did not cause my program to crash, instead I only noticed by the stacktrace printed in the logs, and then my server refusing communication afterwards.

handle: <Handle SerialTransport.intern_read_ready() created at /usr/lib/python3.9/asyncio/selector_events.py:259>
source_traceback: Object created at (most recent call last):
  File "/home/pi/PiLogger/controller-development/water_mfc.py", line 486, in <module>
    asyncio.run(mfc.run_async_server(), debug=True)
  File "/usr/lib/python3.9/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.9/asyncio/base_events.py", line 629, in run_until_complete
    self.run_forever()
  File "/usr/lib/python3.9/asyncio/base_events.py", line 596, in run_forever
    self._run_once()
  File "/usr/lib/python3.9/asyncio/base_events.py", line 1882, in _run_once
    handle._run()
  File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/transport/serialtransport.py", line 44, in setup
    self.async_loop.add_reader(self.sync_serial.fileno(), self.intern_read_ready)
  File "/usr/lib/python3.9/asyncio/selector_events.py", line 336, in add_reader
    self._add_reader(fd, callback, *args)
  File "/usr/lib/python3.9/asyncio/selector_events.py", line 259, in _add_reader
    handle = events.Handle(callback, args, self, None)
Traceback (most recent call last):
  File "/usr/lib/python3.9/asyncio/events.py", line 80, in _run
    self._context.run(self._callback, *self._args)
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/transport/serialtransport.py", line 134, in intern_read_ready
    self.intern_protocol.data_received(data)  # type: ignore[attr-defined]
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/transport/transport.py", line 304, in data_received
    self.datagram_received(data, None)
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/transport/transport.py", line 338, in datagram_received
    cut = self.callback_data(self.recv_buffer, addr=addr)
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/server/requesthandler.py", line 79, in callback_data
    used_len = super().callback_data(data, addr)
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/transaction/transaction.py", line 187, in callback_data
    used_len, pdu = self.framer.processIncomingFrame(self.trace_packet(False, data))
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/framer/base.py", line 76, in processIncomingFrame
    data_len, pdu = self._processIncomingFrame(data[used_len:])
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/framer/base.py", line 97, in _processIncomingFrame
    if (result := self.decoder.decode(frame_data)) is None:
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/pdu/decoders.py", line 112, in decode
    pdu.decode(frame[1:])
  File "/home/pi/venv-new/lib/python3.9/site-packages/pymodbus/pdu/register_message.py", line 26, in decode
    self.address, self.count = struct.unpack(">HH", data)
struct.error: unpack requires a buffer of 4 bytes

I am concerned that some other future error could silently break my program.
But since the exception is handled somewhere deep (perhaps the asyncio library, perhaps pymodbus?), I am unable to handle it myself.

I am starting the server using an adaptation of one of the examples:

asyncio.run(mfc.run_async_server(), debug=True)

where mfc has a method:

    async def run_async_server(self) -> None:
        txt = f"### start ASYNC server, listening on serial: {self.port}"
        _logger.info(txt)
        # socat -d -d PTY,link=/tmp/ptyp0,raw,echo=0,ispeed=9600
        #             PTY,link=/tmp/ttyp0,raw,echo=0,ospeed=9600
        await StartAsyncSerialServer(
            context=self.server_context,  # Data storage
            identity=self.identity,  # server identify
            # timeout=1,  # waiting time for request to complete
            port=self.port,  # serial port
            # custom_functions=[],  # allow custom handling
            framer=self.framer,  # The framer strategy to use
            # stopbits=1,  # The number of stop bits to use
            # bytesize=8,  # The bytesize of the serial messages
            # parity="N",  # Which kind of parity to use
            baudrate=self.baudrate,  # The baud rate to use for the serial device
            # handle_local_echo=False,  # Handle local echo of the USB-to-RS485 adaptor
            ignore_missing_slaves=True,  # ignore request to a missing slave
            # broadcast_enable=False,  # treat slave 0 as broadcast address,
        )
        return

Could you offer some advice on this?
One option would be to add a special logging handler to detect this issue, i.e.:

    def emit(self, record):
        global critical_error_occurred
        log_entry = self.format(record)

        # Checking for specific error messages in the logs
        if "unpack requires a buffer of 4 bytes" in log_entry:
            critical_error_occurred = True

This feels wrong to me, but I don't know a better way.
I'd rather "un-handle" the exception, so the program crashes, and can then be restarted,