drivers: video: hm01b0 and hm0360: How to handle data bus 1,4 or 8 bits

[KurtE]

Summary

As I mentioned earlier in the two PRs to add the hm01b0 #94894 and hm0360 camera #94904.

Suppose I configure in the device tree to use an hm01b0 camera at resolution 324x324 to use the 8 bit
8 bit Grey format.

How many bytes will the drivers such as video_stm32_dcmi.c see from the camera per frame.

Sorry trick question: It depends on the HM01b0 property: data-bits, which can be 1, 4, or 8.

if set to 8, the answer is as expected: 324x324x1=104976 bytes

However if it is set to 4, which is needed to support the HM01b0 cameras sold by Arduino and
Arducam for the Arduino GIGA and the others with the 20 pin connector the actual number of bytes
returned from the camera is: 324x324x2=209952 bytes

And if you use the default value of 1, which I believe is used by the RPI Pico it is 324*324*8=839808 bytes

Currently the hardware like the DCMI on some of the STM32 boards has no knowledge of this camera
setting and for example if one uses the current default val frame of data. And each of these bytes will only have 1 actual
data bit contained in them.

Question is, How does DCMI (or other such hardware) discover how many data bits per pixel clock will be
returned by a camera? And how should this be handled?

For example on the video_stm32_dcmi code, what should it know about this? There are different options:
a) The code that determines the sizes of buffers needed, would find this information out and increase the
buffer size. The hardware code simply returns the data as returned by the camera and it is up to the
app code to deal with it.

b) Again the code somehow detects we are in one of these modes, and knows to receive the full buffer size
When the code like video_stm32_dcmi receives the buffer, and it currently memcpy it to another buffer
it could instead, compress back to 8 bits of data per byte. Note: This might imply that the camera buffer
might be a different size than the ones the applications use.

c) Some hardware drivers can handle 1 or 4 bits per pixel clock, and they simply need to have their setting
configured for it. like the RPI Pico code. Also some of the IMXRT code using FlexIO could be configured.

But in all of these cases, the first question is, how do the Hardware Video bus drivers know that the camera
is not returning 8 bits of data per clock?

Describe the solution you'd like

Not sure:
First part probably need a standard way to determine how many bits of data is returned by the camera
per pixel clock.

Then need to know if this can be handled transparently between the camera and the bus driver.
And then decide what to do with buffer sizes and who does the work to compress the data back.

Alternatives

Earlier when I was testing #94894 on an Arduino GIGA and Portenta H7, I hacked it up by adding a new pixel format:
VIDEO_PIX_FMT_GREY_4

Where if the app asked for it, the code to compute buffer sizes would say that it is 16 bits, which would compute
a buffer large enough and then the test sketch would rebuild the data.

Additional Context

No response

[avolmat-st]

The proper way to address this would be to rely on the bindings bus-width which is meant for this purpose, describing the width of the parallel data bus. It is also possible to mention if there is a shift (aka for example not using 0-7 but instead using 2-9 for example). This is the bindings that drivers are supposed to check in order to set the hardware in order to fetch the right number of bits each cycles.

The issue here, since you mention DCMI is that, the DCMI only support 8/10/12 bits modes. No < 8 support.

The reason why you have not the correct frame size captured is "normal" (considering that DCMI will capture in 8bit). When the sensor sends data on 1 bit only ot 4 bits only, it will finally multiply by 2 or by 8 the number of cycles in order to send the data to the receiver.
So, whenever the sensor tries to sent 4 bit (in 1 cycle), the DCMI will fetch 8bit (from the bus) and store 8 bits into the memory (via its FIFO). So in order to send 8bits, the sensor will do 2 cycles of 4bits, and thus DCMI will fetch 8 bit twice, hence 2 bytes.
(same thing for the 1 bit more).

As you mentioned, considering that currently the DCMI is relying on a temporary buffer, adjustment of the data, (aka only taking 4 bits or 1 bits out of the 8bits captured) could be done, but this is going to be very very time consuming, leading to getting quite bad performances after all.

[KurtE]

Thanks @avolmat-st and all,

As the main HM01b0 camera sold by Arduino/Arducam for the Arduino Giga
https://www.arducam.com/hm01b0-qvga-monochrome-dvp-camera-module-for-arduino-giga-r1-wifi-board.html
does not support 8 bit mode, as for whatever reason they only routed 4 data bits to the 20 bit connector and it
works on the Arduino MBED version of code, I believe it would be good to get something to work.

As I mentioned on the Alternatives in the first post, I was handling this in the pre-released HM01b0 driver by adding a new
format: VIDEO_PIX_FMT_GREY_44, which is an option, which worked.

But maybe I should try a different approach, and update the STM32_DCMI driver to allow you to specify 4 bit, and then
have the code that computes the number of bytes to take that into account. Will try that out next.

As you mentioned, considering that currently the DCMI is relying on a temporary buffer, adjustment of the data, (aka only taking 4 bits or 1 bits out of the 8bits captured) could be done, but this is going to be very very time consuming, leading to getting quite bad performances after all.

I understand what you are saying, it doubles the number of bytes transferred. Or the same number of bytes as other cameras
do for RGB565 mode. That it can impact performance.

Side note: When PJRC was working with Sparkfun to create the Teensy
Micromod, Paul sent of us including @mjs513 and myself, a Carrier board he created to make it easier to test, on it,
he added connectors for the HM01b0, one with 4 bits and the other with 8 bits:

Sorry for the image quality. With it several of us experimented with a camera library to support it on the different Teensy 4.x boards. With T4.1 we used CSI mode as enough of the IO pins were exported to support 8 bit mode. It too was limited to 8, 10...
mode. On the Micromod, there were not enough CSI pins exported, but there were enough FlexIO pins, so we setup the driver
on these boards for FlexIO and we supported either 4 bit or 8 bit mode. And several preferred using it in 4 bit mode as the
speed was fast enough and it allowed them to use the other 4 IOPins for other things.

I believe it is also possible to drive the Pixel clock at a higher speed in 4 bit mode, as they show in the PDF that they can both do 60FPS

Will see what happens

[avolmat-st]

I am not that concerned about the number of cycles needed in order to transmit the data but the amount of CPU that will be needed to copy one byte at a time doing bit shifting. Currently, there is a memcpy, but even if amount of bytes might be identical, in one case that would most probably lead to 4 bytes aligned copy while on the other end you'd have to copy one byte at a type, then doing bit shifting on the data.

[KurtE]

Thanks, I understand what you are saying and I hear you. With that there were two (at least) options
a) yes the memcpy area did the repacking of data. (At least in video mode). Could it it in snapshot as well but...
b) It is up to the app to know that the data is split 4 bits per byte and deal with it.

For an experiment, I tried setting:

	port {
		dcmi_ep_in: endpoint {
			//remote-endpoint-label = "gc2145_ep_out";
			remote-endpoint-label = "hm01b0_ep_out";
			bus-width = <4>;
			hsync-active = <0>;
			vsync-active = <0>;
			pclk-sample = <0>;
		};
	};

And it made no differences in the run, as the only place that looks at it:

#define STM32_DCMI_GET_BUS_WIDTH(bus_width)						\
	((bus_width) == 8 ? DCMI_EXTEND_DATA_8B :					\
	(bus_width) == 10 ? DCMI_EXTEND_DATA_10B :					\
	(bus_width) == 12 ? DCMI_EXTEND_DATA_12B :					\
	(bus_width) == 14 ? DCMI_EXTEND_DATA_14B :					\
	DCMI_EXTEND_DATA_8B)

..
				.ExtendedDataMode = STM32_DCMI_GET_BUS_WIDTH(
							DT_PROP_OR(DT_INST_ENDPOINT_BY_ID(0, 0, 0),
								   bus_width, 8)),

So it still tells the hardware that it is 8 bits.

But then you run into the next issue as:
ret = video_estimate_fmt_size(fmt);
Assumes that it is packed.... it divides by BITS_PER_BYTE which I did not find defined but assume that is hard coded to 8...

So maybe it would be best to go back to the KISS solution I did earlier.
Define a new format:
Changes in video.h

/**
 * Same as Y8 (8-bit luma-only) following the standard FOURCC naming,
 * or L8 in some graphics libraries.
 *
 * @code{.unparsed}
 *   0          1          2          3
 * | Yyyyyyyy | Yyyyyyyy | Yyyyyyyy | Yyyyyyyy | ...
 * @endcode
 */
#define VIDEO_PIX_FMT_GREY VIDEO_FOURCC('G', 'R', 'E', 'Y')

/**
 * @code{.unparsed}
 *   0          1          2          3          3 2 1 0
 * | Yyyyyyyy | Yyyyyyyy | Yyyyyyyy | Yyyyyyyy | yyyyyyyy | ...
 * @endcode
 */
#define VIDEO_PIX_FMT_GREY_44 VIDEO_FOURCC('G', 'R', '4', '4')
,,,
	case VIDEO_PIX_FMT_GREY_44:		
		return 16;

Add entries into HM01b0 (and HM0360 ?) like:

	HM01B0_VIDEO_FORMAT_CAP(326, 244, VIDEO_PIX_FMT_GREY_44),

And then the apps/sketches that wish to handle it do so. This is probably the least disruptive way

[avolmat-st]

I found some reference to such 4bit GREY format, such as here: https://www.kernel.org/doc/html/v6.17-rc3/userspace-api/media/v4l/pixfmt-reserved.html

V4L2_PIX_FMT_Y4, so in our case we would code this as:

#define VIDEO_PIX_FMT_Y4      VIDEO_FOURCC('Y', '0', '4', ' ') /*  4  Greyscale     */

This is defined as 4 bit greyscale with 4 bit significant into each 8bit. So basically, it would be referenced as 8bpp in order to perform the buffer allocation, and this would be up to the application to perform the "packing" if necessary.

If some HW can already dump this as 4bit only, this could be something like Y4P (as packed) ?

[mjs513]

@avolmat-st - @KurtE
As @KurtE is aware spent the day yesterday testing 4-bit mode using VIDEO_PIX_FMT_GREY_44 on both the HM01B0 and HM0360 (yes I updated it use 1, 4, or 8 bits) using the ArduinoCore framework. Was able to capture single images from both cameras:
HM0360

HM01B0

Interesting thing happens when I try to use VIDEO_PIX_FMT_Y4. In the arduino camera library I have:

      case CAMERA_GRAYSCALE_4:
          // There's no support for mono sensors.
          this->yuv_to_gray = false;
          pixformat = VIDEO_PIX_FMT_GREY_44;
          break;

and if I try pixformat = VIDEO_PIX_FMT_Y4 fails to compile with the following error

error: 'VIDEO_PIX_FMT_Y4' was not declared in this scope; did you mean 'VIDEO_PIX_FMT_Y14'?

[KurtE]

@mjs513 @avolmat-st @josuah @iabdalkader

Couple of side note: Neither CAMERA_GRAYSCALE_4 nor VIDEO_PIX_FMT_Y4 exist in the released video.h the CAMERA_GRAYSCALE_4 I added earlier when we were experimenting with the at that time un-merged HM01b0 code.

I am in the process of adding it in current draft PR: #98154
https://github.com/KurtE/zephyr/blob/hm01b0_fixes/include/zephyr/drivers/video.h#L1342-L1348

There is more to be done, but thinking maybe I will mark it as to be reviewed, maybe later today, want to do some more
testing, mostly on Zephyr:

Now the fun of testing on Arduino:
@mjs513 - I know that you are fully aware of the fun, but thought I would mention what we typically do, especially since we
mostly test on Windows.

You need a copy of the updated video.h into your WSL fork of ArduinoCore-zepyr/zephyr on my machine that is at:
\wsl.localhost\Ubuntu-24.04\home\kurte\git\zephyr\include\zephyr\drivers\video.h
However you can not copy it directly there, as the ArduinoCore-zephyr code is based off of:
projects:

    - name: zephyr
      remote: arduino
      revision: zephyr-arduino-v4.2.0

And the hm01b0 and other video parts are off of later rebases, where things have changed in structures and the like, so
you need to edit in that define and add it to the switch statement...

Copy/Edit in the hm01b0, plus some related files, CMake KConfig...

Now hopefully you can then make it for the boards you are interested in, in my case I did:
(After editing the variants to add in the camera information that you wish to test for)

./extra/build.sh giga
./extra/build.sh portentah7

or you could also use the command: ./extra/build_all.sh

Then you need to copy the built stuff back to Windows area. I have a couple of scripts that I use depending on if
I am only worrying about all or just the two above. My faster version looks like:

rsync -av --update --exclude 'cores/arduino/api' /home/kurte/git/ArduinoCore-zephyr/cores /mnt/d/github/ArduinoCore-zephyr
rsync -av --update /home/kurte/git/ArduinoCore-zephyr/loader /mnt/d/github/ArduinoCore-zephyr
rsync -av --update /home/kurte/git/ArduinoCore-zephyr/firmwares /mnt/d/github/ArduinoCore-zephyr
#rsync -av --update /home/kurte/git/ArduinoCore-zephyr/variants /mnt/d/github/ArduinoCore-zephyr
rsync -av --update /home/kurte/git/ArduinoCore-zephyr/variants/arduino_giga_r1_stm32h747xx_m7 /mnt/d/github/ArduinoCore-zephyr/variants
rsync -av --update /home/kurte/git/ArduinoCore-zephyr/variants/arduino_portenta_h7_stm32h747xx_m7 /mnt/d/github/ArduinoCore-zephyr/variants
rsync -av --update /home/kurte/git/ArduinoCore-zephyr/variants/arduino_portenta_c33_r7fa6m5bh3cfc /mnt/d/github/ArduinoCore-zephyr/variants
rsync -av --update /home/kurte/git/ArduinoCore-zephyr/variants/arduino_nano_33_ble_nrf52840_sense /mnt/d/github/ArduinoCore-zephyr/variants

rsync -av --update /home/kurte/git/ArduinoCore-zephyr/libraries /mnt/d/github/ArduinoCore-zephyr

After most of the stuff has been copied up, then you can get back to the windows side and do things like:
Update camera library to add in type for 4 bit mode (and use our own version that has deferred camera start up code.
(As neither PRs have been pulled in that enable camera on H7...)

Now flash the board, compile your sketch and hopefully it works, but if you need to change anything in the driver or
camera configuration, repeat most of these steps.

Sorry for the long answer to a short question.

[mjs513]

@KurtE

Thanks. Just figured it out after 2nd cup of coffee..... just had to update the variants as well as the firmware