Tracealyzer on STM32MP1

[georgenyarangi-eaton]

I am trying to get the Tracealyzer to work with the STM32MP157-C target; however, I am running into issues. Specifically, I'm only trying to profile the M4 core. I have setup my Zephyr project and Tracealyzer configuration per the instructions in the Tracealyzer "Getting Started" page for Zephyr.

Other relevant setup details:

• Zephyr version = v3.0.0
• Tracealyzer version = 4.6.6.13115
• Debugger = SEGGER Jlink v8.0
• J-Link Software version = v7.84
• Tracealyzer mode = Streaming over RTT
• Kconfig options enabled = CONFIG_PERCEPIO_TRACERECORDER & CONFIG_TRACING
• Target boot mode = Engineering boot mode (i.e., Cortex-A7 is halted and only the Cortex-M4 is running)

I am able to "connect" to the target from in the Live Stream mode. However, when I "Start Session", I get "Session error: Failed to send stop command! RTT not connected?". See screenshot below. Any insight into what could be going on will be highly appreciated.

CC: @FRASTM @nashif @eriktamlin

[georgenyarangi-eaton]

@eriktamlin We have been able to get Tracealyzer to work. It seems there's an issue (linker errors) with the built-in hooks in some of the drivers.
We are using zephyr 3.0. It seems to me like this is a bug. Thoughts?

Build log showing linker errors:

FAILED: zephyr/zephyr_pre0.elf zephyr/zephyr_pre0.map C:/Firmware/Projects/zephyrproject/zephyr/build/zephyr/zephyr_pre0.map 
cmd.exe /C "cd . && C:\arm-embedded\gcc-arm-none-eabi-10.3-2021.07\bin\arm-none-eabi-g++.exe   zephyr/CMakeFiles/zephyr_pre0.dir/misc/empty_file.c.obj -o zephyr\zephyr_pre0.elf  zephyr/CMakeFiles/offsets.dir/./arch/arm/core/offsets/offsets.c.obj  -Wl,-T  zephyr/linker_zephyr_pre0.cmd  -Wl,-Map=C:/Firmware/Projects/zephyrproject/zephyr/build/zephyr/zephyr_pre0.map  -Wl,--whole-archive  app/libapp.a  zephyr/libzephyr.a  zephyr/arch/common/libarch__common.a  zephyr/arch/arch/arm/core/aarch32/libarch__arm__core__aarch32.a  zephyr/arch/arch/arm/core/aarch32/cortex_m/libarch__arm__core__aarch32__cortex_m.a  zephyr/arch/arch/arm/core/aarch32/mpu/libarch__arm__core__aarch32__mpu.a  zephyr/lib/libc/newlib/liblib__libc__newlib.a  zephyr/lib/posix/liblib__posix.a  zephyr/soc/arm/common/cortex_m/libsoc__arm__common__cortex_m.a  zephyr/drivers/interrupt_controller/libdrivers__interrupt_controller.a  zephyr/drivers/clock_control/libdrivers__clock_control.a  zephyr/drivers/console/libdrivers__console.a  zephyr/drivers/gpio/libdrivers__gpio.a  zephyr/drivers/ipm/libdrivers__ipm.a  zephyr/drivers/watchdog/libdrivers__watchdog.a  zephyr/drivers/serial/libdrivers__serial.a  zephyr/drivers/timer/libdrivers__timer.a  zephyr/drivers/pinctrl/libdrivers__pinctrl.a  modules/stm32/stm32cube/lib..__modules__hal__stm32__stm32cube.a  modules/libmetal/libmetal/lib/libmetal.a  modules/open-amp/open-amp/lib/libopen_amp.a  modules/segger/libmodules__segger.a  modules/TraceRecorder/libmodules__TraceRecorder.a  -Wl,--no-whole-archive  zephyr/kernel/libkernel.a  -L"c:/arm-embedded/gcc-arm-none-eabi-10.3-2021.07/bin/../lib/gcc/arm-none-eabi/10.3.1/thumb/v7e-m/nofp"  -LC:/Firmware/Projects/zephyrproject/zephyr/build/zephyr  -lgcc  zephyr/arch/common/libisr_tables.a  -no-pie  -mcpu=cortex-m4  -mthumb  -mabi=aapcs  -mfp16-format=ieee  -Wl,--gc-sections  -Wl,--build-id=none  -Wl,--sort-common=descending  -Wl,--sort-section=alignment  -Wl,-u,_OffsetAbsSyms  -Wl,-u,_ConfigAbsSyms  -nostdlib  -static  -Wl,-X  -Wl,-N  -Wl,--orphan-handling=warn  -lstdc++  -lm  -Wl,-lc  -L"C:/arm-embedded/gcc-arm-none-eabi-10.3-2021.07/arm-none-eabi"/lib/thumb/v7e-m/nofp  -Wl,-lgcc  -lc  -specs=nano.specs && cmd.exe /C "cd /D C:\Firmware\Projects\zephyrproject\zephyr\build\zephyr && "C:\Program Files\CMake\bin\cmake.exe" -E echo ""
c:/arm-embedded/gcc-arm-none-eabi-10.3-2021.07/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld.exe: app/libapp.a(Board_Manager.cpp.obj): in function `Board_Manager::Board_Manager()':
C:/Firmware/Projects/catalina_clean/Firmware/MMPlus_Gen2/MMPlus_BL/Code/Board_Manager.cpp:38: undefined reference to `sys_trace_syscall_enter(unsigned int, char const*)'
c:/arm-embedded/gcc-arm-none-eabi-10.3-2021.07/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld.exe: C:/Firmware/Projects/catalina_clean/Firmware/MMPlus_Gen2/MMPlus_BL/Code/Board_Manager.cpp:38: undefined reference to `sys_trace_syscall_exit(unsigned int, char const*)'
c:/arm-embedded/gcc-arm-none-eabi-10.3-2021.07/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld.exe: C:/Firmware/Projects/catalina_clean/Firmware/MMPlus_Gen2/MMPlus_BL/Code/Board_Manager.cpp:44: undefined reference to `sys_trace_syscall_enter(unsigned int, char const*)'
c:/arm-embedded/gcc-arm-none-eabi-10.3-2021.07/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld.exe: C:/Firmware/Projects/catalina_clean/Firmware/MMPlus_Gen2/MMPlus_BL/Code/Board_Manager.cpp:44: undefined reference to `sys_trace_syscall_exit(unsigned int, char const*)'
c:/arm-embedded/gcc-arm-none-eabi-10.3-2021.07/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld.exe: app/libapp.a(Watchdog.cpp.obj): in function `Watchdog::Watchdog(device const*, DSI_Pub*)':
C:/Firmware/Projects/catalina_clean/Firmware/MMPlus_Gen2/MMPlus_BL/Code/Watchdog.cpp:31: undefined reference to `sys_trace_syscall_enter(unsigned int, char const*)'
c:/arm-embedded/gcc-arm-none-eabi-10.3-2021.07/bin/../lib/gcc/arm-none-eabi/10.3.1/../../../../arm-none-eabi/bin/ld.exe: C:/Firmware/Projects/catalina_clean/Firmware/MMPlus_Gen2/MMPlus_BL/Code/Watchdog.cpp:31: undefined reference to `sys_trace_syscall_exit(unsigned int, char const*)'
collect2.exe: error: ld returned 1 exit status
ninja: build stopped: subcommand failed.
"C:/ProgramData/chocolatey/bin/ninja.exe all" terminated with exit code 1. Build might be incomplete.

14:59:24 Build Failed. 8 errors, 0 warnings. (took 4m:17s.475ms)

Sample line that causes linker error (constructor initializer list entry in Board_Manager.cpp line 44): watchdog_device_binding( device_get_binding( DT_LABEL( DT_NODELABEL( wwdg1 ) ) ) )
The WWDG1 device binding is declared as: const struct device* watchdog_device_binding;
DT entry in overlay file: &wwdg1 { status = "okay"; };

In general, it seems like the linker error has to do with device_get_binding()

CC: @scottbronkhorst-eaton

[eriktamlin]

OK.
I believe the main problem is that your project specific source files don't get the proper defines (CONFIG_PERCEPIO_TRACERECORDER). The kernel compiles successfully which means the kernel has those defines.
Something to test: "#define CONFIG_PERCEPIO_TRACERECORDER" before you include <device.h>
If that works, then that is the issue. Otherwise I'm a bit lost as to why the compiler can't see the trace hooks.
Provided that the above test is successful: Can you see where CONFIG_PERCEPIO_TRACERECORDER is defined in the project and see if you can include it before <device.h>? If your .cpp files don't have that define then it will not be able to access the trace hooks from the inlined kernel functions.

[georgenyarangi-eaton]

CONFIG_PERCEPIO_TRACERECORDER=y is set in the prj.conf file at the project level. I can try and move it but I don't think I can have it in both places because I may get a redefinition error. I'll try this and let you know.

[eriktamlin]

In which .h file is it generated? Some cfg file?

[eriktamlin]

You could also generate intermediate source files and see if tracing_tracerecorder.h is properly included from BoardManager.cpp before the trace hook attempt.

[VarunNiranjanNayak]

Hi Erik,

Thank you so much for your assistance so far regarding the Tracealyzer issues we've been experiencing.
As you're aware, we have been able to get past the linker errors we were seeing by using a workaround. With the workaround in place, we've been able to log traces in streaming mode; however, we've run into yet another issue.

It seems like our application is generating a lot of events and as a results we are seeing a lot of missed events in our captures (e.g., we record about 1M events but miss ~2M events). We have tried resolving this issue by filtering out events using the macros in trcConfig.h and increasing the RTT buffer (CONFIG_SEGGER_RTT_BUFFER_SIZE_UP) to 40 KB; however, none of these approaches seem to mitigate the missed events issue. Due to the large number of missed events, the data that we're logging doesn't seem accurate and we really cannot trust it for our analysis.

Do you have any thoughts on how we can go about resolving this missed events issue? Any help will be greatly appreciated. I have attached some snapshots to provide further detail.

Kind Regards
Varun