How do I setup and use the xTelemetryCallback for ESP32 RTOS so I can track messages delivered or not.

[wreyford]

I'm having difficulty figuring out how to implement the callback functionality of the AzureIoTHubClient, so that I can mark a record as sent to Azure form the ESP32S3 once successfully delivered. In the example code, (I assume for simplicity) this has not been implemented.

Due to timing constraints, with the ESP32S3, I'm generating more Records, than actually eventually end up being sent.
I thus need a way to sync these unsent messages when the device is not busy collecting data.

The AzureIoTHubClient_t xAzureIoTHubClient has a AzureIoTTelemetryAckCallback_t xTelemetryCallback;

I need some pointers on how to write this, and how to enable it.

I send my telemetry record with QOS1:

xResult = AzureIoTHubClient_SendTelemetry(&xAzureIoTHubClient, telemetry_record.ucTelemetryBuffer, telemetry_record.TelemetryLength, &xPropertyBag,  eAzureIoTHubMessageQoS1, NULL);

I see that it can return a pointer to the uint16_t * pusTelemetryPacketID

According to the definition of AzureIoTHubClient_SendTelemetry

param[out] pusTelemetryPacketID The packet id for the sent telemetry.
Can be notified of PUBACK for QOS 1 using the #AzureIoTHubClientOptions_t xTelemetryCallback option.
If xQOS is eAzureIoTHubMessageQoS0 this value will not be sent on return.
Can be NULL.

I thin I may need to set the callbackroutine here:

/* Init IoT Hub option */
		ESP_LOGI(TAG, "PNP-Init Hub Options"); 
		xResult = AzureIoTHubClient_OptionsInit(&xHubOptions);
		configASSERT(xResult == eAzureIoTSuccess);

		xHubOptions.pucModuleID = (const uint8_t *)loadassistazureMODULE_ID;
		xHubOptions.ulModuleIDLength = sizeof(loadassistazureMODULE_ID) - 1;
		xHubOptions.pucModelID = (const uint8_t *)loadassistazureiotMODEL_ID;
		xHubOptions.ulModelIDLength = sizeof(loadassistazureiotMODEL_ID) - 1;

		xHubOptions.xTelemetryCallback  =  **??????**

		
#ifdef democonfigPNP_COMPONENTS_LIST_LENGTH
#if democonfigPNP_COMPONENTS_LIST_LENGTH > 0
		xHubOptions.pxComponentList = democonfigPNP_COMPONENTS_LIST;
		xHubOptions.ulComponentListLength = democonfigPNP_COMPONENTS_LIST_LENGTH;

Any pointers on HOW - TO get started with implementing a callback, that can run client side code to mark a telemetry record as delivered will be appreciated.
Thx

[wreyford]

I tried by creating a callback function:

void loadassist_azure_telemetry_cb( uint16_t ulTelemetryPacketID ){ ESP_LOGW(TAG, "We successfully uploaded Telemetry and received TelemetryPacketID: %d", ulTelemetryPacketID); }

It simply logs the Packet ID received.

I added the callback to the xHubOptions:

xHubOptions.xTelemetryCallback = loadassist_azure_telemetry_cb;

I added send_telemetry as follows:

xResult = AzureIoTHubClient_SendTelemetry(&xAzureIoTHubClient, telemetry_record.ucTelemetryBuffer, telemetry_record.TelemetryLength, &xPropertyBag, eAzureIoTHubMessageQoS1, &telemetryPacketID);

I defined telemetryPacketID (globally) as:

uint16_t telemetryPacketID = 32;

Irrespective of the value I set it to before calling AzureIoTHubClient_SendTelemetry, it sends back:
3 after the first telemetry sent, and then sequentially increments.
If I reconnect it starts at 3 again.
so 3,4,5,6...

I need to get a telemetry message IDENTIFIER somehow

I'm obviously misunderstanding something critical here.
Can I control the value of the TelemetryPacketID, so that I can sensibly identify the telemetry message that was actually sent?
It appears to be a return parameter only, so where must I set it before the telemetry is sent?

Must I add this to the PropertiesBag of the Telemetry, and If so what must the precise format be?

Some progress made, but still not there....

[wreyford]

I was hoping I could send the message-Id by adding it as a property entry to the PropertyBag:

xResult = AzureIoTMessage_PropertiesAppend(&xPropertyBag,
			(uint8_t *)"$.ct", sizeof("$.ct") - 1,
			(uint8_t *)"application%2Fjson",
			sizeof("application%2Fjson") - 1);
	         configASSERT(xResult == eAzureIoTSuccess);

xResult = AzureIoTMessage_PropertiesAppend(&xPropertyBag,
			(uint8_t *)"$.message-id", sizeof("$.message-id") - 1,
			(uint8_t *)"2022",
			sizeof("2022") - 1);
		configASSERT(xResult == eAzureIoTSuccess);

This however only resulted in me adding a property called messag-Id to the Properties of the message.

So it is obvious that I also don't know how to assign a SystemProperty properly.
These system Properties however work:
$.ct for iothub-content-type
$.ce for iothub-content-encoding

This is what my telemetry data looks like in CosmosDB Data Explorer.:

from the docs: https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-messages-construct#system-properties-of-d2c-iot-hub-messages

I should be able to set the message-Id:

So if someone can please help my understanding, of how to get a callback to return the ID of the uploaded telemetry message, in such a manner that it can be tracked to a specific message.
Thx

[vaavva]

Try using these values to set properties: https://github.com/Azure/azure-sdk-for-c/blob/main/sdk/inc/azure/iot/az_iot_common.h#L117

xResult = AzureIoTMessage_PropertiesAppend(&xPropertyBag,
			(uint8_t *)AZ_IOT_MESSAGE_PROPERTIES_MESSAGE_ID, sizeof(AZ_IOT_MESSAGE_PROPERTIES_MESSAGE_ID) - 1,
			(uint8_t *)"2022",
			sizeof("2022") - 1);
		configASSERT(xResult == eAzureIoTSuccess);

Note that this won't set the packet ID, but will add it to system properties

[vaavva]

Hello @wreyford! A few things that might help:

• telemetryPacketID will be set by AzureIoTHubClient_SendTelemetry, so I'd recommend initializing it as uint16_t telemetryPacketID; to keep from confusing things. The underlying function that sets it is: https://github.com/FreeRTOS/coreMQTT/blob/6b1336f662a1baa05ac04f248e6eb19ffdf88809/source/core_mqtt.c#L2270C10-L2270C26 (which, as you noted, sequentially increments the value). This is tied to the MQTT context, which is why it would reset on a network reconnection.

Our tests are actually a decent resource of how you would check against this value:

• Setting the callback: https://github.com/Azure/azure-iot-middleware-freertos/blob/main/tests/e2e/iot/device/e2e_device_task.c#L70
• The callback method: https://github.com/Azure/azure-iot-middleware-freertos/blob/main/tests/e2e/iot/device/e2e_device_commands.c#L85-L89
• Sending telemetry and then waiting for the puback: https://github.com/Azure/azure-iot-middleware-freertos/blob/main/tests/e2e/iot/device/e2e_device_commands.c#L465-L476
• Checking against the last value: https://github.com/Azure/azure-iot-middleware-freertos/blob/main/tests/e2e/iot/device/e2e_device_commands.c#L95-L115

In your case, it might make more sense to save the sent publish ID and check/clear it in the puback callback instead of creating a separate method to wait for the puback. Something like this:

static uint16_t usSentTelemetryPacketID;
void prvTelemetryPubackCallback( uint16_t usPacketID )
{
    if (usPacketID == usSentTelemetryPacketID) {
        AZLogInfo( ( "matching puback received: %d", usPacketID ) );
        usSentTelemetryPacketID = 0;
    }
    else
    {
        AZLogInfo( ( "non-matching puback received: %d", usPacketID ) );
    }
}

[wreyford]

Thank you @vaavva for you very complete explanation. I have now managed to put the pieces together.

I do however note in the middleware tests you point me to, that they are passing in Seconds instead om Milliseconds to the process loop.

static uint32_t prvWaitForPuback( AzureIoTHubClient_t * pxAzureIoTHubClient,
                                  uint16_t usSentTelemetryPublishID )
{
    uint32_t xResult;

    AzureIoTHubClient_ProcessLoop( pxAzureIoTHubClient, e2etestPROCESS_LOOP_WAIT_TIMEOUT_IN_SEC );

e2etestPROCESS_LOOP_WAIT_TIMEOUT_IN_SEC is defined as 4, thus only 4 milliseconds as per the definition of Process Loop

AzureIoTResult_t AzureIoTHubClient_ProcessLoop( AzureIoTHubClient_t * pxAzureIoTHubClient,
                                                uint32_t ulTimeoutMilliseconds )

This is misleading, or am I again missing something here?