[exporter/azureblob] Building outputted files in a sequential manner

[azizabah]

Component(s)

exporter/azureblob

Is your feature request related to a problem? Please describe.

Currently there is a limitation of 50,000 appends to an append blob in Azure Storage accounts https://learn.microsoft.com/en-us/rest/api/storageservices/append-block?tabs=microsoft-entra-id#remarks. This leads to some guess work when trying to figure out how many files to create when trying to optimize for least number of files because of data processing.

Describe the solution you'd like

It would be helpful if the append logic included information to see if the current blob was full (returned 409 from the API call saying its at its limit), and instead created the next blob at that point. It would also be helpful if instead of using random values, this could just be sequenced automatically in an incrementing fashion. i.e. create file 0, append 50,000 blobs, create file 1, append 50,000 blobs etc.

Describe alternatives you've considered

No response

Additional context

No response

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}

[github-actions]

Pinging code owners:

• exporter/azureblob: @hgaol @MovieStoreGuy

See Adding Labels via Comments if you do not have permissions to add labels yourself.