GenerateBundle Task fails writing to singlefilehost.exe

[dsplaisted]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1146095
Build error leg or test failing: Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution
Pull request: #50648

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "The process cannot access the file '.*singlefilehost\\.exe' because it is being used by another process",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=1146095
Error message validated: [The process cannot access the file '.*singlefilehost\.exe' because it is being used by another process]
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 9/12/2025 3:27:55 PM UTC

Report

Build	Definition	Test	Pull Request
1207162	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51801
1206490	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51790
1206475	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51791
1206495	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51792
1206509	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51723
1206429	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51762
1206398	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51411
1206404	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51790
1206341	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution	#51411
1206192	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51788
1205093	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution	#51724
1205641	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51781
1205233	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution	#51760
1204153	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51760
1204961	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51411
1204812	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51747
1204493	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution	#51747
1204143	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51758
1201304	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution	#51697
1204019	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51747
1204000	dotnet/sdk	Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishIncrementally.Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishIncrementally.It_cleans_between_renames	#51747
1203943	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution	#51727
1203612	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51727
1203143	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution	#51724
1203335	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51723
1203182	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51108
1203220	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51723
1202929	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51733
1202989	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51724
1202916	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51732
1202877	dotnet/sdk	Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishASingleFileApp.Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishASingleFileApp.It_can_disable_cetcompat	#51727
1202777	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution	#51727
1202318	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51720
1202505	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51723
1201890	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51108
1201132	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51692
1201545	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51709
1201088	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51632
1201085	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51411
1201067	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51223
1200905	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51192
1200923	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51411
1200743	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51411
1200514	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51567
1200456	dotnet/sdk	Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishIncrementally.Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishIncrementally.It_cleans_between_renames	#51223
1200367	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51567
1200146	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51223
1200138	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51652
1200055	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51223
1199950	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51411
1199896	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51192
1199741	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51612
1199680	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution
1198478	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51612
1197398	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51612
1197217	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51612
1197114	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51612
1191467	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51425
1182892	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51425
1182766	dotnet/sdk	Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution	#51425

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
9	43	60

[dsplaisted]

@dotnet/illink @agocke We are seeing failures where the GenerateBundle task is failing because it cannot access the singlnefilehost.exe file. We added retry logic because we thought maybe the issues were caused by defender. I thought that helped, but now the issue is happening pretty consistently, so it may be a product issue.

Interestingly, it seems to only fail on Mac OS and Linux when bundling a Windows app. I can't remember if it was failing on Windows before we added the retry logic or not.

Can someone investigate this?

I'm also wondering whether we should merge this codeflow PR which is blocked by the failure. That's where I've seen the failure most consistently recently, so it's possible there's a code change coming in that made the problem worse.

[agocke]

@jtschuster can you take a look?

[dsplaisted]

Ping @jtschuster, any progress on this? I just noticed it in #50876

[agocke]

I'm going to load balance a bit. @sbomer can you look instead?

[jtschuster]

Still investigating, but I am able to repro locally on a mac, but only on the first run of the test. Subsequent runs use the same path for the tmp project and don't repro. It's also interesting that it's failing on a read and not a write.

[dsplaisted]

@jtschuster Any update on this? This is impacting a bunch of our CI runs.

[jtschuster]

Still investigating exactly what's going on, but my hunch is that there's an async memory map flush that keeps the apphost mapped from CreateAppHost. In particular it looks like this todo might fix the issue: https://github.com/dotnet/runtime/blob/b8c39cc4d102518f8608df70e6e3cb7a5c10d48d/src/installer/managed/Microsoft.NET.HostModel/AppHost/HostWriter.cs#L172
The ResourceUpdater.Update() maps the destination apphost back into memory, and if there's async flushing after Dispose(), we could run into issues.

As a workaround, it looks like adding --disable-build-servers to the PublishCommand.Execute() method does something to make the issue fail to repro for both me and @sbomer. I'm not sure why, though.