Skip to content

Commit 57a3b46

Browse files
lewingCopilot
andauthored
[release/9.0] Fix intermediate artifact upload collisions for NativeAOT and Mono (#125564)
## Problem The `dotnet-runtime-official` pipeline on `release/9.0` has intermittent "File upload failed even after retry" errors caused by multiple platform jobs uploading arch-independent packages (like `Microsoft.NETCore.App.Ref.9.0.15.nupkg`) to the same artifact path simultaneously. **Root cause:** NativeAOT (27 platforms) all upload to `name: NativeAOTRuntimePacks` and Mono (7+ matrix expansions) all upload to `name: MonoRuntimePacks` — using a shared subdirectory within the `IntermediateArtifacts` artifact. When two jobs finish at the same time, they race to upload the same file, causing blob storage write conflicts. Compare with CoreCLR, which correctly uses `name: $(osGroup)$(osSubgroup)_$(archType)` — unique per platform. ✅ ## Fix Make each upload name unique per platform (matching the CoreCLR pattern): | Build Type | Before | After | |---|---|---| | NativeAOT | `NativeAOTRuntimePacks` | `NativeAOTRuntimePacks_$(osGroup)$(osSubgroup)_$(archType)` | | Mono | `MonoRuntimePacks` | `MonoRuntimePacks_$(osGroup)$(osSubgroup)_$(archType)` | | Mono multithread | `MonoRuntimePacks` | `MonoRuntimePacks_multithread_$(osGroup)$(osSubgroup)_$(archType)` | | CrossAOT | `MonoRuntimePacks` | `MonoRuntimePacks_crossaot_$(osGroup)$(osSubgroup)_$(archType)` | | LLVM AOT | `MonoRuntimePacks` | `MonoRuntimePacks_llvmaot_$(osGroup)$(osSubgroup)_$(archType)` | Updated the Workloads job download patterns from `IntermediateArtifacts/MonoRuntimePacks/Shipping/...` to `IntermediateArtifacts/MonoRuntimePacks*/Shipping/...` so the wildcard matches all per-platform subdirectories. The `CopyFiles@2` flatten step (`*/Shipping/*.nupkg`) still works because each platform subdirectory is still a single path segment. ## Notes - Only affects `release/9.0` — `main`/`release/10.0` have moved to VMR builds - Perf pipelines (`perf-non-wasm-jobs.yml`) use the same pattern but only have single-platform uploads (no collision risk), so no changes needed there Fixes #125561 Ref: dotnet/dnceng#1916 --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
1 parent dd1be77 commit 57a3b46

1 file changed

Lines changed: 33 additions & 34 deletions

File tree

eng/pipelines/runtime-official.yml

Lines changed: 33 additions & 34 deletions
Original file line numberDiff line numberDiff line change
@@ -300,7 +300,7 @@ extends:
300300
postBuildSteps:
301301
- template: /eng/pipelines/common/upload-intermediate-artifacts-step.yml
302302
parameters:
303-
name: NativeAOTRuntimePacks
303+
name: NativeAOTRuntimePacks_$(osGroup)$(osSubgroup)_$(archType)
304304

305305
#
306306
# Build Mono runtime packs
@@ -344,7 +344,7 @@ extends:
344344
postBuildSteps:
345345
- template: /eng/pipelines/common/upload-intermediate-artifacts-step.yml
346346
parameters:
347-
name: MonoRuntimePacks
347+
name: MonoRuntimePacks_$(osGroup)$(osSubgroup)_$(archType)
348348

349349
- template: /eng/pipelines/common/platform-matrix.yml
350350
parameters:
@@ -362,7 +362,7 @@ extends:
362362
postBuildSteps:
363363
- template: /eng/pipelines/common/upload-intermediate-artifacts-step.yml
364364
parameters:
365-
name: MonoRuntimePacks
365+
name: MonoRuntimePacks_$(osGroup)$(osSubgroup)_$(archType)
366366

367367
- template: /eng/pipelines/common/platform-matrix.yml
368368
parameters:
@@ -380,7 +380,7 @@ extends:
380380
postBuildSteps:
381381
- template: /eng/pipelines/common/upload-intermediate-artifacts-step.yml
382382
parameters:
383-
name: MonoRuntimePacks
383+
name: MonoRuntimePacks_multithread_$(osGroup)$(osSubgroup)_$(archType)
384384

385385
# Build Mono AOT offset headers once, for consumption elsewhere
386386
#
@@ -452,7 +452,7 @@ extends:
452452
postBuildSteps:
453453
- template: /eng/pipelines/common/upload-intermediate-artifacts-step.yml
454454
parameters:
455-
name: MonoRuntimePacks
455+
name: MonoRuntimePacks_crossaot_$(osGroup)$(osSubgroup)_$(archType)
456456

457457
- template: /eng/pipelines/common/platform-matrix.yml
458458
parameters:
@@ -480,7 +480,7 @@ extends:
480480
postBuildSteps:
481481
- template: /eng/pipelines/common/upload-intermediate-artifacts-step.yml
482482
parameters:
483-
name: MonoRuntimePacks
483+
name: MonoRuntimePacks_crossaot_$(osGroup)$(osSubgroup)_$(archType)
484484

485485
- template: /eng/pipelines/common/platform-matrix.yml
486486
parameters:
@@ -514,7 +514,7 @@ extends:
514514
postBuildSteps:
515515
- template: /eng/pipelines/common/upload-intermediate-artifacts-step.yml
516516
parameters:
517-
name: MonoRuntimePacks
517+
name: MonoRuntimePacks_crossaot_$(osGroup)$(osSubgroup)_$(archType)
518518

519519
#
520520
# Build Mono LLVM runtime packs
@@ -546,7 +546,7 @@ extends:
546546
postBuildSteps:
547547
- template: /eng/pipelines/common/upload-intermediate-artifacts-step.yml
548548
parameters:
549-
name: MonoRuntimePacks
549+
name: MonoRuntimePacks_llvmaot_$(osGroup)$(osSubgroup)_$(archType)
550550

551551
#
552552
# Build libraries AllConfigurations for packages
@@ -620,35 +620,34 @@ extends:
620620
artifact: 'IntermediateArtifacts'
621621
path: $(Build.SourcesDirectory)/artifacts/workloadPackages
622622
patterns: |
623-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.AOT.win-x64.Cross.android-*.nupkg
624-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.AOT.win-arm64.Cross.android-*.nupkg
625-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.AOT.win-x64.Cross.browser-wasm*.nupkg
626-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.AOT.win-arm64.Cross.browser-wasm*.nupkg
627-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.AOT.win-x64.Cross.wasi-wasm*.nupkg
628-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.AOT.win-arm64.Cross.wasi-wasm*.nupkg
629-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.Mono.android-*.nupkg
630-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.Mono.browser-wasm*.nupkg
631-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.Mono.multithread.browser-wasm*.nupkg
632-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.Mono.ios-*.nupkg
633-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.Mono.iossimulator-*.nupkg
634-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.Mono.maccatalyst-*.nupkg
635-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.Mono.multithread.browser-wasm*.nupkg
636-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.Mono.tvos-*.nupkg
637-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.Mono.tvossimulator-*.nupkg
638-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NETCore.App.Runtime.Mono.wasi-wasm*.nupkg
639-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NET.Workload.Mono.ToolChain.Current.Manifest*.nupkg
640-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NET.Workload.Mono.ToolChain.net6.Manifest*.nupkg
641-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NET.Workload.Mono.ToolChain.net7.Manifest*.nupkg
642-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NET.Workload.Mono.ToolChain.net8.Manifest*.nupkg
643-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NET.Runtime.MonoTargets.Sdk*.nupkg
644-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NET.Runtime.MonoAOTCompiler.Task*.nupkg
645-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NET.Runtime.WebAssembly.Sdk*.nupkg
646-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NET.Runtime.WebAssembly.Wasi*.nupkg
647-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NET.Runtime.WebAssembly.Templates*.nupkg
623+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.AOT.win-x64.Cross.android-*.nupkg
624+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.AOT.win-arm64.Cross.android-*.nupkg
625+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.AOT.win-x64.Cross.browser-wasm*.nupkg
626+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.AOT.win-arm64.Cross.browser-wasm*.nupkg
627+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.AOT.win-x64.Cross.wasi-wasm*.nupkg
628+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.AOT.win-arm64.Cross.wasi-wasm*.nupkg
629+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.Mono.android-*.nupkg
630+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.Mono.browser-wasm*.nupkg
631+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.Mono.multithread.browser-wasm*.nupkg
632+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.Mono.ios-*.nupkg
633+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.Mono.iossimulator-*.nupkg
634+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.Mono.maccatalyst-*.nupkg
635+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.Mono.tvos-*.nupkg
636+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.Mono.tvossimulator-*.nupkg
637+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NETCore.App.Runtime.Mono.wasi-wasm*.nupkg
638+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NET.Workload.Mono.ToolChain.Current.Manifest*.nupkg
639+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NET.Workload.Mono.ToolChain.net6.Manifest*.nupkg
640+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NET.Workload.Mono.ToolChain.net7.Manifest*.nupkg
641+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NET.Workload.Mono.ToolChain.net8.Manifest*.nupkg
642+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NET.Runtime.MonoTargets.Sdk*.nupkg
643+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NET.Runtime.MonoAOTCompiler.Task*.nupkg
644+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NET.Runtime.WebAssembly.Sdk*.nupkg
645+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NET.Runtime.WebAssembly.Wasi*.nupkg
646+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NET.Runtime.WebAssembly.Templates*.nupkg
648647
IntermediateArtifacts/windows_arm64/Shipping/Microsoft.NETCore.App.Runtime.win-arm64*.nupkg
649648
IntermediateArtifacts/windows_x64/Shipping/Microsoft.NETCore.App.Runtime.win-x64*.nupkg
650649
IntermediateArtifacts/windows_x86/Shipping/Microsoft.NETCore.App.Runtime.win-x86*.nupkg
651-
IntermediateArtifacts/MonoRuntimePacks/Shipping/Microsoft.NET.Sdk.WebAssembly.Pack*.nupkg
650+
IntermediateArtifacts/MonoRuntimePacks*/Shipping/Microsoft.NET.Sdk.WebAssembly.Pack*.nupkg
652651
653652
- task: CopyFiles@2
654653
displayName: Flatten packages

0 commit comments

Comments
 (0)