Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Crank runner for Helix #3230

Merged
merged 39 commits into from
Aug 11, 2023
Merged

Crank runner for Helix #3230

Show file tree
Hide file tree
Changes from 17 commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
879da59
WIP work
caaavik-msft Jun 8, 2023
f8e29e4
Merge from master
caaavik-msft Jul 20, 2023
1fcc834
Merge branch 'main' into caaavik/crank-runner-wip
caaavik-msft Jul 25, 2023
b649527
wip work
caaavik-msft Aug 4, 2023
05ab6e6
Merge branch 'main' into caaavik/crank-runner-wip
caaavik-msft Aug 4, 2023
b1db130
last tweaks for today
caaavik-msft Aug 4, 2023
da22748
More bug fixes and work towards scenarios support
caaavik-msft Aug 8, 2023
0b07360
Ensure python 3.10 is used in the pipeline
caaavik-msft Aug 8, 2023
c7af027
Remove unused args from ci_setup
caaavik-msft Aug 8, 2023
a15f079
Fix build configs arg
caaavik-msft Aug 8, 2023
4ce2b1d
handle public builds when upload token is missing
caaavik-msft Aug 9, 2023
311b71f
Fix when run name is missing
caaavik-msft Aug 9, 2023
17e1236
Add support back to helix.proj for non-corerun
caaavik-msft Aug 9, 2023
8a93ac7
Fix upload being imported on public runs
caaavik-msft Aug 9, 2023
b8a030f
Attempt at fixing ubuntu builds
caaavik-msft Aug 9, 2023
f201017
Try Python 3.11 in case that works, fix utf8 issue
caaavik-msft Aug 9, 2023
8c68777
Check in all the files
caaavik-msft Aug 9, 2023
d1d1359
test if pypy works?
caaavik-msft Aug 10, 2023
c280926
try actual way to set environment variables
caaavik-msft Aug 10, 2023
471ece6
try get ci_setup to work with dataclasses backport
caaavik-msft Aug 10, 2023
f9817c3
ensure pip is installed
caaavik-msft Aug 10, 2023
818c41d
Use ensurepip to add pip to path
caaavik-msft Aug 10, 2023
d859fc3
Try using get-pip.py to get pip instead
caaavik-msft Aug 10, 2023
53ef92c
Use the 3.6 version of get-pip...
caaavik-msft Aug 10, 2023
c1e6cce
Use typing.List instead of list for type hints
caaavik-msft Aug 10, 2023
dc00796
And use typing.Set too
caaavik-msft Aug 10, 2023
1a4b6ec
Use Optional instead of union type
caaavik-msft Aug 10, 2023
ade6cb6
Use typing.List in shared python files as well
caaavik-msft Aug 10, 2023
589fc9d
More bug fixes
caaavik-msft Aug 10, 2023
6b00d44
Pass --target-windows param in CI pipeline
caaavik-msft Aug 10, 2023
538fdde
Change target windows to boolean param
caaavik-msft Aug 10, 2023
81ec8ab
use more typing.List
caaavik-msft Aug 10, 2023
ef54698
Address review comments
caaavik-msft Aug 10, 2023
836a4be
Fix type hints for ci setup
caaavik-msft Aug 10, 2023
897e78e
Add documentation
caaavik-msft Aug 10, 2023
864a8e0
Fix markdown lint issues
caaavik-msft Aug 10, 2023
aa3acdd
More markdown lint fixes
caaavik-msft Aug 10, 2023
7433420
Address PR comments
caaavik-msft Aug 11, 2023
6a0920d
Make ci_setup.py backwards compatible with runtime
caaavik-msft Aug 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
!.git
# Other folders to ignore
**/logs/
/tools/
Expand Down
12 changes: 11 additions & 1 deletion .vscode/settings.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,5 +9,15 @@
"**/obj/": true,
"**/packages/": true,
"tools/": true,
}
},
"python.analysis.typeCheckingMode": "strict",
"python.analysis.diagnosticSeverityOverrides": {
"reportMissingParameterType": "none",
"reportUnnecessaryComparison": "information",
"reportUnnecessaryIsInstance": "information",
"reportUnusedVariable": "information",
"reportMissingTypeStubs": "information"
},
"python.analysis.extraPaths": ["scripts"],
"python.analysis.diagnosticMode": "workspace"
}
1 change: 1 addition & 0 deletions eng/performance/Scenarios.Common.props
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@

<PropertyGroup>
<FrameworkVersion>$(PERFLAB_Framework.Substring($([MSBuild]::Subtract($(PERFLAB_Framework.Length), 3))))</FrameworkVersion>
<HelixResultsDestinationDir>$(CorrelationPayloadDirectory)performance/artifacts/helix-results</HelixResultsDestinationDir>
</PropertyGroup>

<PropertyGroup Condition="'$(TargetsWindows)' == 'true'">
Expand Down
6 changes: 5 additions & 1 deletion eng/performance/benchmark_jobs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,11 +123,15 @@ jobs:
${{ channel }}:
_Channel: ${{ channel }}
_Configs: CompilationMode=Tiered RunKind="${{ parameters.kind }}"
_PerfLabArguments: $(PerfLabArguments)
_PerfLabArguments: $(PerfLabArguments) # not necessary?
caaavik-msft marked this conversation as resolved.
Show resolved Hide resolved
_BuildConfig: ${{ parameters.architecture }}_$(_Channel)_${{ parameters.kind }} # needs to be unique to avoid logs overwriting in mc.dot.net
steps:
- checkout: self
clean: true
- task: UsePythonVersion@0
inputs:
versionSpec: '3.11'
displayName: Install Python 3.11
- script: $(Python) scripts/ci_setup.py --channel $(_Channel) --architecture ${{parameters.architecture}} --perf-hash $(Build.SourceVersion) --queue ${{parameters.queue}} --build-number $(Build.BuildNumber) --build-configs $(_Configs) $(AffinityParam) $(runEnvVarsParam)
displayName: Run ci_setup.py
- ${{ if eq(parameters.osName, 'windows') }}:
Expand Down
128 changes: 105 additions & 23 deletions eng/performance/helix.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,63 +1,145 @@
<Project Sdk="Microsoft.DotNet.Helix.Sdk" DefaultTargets="Test">

<PropertyGroup Condition="'$(TargetsWindows)' == 'true'">
<PropertyGroup Condition="'$(TargetsWindows)' == 'true' AND '$(UseCoreRun)' == 'true'">
<PerformanceDirectory>%HELIX_WORKITEM_ROOT%\performance</PerformanceDirectory>
<HelixPreCommands>$(HelixPreCommands) &amp;&amp; robocopy /np /nfl /e %HELIX_CORRELATION_PAYLOAD%\performance $(PerformanceDirectory) /XD %HELIX_CORRELATION_PAYLOAD%\performance\.git</HelixPreCommands>
DrewScoggins marked this conversation as resolved.
Show resolved Hide resolved
<WorkItemCommand>$(PerformanceDirectory)\scripts\benchmarks_ci.py --csproj $(PerformanceDirectory)\$(TargetCsproj)</WorkItemCommand>
<Python>py -3</Python>
<CoreRun>%HELIX_CORRELATION_PAYLOAD%\Core_Root\CoreRun.exe</CoreRun>
<BaselineCoreRun>%HELIX_CORRELATION_PAYLOAD%\Baseline_Core_Root\CoreRun.exe</BaselineCoreRun>
<HelixPreCommands>$(HelixPreCommands);call $(PerformanceDirectory)\tools\machine-setup.cmd;set PYTHONPATH=%HELIX_WORKITEM_PAYLOAD%\scripts%3B%HELIX_WORKITEM_PAYLOAD%</HelixPreCommands>
<ArtifactsDirectory>%HELIX_WORKITEM_ROOT%\artifacts\BenchmarkDotNet.Artifacts</ArtifactsDirectory>
<BaselineArtifactsDirectory>%HELIX_WORKITEM_ROOT%\artifacts\BenchmarkDotNet.Artifacts_Baseline</BaselineArtifactsDirectory>
</PropertyGroup>

<PropertyGroup Condition="'$(TargetsWindows)' != 'true' AND '$(UseCoreRun)' == 'true'">
<PerformanceDirectory>$HELIX_WORKITEM_ROOT/performance</PerformanceDirectory>
<HelixPreCommands>$(HelixPreCommands);cp -R $(BaseDirectory)/performance $(PerformanceDirectory)</HelixPreCommands>
<WorkItemCommand>$(PerformanceDirectory)/scripts/benchmarks_ci.py --csproj $(PerformanceDirectory)/$(TargetCsproj)</WorkItemCommand>
<Python>python3</Python>
<CoreRun>$HELIX_CORRELATION_PAYLOAD/Core_Root/corerun</CoreRun>
<BaselineCoreRun>$HELIX_CORRELATION_PAYLOAD/Baseline_Core_Root/corerun</BaselineCoreRun>
<HelixPreCommands>$(HelixPreCommands);cp -R $HELIX_CORRELATION_PAYLOAD/performance $(PerformanceDirectory);chmod +x $(PerformanceDirectory)/tools/machine-setup.sh;. $(PerformanceDirectory)/tools/machine-setup.sh</HelixPreCommands>
<ArtifactsDirectory>$HELIX_WORKITEM_ROOT/artifacts/BenchmarkDotNet.Artifacts</ArtifactsDirectory>
<BaselineArtifactsDirectory>$HELIX_WORKITEM_ROOT/artifacts/BenchmarkDotNet.Artifacts_Baseline</BaselineArtifactsDirectory>
</PropertyGroup>

<PropertyGroup Condition="'$(TargetsWindows)' == 'true' AND '$(UseCoreRun)' != 'true'">
<WorkItemCommand>%HELIX_CORRELATION_PAYLOAD%\scripts\benchmarks_ci.py --csproj %HELIX_CORRELATION_PAYLOAD%\$(TargetCsproj)</WorkItemCommand>
</PropertyGroup>

<PropertyGroup Condition="'$(TargetsWindows)' != 'true'">
<PropertyGroup Condition="'$(TargetsWindows)' != 'true' AND '$(UseCoreRun)' != 'true'">
<WorkItemCommand>scripts/benchmarks_ci.py --csproj $(TargetCsproj)</WorkItemCommand>
</PropertyGroup>

<PropertyGroup Condition="'$(WasmDotnet)' == 'true'">
<CliArguments>$(CliArguments) --run-isolated --wasm --dotnet-path %24HELIX_CORRELATION_PAYLOAD/dotnet/</CliArguments>
</PropertyGroup>

<PropertyGroup Condition="'$(MonoDotnet)' == 'true' and '$(AGENT_OS)' == 'Windows_NT'">
<CoreRunArgument>--corerun %HELIX_CORRELATION_PAYLOAD%\dotnet-mono\shared\Microsoft.NETCore.App\$(ProductVersion)\corerun.exe</CoreRunArgument>
</PropertyGroup>
<PropertyGroup Condition="'$(MonoDotnet)' == 'true' and '$(AGENT_OS)' != 'Windows_NT'">
<CoreRunArgument>--corerun $(BaseDirectory)/dotnet-mono/shared/Microsoft.NETCore.App/$(ProductVersion)/corerun</CoreRunArgument>
</PropertyGroup>

<PropertyGroup Condition="'$(UseCoreRun)' == 'true'">
<CoreRunArgument>--corerun $(CoreRun)</CoreRunArgument>
</PropertyGroup>

<PropertyGroup Condition="'$(UseBaselineCoreRun)' == 'true'">
<BaselineCoreRunArgument>--corerun $(BaselineCoreRun)</BaselineCoreRunArgument>
</PropertyGroup>

<PropertyGroup Condition="'$(WorkItemCommand)' != ''">
<WorkItemCommand>$(Python) $(WorkItemCommand) --incremental no --architecture $(Architecture) -f $(PERFLAB_Framework) $(_PerfLabArguments)</WorkItemCommand>
<WorkItemCommand>$(Python) $(WorkItemCommand) --incremental no --architecture $(Architecture) -f $(PERFLAB_Framework) $(PerfLabArguments) --bdn-artifacts $(ArtifactsDirectory) </WorkItemCommand>
</PropertyGroup>

<!-- TODO: Check for net462 and net461 -->
<PropertyGroup Condition="'$(PERFLAB_Framework)' != 'net462'">
<WorkItemCommand>$(WorkItemCommand) $(CliArguments)</WorkItemCommand>
</PropertyGroup>

<!-- TODO: Take this property in using python-->
<PropertyGroup>
<WorkItemTimeout>6:00</WorkItemTimeout>
<WorkItemTimeout Condition="'$(OnlySanityCheck)' == 'true'">1:30</WorkItemTimeout>
<HelixResultsDestinationDir>$(CorrelationPayloadDirectory)/performance/artifacts/helix-results</HelixResultsDestinationDir>
</PropertyGroup>

<ItemGroup>
<HelixCorrelationPayload Include="$(CorrelationPayloadDirectory)">
<HelixCorrelationPayload Include="$(CorrelationPayloadDirectory)/Core_Root" Condition="'$(UseCoreRun)' == 'true'">
<PayloadDirectory>%(Identity)</PayloadDirectory>
<Destination>Core_Root</Destination>
</HelixCorrelationPayload>
<HelixCorrelationPayload Include="$(CorrelationPayloadDirectory)/performance" Condition="'$(UseCoreRun)' == 'true'">
<PayloadDirectory>%(Identity)</PayloadDirectory>
<Destination>performance</Destination>
</HelixCorrelationPayload>
<HelixCorrelationPayload Include="$(CorrelationPayloadDirectory)" Condition="'$(UseCoreRun)' != 'true'">
<PayloadDirectory>%(Identity)</PayloadDirectory>
</HelixCorrelationPayload>
</ItemGroup>

<PropertyGroup>
<PartitionCount>15</PartitionCount>
<PartitionCount Condition="'$(PartitionCount)' == ''">15</PartitionCount>
</PropertyGroup>
<ItemGroup>
<Partition Include="Partition0" Index="0" />
<Partition Include="Partition1" Index="1" />
<Partition Include="Partition2" Index="2" />
<Partition Include="Partition3" Index="3" />
<Partition Include="Partition4" Index="4" />
<Partition Include="Partition5" Index="5" />
<Partition Include="Partition6" Index="6" />
<Partition Include="Partition7" Index="7" />
<Partition Include="Partition8" Index="8" />
<Partition Include="Partition9" Index="9" />
<Partition Include="Partition10" Index="10" />
<Partition Include="Partition11" Index="11" />
<Partition Include="Partition12" Index="12" />
<Partition Include="Partition13" Index="13" />
<Partition Include="Partition14" Index="14" />
<Partition Include="Partition0" Index="0" Condition="$(PartitionCount) &gt; 0" />
<Partition Include="Partition1" Index="1" Condition="$(PartitionCount) &gt; 1" />
<Partition Include="Partition2" Index="2" Condition="$(PartitionCount) &gt; 2" />
<Partition Include="Partition3" Index="3" Condition="$(PartitionCount) &gt; 3" />
<Partition Include="Partition4" Index="4" Condition="$(PartitionCount) &gt; 4" />
<Partition Include="Partition5" Index="5" Condition="$(PartitionCount) &gt; 5" />
<Partition Include="Partition6" Index="6" Condition="$(PartitionCount) &gt; 6" />
<Partition Include="Partition7" Index="7" Condition="$(PartitionCount) &gt; 7" />
<Partition Include="Partition8" Index="8" Condition="$(PartitionCount) &gt; 8" />
<Partition Include="Partition9" Index="9" Condition="$(PartitionCount) &gt; 9" />
<Partition Include="Partition10" Index="10" Condition="$(PartitionCount) &gt; 10" />
<Partition Include="Partition11" Index="11" Condition="$(PartitionCount) &gt; 11" />
<Partition Include="Partition12" Index="12" Condition="$(PartitionCount) &gt; 12" />
<Partition Include="Partition13" Index="13" Condition="$(PartitionCount) &gt; 13" />
<Partition Include="Partition14" Index="14" Condition="$(PartitionCount) &gt; 14" />
<Partition Include="Partition15" Index="15" Condition="$(PartitionCount) &gt; 15" />
<Partition Include="Partition16" Index="16" Condition="$(PartitionCount) &gt; 16" />
<Partition Include="Partition17" Index="17" Condition="$(PartitionCount) &gt; 17" />
<Partition Include="Partition18" Index="18" Condition="$(PartitionCount) &gt; 18" />
<Partition Include="Partition19" Index="19" Condition="$(PartitionCount) &gt; 19" />
<Partition Include="Partition20" Index="20" Condition="$(PartitionCount) &gt; 20" />
<Partition Include="Partition21" Index="21" Condition="$(PartitionCount) &gt; 21" />
<Partition Include="Partition22" Index="22" Condition="$(PartitionCount) &gt; 22" />
<Partition Include="Partition23" Index="23" Condition="$(PartitionCount) &gt; 23" />
<Partition Include="Partition24" Index="24" Condition="$(PartitionCount) &gt; 24" />
<Partition Include="Partition25" Index="25" Condition="$(PartitionCount) &gt; 25" />
<Partition Include="Partition26" Index="26" Condition="$(PartitionCount) &gt; 26" />
<Partition Include="Partition27" Index="27" Condition="$(PartitionCount) &gt; 27" />
<Partition Include="Partition28" Index="28" Condition="$(PartitionCount) &gt; 28" />
<Partition Include="Partition29" Index="29" Condition="$(PartitionCount) &gt; 29" />
</ItemGroup>

<PropertyGroup>
<BenchmarkDotNetArguments>$(BenchmarkDotNetArguments) $(ExtraBenchmarkDotNetArguments) $(CoreRunArgument)</BenchmarkDotNetArguments>
</PropertyGroup>

<!-- TODO: Support Compare from runtime repo -->

<!--
Partition the MicroBenchmarks project, but nothing else
-->
<ItemGroup Condition="$(TargetCsproj.Contains('MicroBenchmarks.csproj'))">
<HelixWorkItem Include="@(Partition)">
<PayloadDirectory>$(WorkItemDirectory)</PayloadDirectory>
<Command>$(WorkItemCommand) --bdn-arguments="$(BenchmarkDotNetArguments) --partition-count $(PartitionCount) --partition-index %(HelixWorkItem.Index)"</Command>
<Timeout>4:00</Timeout>
<Command>$(WorkItemCommand) --partition=%(HelixWorkItem.Index) --bdn-arguments="$(BenchmarkDotNetArguments) --partition-count $(PartitionCount) --partition-index %(HelixWorkItem.Index)"</Command>
<Timeout>$(WorkItemTimeout)</Timeout>
<DownloadFilesFromResults>Partition%(HelixWorkItem.Index)-combined-perf-lab-report.json</DownloadFilesFromResults>
</HelixWorkItem>
</ItemGroup>
<ItemGroup Condition="!$(TargetCsproj.Contains('MicroBenchmarks.csproj'))">
<HelixWorkItem Include="WorkItem">
<PayloadDirectory>$(WorkItemDirectory)</PayloadDirectory>
<Command>$(WorkItemCommand) --bdn-arguments="$(BenchmarkDotNetArguments)"</Command>
<Timeout>4:00</Timeout>
<Timeout>$(WorkItemTimeout)</Timeout>
<DownloadFilesFromResults>combined-perf-lab-report.json</DownloadFilesFromResults>
</HelixWorkItem>
</ItemGroup>
</Project>
6 changes: 6 additions & 0 deletions eng/performance/helix_sender.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
FROM python:3.10-bullseye
caaavik-msft marked this conversation as resolved.
Show resolved Hide resolved

WORKDIR /performance

# copy files in
COPY . .
6 changes: 3 additions & 3 deletions eng/performance/scenarios.proj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,15 +35,15 @@
<ScenarioDirectoryName>emptyvbconsoletemplate</ScenarioDirectoryName>
<PayloadDirectory>$(ScenariosDir)%(ScenarioDirectoryName)</PayloadDirectory>
</Scenario>
<UIScenario Include="WinForms Large" Condition="'$(TargetsWindows)' == 'true'">
<UIScenario Include="WinForms Large" Condition="'$(TargetsWindows)' == 'true' AND '$(OS)' == 'Windows_NT'">
<ScenarioDirectoryName>windowsformslarge</ScenarioDirectoryName>
<PayloadDirectory>$(ScenariosDir)%(ScenarioDirectoryName)</PayloadDirectory>
</UIScenario>
<UIScenario Include="WPF Template" Condition="'$(TargetsWindows)' == 'true'">
<UIScenario Include="WPF Template" Condition="'$(TargetsWindows)' == 'true' AND '$(OS)' == 'Windows_NT'">
<ScenarioDirectoryName>wpf</ScenarioDirectoryName>
<PayloadDirectory>$(ScenariosDir)%(ScenarioDirectoryName)</PayloadDirectory>
</UIScenario>
<UIScenario Include="WPF SFC" Condition="'$(TargetsWindows)' == 'true'">
<UIScenario Include="WPF SFC" Condition="'$(TargetsWindows)' == 'true' AND '$(OS)' == 'Windows_NT'">
<ScenarioDirectoryName>wpfsfc</ScenarioDirectoryName>
<PayloadDirectory>$(ScenariosDir)%(ScenarioDirectoryName)</PayloadDirectory>
</UIScenario>
Expand Down
4 changes: 4 additions & 0 deletions eng/performance/scenarios.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,10 @@ jobs:
steps:
- checkout: self
clean: true
- task: UsePythonVersion@0
inputs:
versionSpec: '3.11'
displayName: Install Python 3.11
- ${{ if ne(length(parameters.channels), 0) }}:
- script: $(Python) scripts/ci_setup.py --channel $(_Channel) --architecture ${{parameters.architecture}} --perf-hash $(Build.SourceVersion) --queue ${{parameters.queue}} --build-number $(Build.BuildNumber) --build-configs $(_Configs) --output-file $(CorrelationStaging)machine-setup$(ScriptExtension) --install-dir $(CorrelationStaging)dotnet $(runEnvVarsParam) $(AffinityParam)
displayName: Run ci_setup.py
Expand Down
Loading
Loading