Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update CodeQL CLI dependency to 2.12.7. #343

Merged
merged 24 commits into from
Aug 30, 2023
Merged

Conversation

lcartey
Copy link
Collaborator

@lcartey lcartey commented Aug 16, 2023

Description

This PR updates the CodeQL CLI dependency to 2.12.7, which includes updating the codeql/cpp-all dependency to 0.6.1.

I have made this update by merging the relevant commits from next into this branch, and updating the supported_codeql_configuration.json and relevant qlpack files.

Change request type

  • Release or process automation (GitHub workflows, internal scripts)
  • Internal documentation
  • External documentation
  • Query files (.ql, .qll, .qls or unit tests)
  • External scripts (analysis report or other code shipped as part of a release)

Rules with added or modified queries

  • No rules added
  • Queries have been added for the following rules:
    • rule number here
  • Queries have been modified for the following rules:
    • FIO32-C

Release change checklist

A change note (development_handbook.md#change-notes) is required for any pull request which modifies:

  • The structure or layout of the release artifacts.
  • The evaluation performance (memory, execution time) of an existing query.
  • The results of an existing query in any circumstance.

If you are only adding new rule queries, a change note is not required.

Author: Is a change note required?

  • Yes
  • No

🚨🚨🚨
Reviewer: Confirm that format of shared queries (not the .qll file, the
.ql file that imports it) is valid by running them within VS Code.

  • Confirmed

Reviewer: Confirm that either a change note is not required or the change note is required and has been added.

  • Confirmed

Query development review checklist

For PRs that add new queries or modify existing queries, the following checklist should be completed by both the author and reviewer:

Author

  • Have all the relevant rule package description files been checked in?
  • Have you verified that the metadata properties of each new query is set appropriately?
  • Do all the unit tests contain both "COMPLIANT" and "NON_COMPLIANT" cases?
  • Are the alert messages properly formatted and consistent with the style guide?
  • Have you run the queries on OpenPilot and verified that the performance and results are acceptable?
    As a rule of thumb, predicates specific to the query should take no more than 1 minute, and for simple queries be under 10 seconds. If this is not the case, this should be highlighted and agreed in the code review process.
  • Does the query have an appropriate level of in-query comments/documentation?
  • Have you considered/identified possible edge cases?
  • Does the query not reinvent features in the standard library?
  • Can the query be simplified further (not golfed!)

Reviewer

  • Have all the relevant rule package description files been checked in?
  • Have you verified that the metadata properties of each new query is set appropriately?
  • Do all the unit tests contain both "COMPLIANT" and "NON_COMPLIANT" cases?
  • Are the alert messages properly formatted and consistent with the style guide?
  • Have you run the queries on OpenPilot and verified that the performance and results are acceptable?
    As a rule of thumb, predicates specific to the query should take no more than 1 minute, and for simple queries be under 10 seconds. If this is not the case, this should be highlighted and agreed in the code review process.
  • Does the query have an appropriate level of in-query comments/documentation?
  • Have you considered/identified possible edge cases?
  • Does the query not reinvent features in the standard library?
  • Can the query be simplified further (not golfed!)

@lcartey lcartey marked this pull request as ready for review August 17, 2023 00:00
@lcartey
Copy link
Collaborator Author

lcartey commented Aug 17, 2023

/test-performance

@github-actions
Copy link

🏁 Beep Boop! Performance testing for this PR has been initiated. Please check back later for results. Note that the query package generation step must complete before testing will start so it might be a minute.

💡 If you do not hear back from me please check my status! I will report even if I fail!

@jsinglet
Copy link
Contributor

🏁 Beep Boop! One or things failed during performance testing. Please check the release engineering repo for details.

@lcartey
Copy link
Collaborator Author

lcartey commented Aug 17, 2023

/test-performance

@github-actions
Copy link

🏁 Beep Boop! Performance testing for this PR has been initiated. Please check back later for results. Note that the query package generation step must complete before testing will start so it might be a minute.

💡 If you do not hear back from me please check my status! I will report even if I fail!

@jsinglet
Copy link
Contributor

🏁 Beep Boop! One or things failed during performance testing. Please check the release engineering repo for details.

Missing target call in DeletedExpr.
In CodeQL CLI 2.12.7 there is a bug which causes an infinite loop
during results interpretation when a result includes more than maxPaths
paths and also includes a path with no edges i.e. where the source and
sink node are the same.

To avoid this edge case, if we report a path where the source and sink
are the same (i.e the throwingExpr directly throws an exception), we
adjust the sink node to report the constructor, which creates a one
step path from the throwingExprFlowNode to the constructor node.

This also means we can delete the `nodes` query predicate, as we only
included it to enable zero-path elements to display.
@lcartey
Copy link
Collaborator Author

lcartey commented Aug 30, 2023

/test-performance

@github-actions
Copy link

🏁 Beep Boop! Performance testing for this PR has been initiated. Please check back later for results. Note that the query package generation step must complete before testing will start so it might be a minute.

💡 If you do not hear back from me please check my status! I will report even if I fail!

Copy link
Contributor

@nicolaswill nicolaswill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've reviewed the changes to A15-2-2: LGTM 👍

@jsinglet
Copy link
Contributor

🏁 Beep Boop! Performance testing complete! See below for performance of the last 3 runs vs your PR. Times are based on predicate performance. You can find full graphs and stats in the PR that was created for this test in the release engineering repo.


Release                            : v2.20.0
Platform                           : x86-linux
Language                           : c
Total_Serialized_Execution_Time_Ms : 2983369
Mean_Predicate_Execution_Time_Ms   : 80.49019290435722
Median_Predicate_Execution_Time_Ms : 1.0
Standard_Deviation_Ms              : 812.7769764280572
Total_Serialized_Execution_Time_s  : 2983.369
Mean_Query_Execution_Time_s        : 0.0804901929043572
Median_Predicate_Execution_Time_s  : 0.001
Percentile95_Ms                    : 183.0
Number_of_Predicates               : 37065

Release                            : v2.20.0
Platform                           : x86-linux
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 3271097
Mean_Predicate_Execution_Time_Ms   : 79.19756434157325
Median_Predicate_Execution_Time_Ms : 1.0
Standard_Deviation_Ms              : 1216.7021436787627
Total_Serialized_Execution_Time_s  : 3271.097
Mean_Query_Execution_Time_s        : 0.0791975643415732
Median_Predicate_Execution_Time_s  : 0.001
Percentile95_Ms                    : 156.0
Number_of_Predicates               : 41303

Release                            : v2.20.0
Platform                           : x86-windows
Language                           : c
Total_Serialized_Execution_Time_Ms : 3323606
Mean_Predicate_Execution_Time_Ms   : 89.50063282617477
Median_Predicate_Execution_Time_Ms : 2.0
Standard_Deviation_Ms              : 885.803234847289
Total_Serialized_Execution_Time_s  : 3323.606
Mean_Query_Execution_Time_s        : 0.0895006328261747
Median_Predicate_Execution_Time_s  : 0.002
Percentile95_Ms                    : 206.29999999999563
Number_of_Predicates               : 37135

Release                            : v2.20.0
Platform                           : x86-windows
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 3698496
Mean_Predicate_Execution_Time_Ms   : 90.41009093575828
Median_Predicate_Execution_Time_Ms : 2.0
Standard_Deviation_Ms              : 1369.993033711765
Total_Serialized_Execution_Time_s  : 3698.496
Mean_Query_Execution_Time_s        : 0.0904100909357582
Median_Predicate_Execution_Time_s  : 0.002
Percentile95_Ms                    : 183.0
Number_of_Predicates               : 40908

Release                            : v2.21.0
Platform                           : x86-windows
Language                           : c
Total_Serialized_Execution_Time_Ms : 3343159
Mean_Predicate_Execution_Time_Ms   : 90.05384656825773
Median_Predicate_Execution_Time_Ms : 2.0
Standard_Deviation_Ms              : 895.0686158000931
Total_Serialized_Execution_Time_s  : 3343.159
Mean_Query_Execution_Time_s        : 0.0900538465682577
Median_Predicate_Execution_Time_s  : 0.002
Percentile95_Ms                    : 208.0
Number_of_Predicates               : 37124

Release                            : v2.21.0
Platform                           : x86-linux
Language                           : c
Total_Serialized_Execution_Time_Ms : 2958950
Mean_Predicate_Execution_Time_Ms   : 80.16010619564923
Median_Predicate_Execution_Time_Ms : 1.0
Standard_Deviation_Ms              : 793.7839873192156
Total_Serialized_Execution_Time_s  : 2958.95
Mean_Query_Execution_Time_s        : 0.0801601061956492
Median_Predicate_Execution_Time_s  : 0.001
Percentile95_Ms                    : 189.0
Number_of_Predicates               : 36913

Release                            : v2.21.0
Platform                           : x86-linux
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 3230480
Mean_Predicate_Execution_Time_Ms   : 78.4916296134315
Median_Predicate_Execution_Time_Ms : 1.0
Standard_Deviation_Ms              : 1199.1949622346183
Total_Serialized_Execution_Time_s  : 3230.48
Mean_Query_Execution_Time_s        : 0.0784916296134315
Median_Predicate_Execution_Time_s  : 0.001
Percentile95_Ms                    : 160.0
Number_of_Predicates               : 41157

Release                            : v2.21.0
Platform                           : x86-windows
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 3575250
Mean_Predicate_Execution_Time_Ms   : 87.56643562173944
Median_Predicate_Execution_Time_Ms : 2.0
Standard_Deviation_Ms              : 1351.655128024238
Total_Serialized_Execution_Time_s  : 3575.25
Mean_Query_Execution_Time_s        : 0.0875664356217394
Median_Predicate_Execution_Time_s  : 0.002
Percentile95_Ms                    : 175.0
Number_of_Predicates               : 40829

Release                            : 343
Platform                           : x86-linux
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 3068114
Mean_Predicate_Execution_Time_Ms   : 70.59139958125301
Median_Predicate_Execution_Time_Ms : 1.0
Standard_Deviation_Ms              : 915.0260824178082
Total_Serialized_Execution_Time_s  : 3068.114
Mean_Query_Execution_Time_s        : 0.070591399581253
Median_Predicate_Execution_Time_s  : 0.001
Percentile95_Ms                    : 180.0
Number_of_Predicates               : 43463

Release                            : 343
Platform                           : x86-linux
Language                           : c
Total_Serialized_Execution_Time_Ms : 2937489
Mean_Predicate_Execution_Time_Ms   : 76.1224442198554
Median_Predicate_Execution_Time_Ms : 1.0
Standard_Deviation_Ms              : 856.4970509347064
Total_Serialized_Execution_Time_s  : 2937.489
Mean_Query_Execution_Time_s        : 0.0761224442198554
Median_Predicate_Execution_Time_s  : 0.001
Percentile95_Ms                    : 184.0
Number_of_Predicates               : 38589


🏁 Below are the slowest predicates for the last 2 releases vs this PR.


Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-windows
Language          : cpp
Suite             : autosar-default
Predicate         : PreProcessorShallOnlyBeUsedForCertainDirectivesPatterns#32e68f1c::getAGuard#1#ff
Execution_Time_Ms : 30957

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : RepeatedInitializationOfAggregateObjectElement#76b22bdc::hasMultipleInitializerExprsForSameField#2#ff
Execution_Time_Ms : 55357

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-windows
Language          : cpp
Suite             : cert-default
Predicate         : SSAConstruction#2b11997e::DefUse::hasNonPhiDefinition#4#ffff
Execution_Time_Ms : 32596

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-windows
Language          : cpp
Suite             : autosar-default
Predicate         : m##DataFlowUtil#7572fbec::localFlowStep#2Plus#bf
Execution_Time_Ms : 146724

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-windows
Language          : cpp
Suite             : autosar-default
Predicate         : Dependency#b0c9183e::dependsOnTransitive#2#ff
Execution_Time_Ms : 180746

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-windows
Language          : cpp
Suite             : autosar-default
Predicate         : SSAConstruction#2b11997e::DefUse::hasNonPhiDefinition#4#ffff
Execution_Time_Ms : 29296

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-windows
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#123#fff
Execution_Time_Ms : 33035

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-windows
Language          : c
Suite             : misra-default
Predicate         : RepeatedInitializationOfAggregateObjectElement#76b22bdc::hasMultipleInitializerExprsForSameIndexInternal#4#ffff_3012#join_rhs
Execution_Time_Ms : 34287

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-windows
Language          : c
Suite             : misra-default
Predicate         : RepeatedInitializationOfAggregateObjectElement#76b22bdc::hasMultipleInitializerExprsForSameField#2#ff
Execution_Time_Ms : 51452

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-windows
Language          : c
Suite             : misra-default
Predicate         : _Literal#c5958f4a::ArrayOrVectorAggregateLiteral::getArraySize#0#dispred#bf_Literal#c5958f4a::ArrayO__#join_rhs
Execution_Time_Ms : 65074

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-windows
Language          : c
Suite             : cert-default
Predicate         : SSAConstruction#2b11997e::PhiInsertion::definitionHasRedefinition#3#fff
Execution_Time_Ms : 30844

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : UnusedIncludeDirectives#574b69bc::getANonLocalDependency#1#ff
Execution_Time_Ms : 26110

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : PreProcessorShallOnlyBeUsedForCertainDirectivesPatterns#32e68f1c::getAGuard#1#ff
Execution_Time_Ms : 30378

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : Dependency#b0c9183e::dependsOnTransitive#2#ff
Execution_Time_Ms : 139733

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : m##DataFlowUtil#7572fbec::localFlowStep#2Plus#bf
Execution_Time_Ms : 154688

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#123#fff
Execution_Time_Ms : 26663

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : _Contracts3#5e6bc86c::Contracts3Package::onlyTestErrnoRightAfterErrnoSettingFunctionQuery#0#f_Deviat__#antijoin_rhs
Execution_Time_Ms : 31364

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : RepeatedInitializationOfAggregateObjectElement#76b22bdc::hasMultipleInitializerExprsForSameIndexInternal#4#ffff_3012#join_rhs
Execution_Time_Ms : 33723

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : _Literal#c5958f4a::ArrayOrVectorAggregateLiteral::getArraySize#0#dispred#bf_Literal#c5958f4a::ArrayO__#join_rhs
Execution_Time_Ms : 50415

Release           : v2.21.0
Run               : 2023-08-02_20-20-14
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : SSAConstruction#2b11997e::DefUse::hasNonPhiDefinition#4#ffff
Execution_Time_Ms : 24291

Release           : 343
Run               : 2023-08-30_10-19-30
Platform          : x86-linux
Language          : c
Suite             : cert-default
Predicate         : DoNotSubtractPointersAddressingDifferentArrays#e86f6fc0::problems#8#ffffffff
Execution_Time_Ms : 49808

Release           : 343
Run               : 2023-08-30_10-19-30
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : DoNotSubtractPointersAddressingDifferentArrays#e86f6fc0::problems#8#ffffffff
Execution_Time_Ms : 45647

Release           : 343
Run               : 2023-08-30_10-19-30
Platform          : x86-linux
Language          : c
Suite             : cert-default
Predicate         : _@element#f_Declarations2#dda34f37::Declarations2Package::excessLengthNamesIdentifiersNotDistinctQue__#antijoin_rhs
Execution_Time_Ms : 40889

Release           : 343
Run               : 2023-08-30_10-19-30
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : _@element#f_Declarations1#f6cea675::Declarations1Package::externalIdentifiersNotDistinctQuery#0#f_De__#antijoin_rhs
Execution_Time_Ms : 37447

Release           : 343
Run               : 2023-08-30_10-19-30
Platform          : x86-linux
Language          : cpp
Suite             : cert-default
Predicate         : _@element#f_DataFlowUtil#7572fbec::TExprNode#ff_Deviations#ac004ef9::DeviationRecord::getACodeIdenti__#antijoin_rhs
Execution_Time_Ms : 30891

Release           : 343
Run               : 2023-08-30_10-19-30
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : m##DataFlowUtil#7572fbec::localFlowStep#2Plus#bf
Execution_Time_Ms : 88419

Release           : 343
Run               : 2023-08-30_10-19-30
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : #select#ffffffff
Execution_Time_Ms : 45004

Release           : 343
Run               : 2023-08-30_10-19-30
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : _@element#f_DataFlowUtil#7572fbec::TExprNode#ff_Deviations#ac004ef9::DeviationRecord::getACodeIdenti__#antijoin_rhs
Execution_Time_Ms : 30718

Release           : 343
Run               : 2023-08-30_10-19-30
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : Dependency#b0c9183e::dependsOnTransitive#2#ff
Execution_Time_Ms : 117564

Release           : 343
Run               : 2023-08-30_10-19-30
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#123#fff
Execution_Time_Ms : 66532


Copy link
Contributor

@jsinglet jsinglet left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work @lcartey and @kraiouchkine!

@jsinglet jsinglet added this pull request to the merge queue Aug 30, 2023
Merged via the queue into main with commit 9c7da19 Aug 30, 2023
22 checks passed
@jsinglet jsinglet deleted the lcartey/update-to-2.12 branch August 30, 2023 14:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

4 participants