Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update CodeQL CLI dependency to 2.11.6. #333

Merged
merged 48 commits into from
Aug 1, 2023
Merged

Conversation

lcartey
Copy link
Collaborator

@lcartey lcartey commented Jul 29, 2023

Description

This PR updates the CodeQL CLI dependency to 2.11.6, which includes updating the codeql/cpp-all dependency to 0.4.6.

I have made this update by merging the relevant commits from next into this branch. This should make it easier to identify unmerged commits in future (by comparing next with main). Additional commits have been made to ensure full compatibility with CodeQL 2.11.6.

The next commits included in this pull request includes those in PR:
#149

This was slightly premature, as the related codeql/cpp-all change shipped with 2.12.0. However, the change is backwards compatible, and so I've chosen to include it here for simplicity of merging.

Change request type

  • Release or process automation (GitHub workflows, internal scripts)
  • Internal documentation
  • External documentation
  • Query files (.ql, .qll, .qls or unit tests)
  • External scripts (analysis report or other code shipped as part of a release)

Rules with added or modified queries

  • No rules added
  • Queries have been added for the following rules:
    • rule number here
  • Queries have been modified for the following rules:
    • FIO32-C

Release change checklist

A change note (development_handbook.md#change-notes) is required for any pull request which modifies:

  • The structure or layout of the release artifacts.
  • The evaluation performance (memory, execution time) of an existing query.
  • The results of an existing query in any circumstance.

If you are only adding new rule queries, a change note is not required.

Author: Is a change note required?

  • Yes
  • No

🚨🚨🚨
Reviewer: Confirm that format of shared queries (not the .qll file, the
.ql file that imports it) is valid by running them within VS Code.

  • Confirmed

Reviewer: Confirm that either a change note is not required or the change note is required and has been added.

  • Confirmed

Query development review checklist

For PRs that add new queries or modify existing queries, the following checklist should be completed by both the author and reviewer:

Author

  • Have all the relevant rule package description files been checked in?
  • Have you verified that the metadata properties of each new query is set appropriately?
  • Do all the unit tests contain both "COMPLIANT" and "NON_COMPLIANT" cases?
  • Are the alert messages properly formatted and consistent with the style guide?
  • Have you run the queries on OpenPilot and verified that the performance and results are acceptable?
    As a rule of thumb, predicates specific to the query should take no more than 1 minute, and for simple queries be under 10 seconds. If this is not the case, this should be highlighted and agreed in the code review process.
  • Does the query have an appropriate level of in-query comments/documentation?
  • Have you considered/identified possible edge cases?
  • Does the query not reinvent features in the standard library?
  • Can the query be simplified further (not golfed!)

Reviewer

  • Have all the relevant rule package description files been checked in?
  • Have you verified that the metadata properties of each new query is set appropriately?
  • Do all the unit tests contain both "COMPLIANT" and "NON_COMPLIANT" cases?
  • Are the alert messages properly formatted and consistent with the style guide?
  • Have you run the queries on OpenPilot and verified that the performance and results are acceptable?
    As a rule of thumb, predicates specific to the query should take no more than 1 minute, and for simple queries be under 10 seconds. If this is not the case, this should be highlighted and agreed in the code review process.
  • Does the query have an appropriate level of in-query comments/documentation?
  • Have you considered/identified possible edge cases?
  • Does the query not reinvent features in the standard library?
  • Can the query be simplified further (not golfed!)

jketema and others added 30 commits July 15, 2022 21:50
Modernize qlpacks and update to CodeQL 2.8.5
Rename isPOD to isPod to silence deprecation warnings
`XMLElement` and `XMLFile` are deprecated from CodeQL 2.10.4 onwards.
`XmlElement` and `XmlFile` are their direct replacements.
Replace `XMLElement` by `XmlElement` and `XMLFile` by `XmlFile`
jketema and others added 18 commits December 6, 2022 17:13
C++: Pull in the latest version of `TaintedPath.ql` from CodeQL
C++: Accept test changes after CodeQL `scanf` work
Fix references to pack names after pack renaming
This matches the v2.11.6 CodeQL CLI bundle.
Since 2.11.3 pre-release packs have been ignored CodeQL when installing
packs. We re-enable this since we use pre-release packs internally (e.g.
in the common packs).
This reverts commit 760ccde.

This was prematurely included, but was only shipped in 2.12.*.
These were already released with a previous version.
@lcartey lcartey marked this pull request as ready for review July 30, 2023 22:24
@lcartey
Copy link
Collaborator Author

lcartey commented Jul 30, 2023

/test-performance

1 similar comment
@jsinglet
Copy link
Contributor

/test-performance

@github-actions
Copy link

🏁 Beep Boop! Performance testing for this PR has been initiated. Please check back later for results. Note that the query package generation step must complete before testing will start so it might be a minute.

💡 If you do not hear back from me please check my status! I will report even if I fail!

@jsinglet
Copy link
Contributor

🏁 Beep Boop! Performance testing complete! See below for performance of the last 3 runs vs your PR. Times are based on predicate performance. You can find full graphs and stats in the PR that was created for this test in the release engineering repo.


Release                            : v2.19.0
Platform                           : x86-windows
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 3631890
Mean_Predicate_Execution_Time_Ms   : 87.82651802771262
Median_Predicate_Execution_Time_Ms : 2.0
Standard_Deviation_Ms              : 1393.9322266043052
Total_Serialized_Execution_Time_s  : 3631.89
Mean_Query_Execution_Time_s        : 0.0878265180277126
Median_Predicate_Execution_Time_s  : 0.002
Percentile95_Ms                    : 172.0
Number_of_Predicates               : 41353

Release                            : v2.19.0
Platform                           : x86-windows
Language                           : c
Total_Serialized_Execution_Time_Ms : 3382360
Mean_Predicate_Execution_Time_Ms   : 90.76506104924192
Median_Predicate_Execution_Time_Ms : 2.0
Standard_Deviation_Ms              : 904.0785242728956
Total_Serialized_Execution_Time_s  : 3382.36
Mean_Query_Execution_Time_s        : 0.0907650610492419
Median_Predicate_Execution_Time_s  : 0.002
Percentile95_Ms                    : 207.0
Number_of_Predicates               : 37265

Release                            : v2.19.0
Platform                           : x86-linux
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 3233900
Mean_Predicate_Execution_Time_Ms   : 78.71434134943044
Median_Predicate_Execution_Time_Ms : 1.0
Standard_Deviation_Ms              : 1206.420132498584
Total_Serialized_Execution_Time_s  : 3233.9
Mean_Query_Execution_Time_s        : 0.0787143413494304
Median_Predicate_Execution_Time_s  : 0.001
Percentile95_Ms                    : 157.0
Number_of_Predicates               : 41084

Release                            : v2.19.0
Platform                           : x86-linux
Language                           : c
Total_Serialized_Execution_Time_Ms : 2971962
Mean_Predicate_Execution_Time_Ms   : 80.31244426428862
Median_Predicate_Execution_Time_Ms : 1.0
Standard_Deviation_Ms              : 795.5116560265172
Total_Serialized_Execution_Time_s  : 2971.962
Mean_Query_Execution_Time_s        : 0.0803124442642886
Median_Predicate_Execution_Time_s  : 0.001
Percentile95_Ms                    : 183.0
Number_of_Predicates               : 37005

Release                            : v2.20.0
Platform                           : x86-windows
Language                           : c
Total_Serialized_Execution_Time_Ms : 3323606
Mean_Predicate_Execution_Time_Ms   : 89.50063282617477
Median_Predicate_Execution_Time_Ms : 2.0
Standard_Deviation_Ms              : 885.803234847289
Total_Serialized_Execution_Time_s  : 3323.606
Mean_Query_Execution_Time_s        : 0.0895006328261747
Median_Predicate_Execution_Time_s  : 0.002
Percentile95_Ms                    : 206.29999999999563
Number_of_Predicates               : 37135

Release                            : v2.20.0
Platform                           : x86-linux
Language                           : c
Total_Serialized_Execution_Time_Ms : 2983369
Mean_Predicate_Execution_Time_Ms   : 80.49019290435722
Median_Predicate_Execution_Time_Ms : 1.0
Standard_Deviation_Ms              : 812.7769764280572
Total_Serialized_Execution_Time_s  : 2983.369
Mean_Query_Execution_Time_s        : 0.0804901929043572
Median_Predicate_Execution_Time_s  : 0.001
Percentile95_Ms                    : 183.0
Number_of_Predicates               : 37065

Release                            : v2.20.0
Platform                           : x86-linux
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 3271097
Mean_Predicate_Execution_Time_Ms   : 79.19756434157325
Median_Predicate_Execution_Time_Ms : 1.0
Standard_Deviation_Ms              : 1216.7021436787627
Total_Serialized_Execution_Time_s  : 3271.097
Mean_Query_Execution_Time_s        : 0.0791975643415732
Median_Predicate_Execution_Time_s  : 0.001
Percentile95_Ms                    : 156.0
Number_of_Predicates               : 41303

Release                            : v2.20.0
Platform                           : x86-windows
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 3698496
Mean_Predicate_Execution_Time_Ms   : 90.41009093575828
Median_Predicate_Execution_Time_Ms : 2.0
Standard_Deviation_Ms              : 1369.993033711765
Total_Serialized_Execution_Time_s  : 3698.496
Mean_Query_Execution_Time_s        : 0.0904100909357582
Median_Predicate_Execution_Time_s  : 0.002
Percentile95_Ms                    : 183.0
Number_of_Predicates               : 40908

Release                            : 333
Platform                           : x86-linux
Language                           : cpp
Total_Serialized_Execution_Time_Ms : 3223358
Mean_Predicate_Execution_Time_Ms   : 78.53420719228146
Median_Predicate_Execution_Time_Ms : 1.0
Standard_Deviation_Ms              : 1194.7120047258757
Total_Serialized_Execution_Time_s  : 3223.358
Mean_Query_Execution_Time_s        : 0.0785342071922814
Median_Predicate_Execution_Time_s  : 0.001
Percentile95_Ms                    : 159.84999999999854
Number_of_Predicates               : 41044

Release                            : 333
Platform                           : x86-linux
Language                           : c
Total_Serialized_Execution_Time_Ms : 3013822
Mean_Predicate_Execution_Time_Ms   : 81.21758111458446
Median_Predicate_Execution_Time_Ms : 1.0
Standard_Deviation_Ms              : 811.2186999231606
Total_Serialized_Execution_Time_s  : 3013.822
Mean_Query_Execution_Time_s        : 0.0812175811145844
Median_Predicate_Execution_Time_s  : 0.001
Percentile95_Ms                    : 190.0
Number_of_Predicates               : 37108


🏁 Below are the slowest predicates for the last 2 releases vs this PR.


Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-windows
Language          : cpp
Suite             : autosar-default
Predicate         : UnusedIncludeDirectives#574b69bc::getANonLocalDependency#1#ff
Execution_Time_Ms : 30264

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : RepeatedInitializationOfAggregateObjectElement#76b22bdc::hasMultipleInitializerExprsForSameField#2#ff
Execution_Time_Ms : 55378

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-windows
Language          : cpp
Suite             : cert-default
Predicate         : SSAConstruction#2b11997e::DefUse::hasNonPhiDefinition#4#ffff
Execution_Time_Ms : 31204

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-windows
Language          : cpp
Suite             : autosar-default
Predicate         : m##DataFlowUtil#7572fbec::localFlowStep#2Plus#bf
Execution_Time_Ms : 150906

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-windows
Language          : cpp
Suite             : autosar-default
Predicate         : Dependency#b0c9183e::dependsOnTransitive#2#ff
Execution_Time_Ms : 180361

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-windows
Language          : cpp
Suite             : autosar-default
Predicate         : SSAConstruction#2b11997e::DefUse::hasNonPhiDefinition#4#ffff
Execution_Time_Ms : 29897

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-windows
Language          : c
Suite             : cert-default
Predicate         : SSAConstruction#2b11997e::PhiInsertion::definitionHasRedefinition#3#fff
Execution_Time_Ms : 28852

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-windows
Language          : c
Suite             : misra-default
Predicate         : RepeatedInitializationOfAggregateObjectElement#76b22bdc::hasMultipleInitializerExprsForSameIndexInternal#4#ffff_3012#join_rhs
Execution_Time_Ms : 32749

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-windows
Language          : c
Suite             : misra-default
Predicate         : RepeatedInitializationOfAggregateObjectElement#76b22bdc::hasMultipleInitializerExprsForSameField#2#ff
Execution_Time_Ms : 51380

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-windows
Language          : c
Suite             : misra-default
Predicate         : RepeatedInitializationOfAggregateObjectElement#76b22bdc::hasMultipleInitializerExprsForSameIndexInternal#4#ffff#join_rhs
Execution_Time_Ms : 64522

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-windows
Language          : c
Suite             : cert-default
Predicate         : SSAConstruction#2b11997e::DefUse::hasNonPhiDefinition#4#ffff
Execution_Time_Ms : 28650

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : UnusedIncludeDirectives#574b69bc::getANonLocalDependency#1#ff
Execution_Time_Ms : 26730

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : PreProcessorShallOnlyBeUsedForCertainDirectivesPatterns#32e68f1c::getAGuard#1#ff
Execution_Time_Ms : 30558

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : Dependency#b0c9183e::dependsOnTransitive#2#ff
Execution_Time_Ms : 137629

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : m##DataFlowUtil#7572fbec::localFlowStep#2Plus#bf
Execution_Time_Ms : 160547

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#123#fff
Execution_Time_Ms : 31011

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#1#f#antijoin_rhs
Execution_Time_Ms : 31775

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : RepeatedInitializationOfAggregateObjectElement#76b22bdc::hasMultipleInitializerExprsForSameIndexInternal#4#ffff_3012#join_rhs
Execution_Time_Ms : 34004

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : RepeatedInitializationOfAggregateObjectElement#76b22bdc::hasMultipleInitializerExprsForSameIndexInternal#4#ffff#join_rhs
Execution_Time_Ms : 51146

Release           : v2.20.0
Run               : 2023-07-20_15-00-48
Platform          : x86-linux
Language          : cpp
Suite             : cert-default
Predicate         : SSAConstruction#2b11997e::DefUse::hasNonPhiDefinition#4#ffff
Execution_Time_Ms : 25035

Release           : 333
Run               : 2023-07-31_16-26-12
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : _Literal#c5958f4a::ArrayOrVectorAggregateLiteral::getArraySize#0#dispred#bf_Literal#c5958f4a::ArrayO__#join_rhs
Execution_Time_Ms : 51896

Release           : 333
Run               : 2023-07-31_16-26-12
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : RepeatedInitializationOfAggregateObjectElement#76b22bdc::hasMultipleInitializerExprsForSameIndexInternal#4#ffff_3012#join_rhs
Execution_Time_Ms : 32428

Release           : 333
Run               : 2023-07-31_16-26-12
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : #select#cpe#123#fff
Execution_Time_Ms : 31545

Release           : 333
Run               : 2023-07-31_16-26-12
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : _Contracts3#5e6bc86c::Contracts3Package::onlyTestErrnoRightAfterErrnoSettingFunctionQuery#0#f_Deviat__#antijoin_rhs
Execution_Time_Ms : 30989

Release           : 333
Run               : 2023-07-31_16-26-12
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : UnusedIncludeDirectives#574b69bc::getANonLocalDependency#1#ff
Execution_Time_Ms : 25980

Release           : 333
Run               : 2023-07-31_16-26-12
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : Dependency#b0c9183e::dependsOnTransitive#2#ff
Execution_Time_Ms : 138282

Release           : 333
Run               : 2023-07-31_16-26-12
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : PreProcessorShallOnlyBeUsedForCertainDirectivesPatterns#32e68f1c::getAGuard#1#ff
Execution_Time_Ms : 29930

Release           : 333
Run               : 2023-07-31_16-26-12
Platform          : x86-linux
Language          : cpp
Suite             : cert-default
Predicate         : SSAConstruction#2b11997e::DefUse::hasNonPhiDefinition#4#ffff
Execution_Time_Ms : 25079

Release           : 333
Run               : 2023-07-31_16-26-12
Platform          : x86-linux
Language          : cpp
Suite             : autosar-default
Predicate         : m##DataFlowUtil#7572fbec::localFlowStep#2Plus#bf
Execution_Time_Ms : 153982

Release           : 333
Run               : 2023-07-31_16-26-12
Platform          : x86-linux
Language          : c
Suite             : misra-default
Predicate         : RepeatedInitializationOfAggregateObjectElement#76b22bdc::hasMultipleInitializerExprsForSameField#2#ff
Execution_Time_Ms : 55940


Copy link
Contributor

@mbaluda mbaluda left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, no regression in performance 🚀

@lcartey lcartey added this pull request to the merge queue Aug 1, 2023
Merged via the queue into main with commit c2b14e0 Aug 1, 2023
22 checks passed
@lcartey lcartey deleted the lcartey/update-to-2.11 branch August 1, 2023 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Development

Successfully merging this pull request may close these issues.

6 participants