Benchmarking

[zregvart]

Implementation of a simple macro benchmark executing ec validate image with the fixed state and no external dependencies.

For this the OCI distribution registry is run in a container with the data from the data/registry/data and the git repository mirrored in the data/git/rhtap-ec-policy.git.

The benchmark outputs in the golang standard benchmark format that can be utilized with tools in the golang benchmarking ecosystem.

Reference: https://issues.redhat.com/browse/EC-968

Changes needed to be done to eliminate mutable global state that would influence the execution unfavorably (b0f2ecf).

Data needed for the benchmark to be local and deterministic is in a separate commit (eb2f1d6).

[codecov]

Codecov Report

Attention: Patch coverage is 18.75000% with 247 lines in your changes missing coverage. Please review.

Project coverage is 69.32%. Comparing base (b8c77a0) to head (eb2f1d6).

Files with missing lines	Patch %	Lines
benchmark/offliner/offliner.go	8.57%	128 Missing ⚠️
benchmark/simple/simple.go	0.00%	61 Missing ⚠️
benchmark/internal/registry/registry.go	0.00%	13 Missing ⚠️
cmd/root.go	0.00%	11 Missing ⚠️
benchmark/internal/suite/suite.go	0.00%	6 Missing ⚠️
internal/policy/source/git_config.go	0.00%	5 Missing ⚠️
internal/validate/helpers.go	44.44%	5 Missing ⚠️
internal/evaluator/conftest_evaluator.go	55.55%	4 Missing ⚠️
internal/policy/policy.go	42.85%	4 Missing ⚠️
cmd/fetch/fetch_policy.go	0.00%	3 Missing ⚠️
... and 3 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2192      +/-   ##
==========================================
- Coverage   71.21%   69.32%   -1.90%     
==========================================
  Files          88       92       +4     
  Lines        7501     7739     +238     
==========================================
+ Hits         5342     5365      +23     
- Misses       2159     2374     +215     

Flag	Coverage Δ
generative	69.32% <18.75%> (-1.90%)	⬇️
integration	69.32% <18.75%> (-1.90%)	⬇️
unit	69.32% <18.75%> (-1.90%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
cmd/inspect/inspect_policy.go	67.33% <100.00%> (+0.21%)	⬆️
cmd/validate/image.go	91.61% <100.00%> (+0.05%)	⬆️
cmd/validate/policy.go	92.15% <100.00%> (+0.15%)	⬆️
internal/image/validate.go	69.33% <100.00%> (+0.20%)	⬆️
internal/input/validate.go	77.77% <100.00%> (+0.24%)	⬆️
cmd/inspect/inspect_policy_data.go	35.13% <0.00%> (-0.32%)	⬇️
cmd/validate/input.go	42.02% <0.00%> (-0.23%)	⬇️
cmd/fetch/fetch_policy.go	0.00% <0.00%> (ø)
internal/policy/source/source.go	85.71% <86.36%> (+0.32%)	⬆️
internal/evaluator/conftest_evaluator.go	81.14% <55.55%> (ø)
... and 8 more