Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

RbacAuthorizationV1alpha1 deprecated from kubernetes v1.22 onwards - pytest fails with AttributeError #220

Open
MuraliK7hpe opened this issue Apr 13, 2022 · 6 comments
Open

RbacAuthorizationV1alpha1 deprecated from kubernetes v1.22 onwards - pytest fails with AttributeError #220

MuraliK7hpe opened this issue Apr 13, 2022 · 6 comments

Comments

@MuraliK7hpe
Copy link

kubetest fails to run in latest version on K8s and following is the error observed

File "/usr/local/lib/python3.6/site-packages/kubetest/objects/clusterrolebinding.py", line 12, in
class ClusterRoleBinding(ApiObject):
File "/usr/local/lib/python3.6/site-packages/kubetest/objects/clusterrolebinding.py", line 30, in ClusterRoleBinding
'rbac.authorization.k8s.io/v1alpha1': client.RbacAuthorizationV1alpha1Api,
AttributeError: module 'kubernetes.client' has no attribute 'RbacAuthorizationV1alpha1Api'

Refer to K8s page: https://kubernetes.io/docs/reference/using-api/deprecation-guide/

RBAC resources
The rbac.authorization.k8s.io/v1beta1 API version of ClusterRole, ClusterRoleBinding, Role, and RoleBinding is no longer served as of v1.22.

Migrate manifests and API clients to use the rbac.authorization.k8s.io/v1 API version, available since v1.8.
All existing persisted objects are accessible via the new APIs
No notable changes
@tyxeron
Copy link

tyxeron commented Apr 19, 2022

There are many other deprecations and issues. I tried to monkey patch the issue described above and got other errors of the same kind

@timblaktu
Copy link

timblaktu commented Apr 20, 2022 •
edited
Loading

I hit this error, pointing my tests' kubeconfig/context at a v1.21 k8s cluster.

TL;DR: How do I find the version of kubetest (and accompanying kubernetes client package) that officially supports kubernetes v1.21? Neither 0.9.5 nor 0.9.4 seem to fit the bill.

I have enabled debug logging, but do not see any output until the exception is hit (expand excerpt below for detail).

I can talk to this cluster just fine using kubectl. Since my cluster is using v1.21, perhaps my setup is trying to use a newer Api version? Is a way to specify the Api version that kubetest's client should use? I haven't found anything in the kubetest docs about this, and assume that it automatically detects the k8s version.

Click to expand pytest command and error 
$ poetry run pytest --kube-log-level=debug eks/test
Traceback (most recent call last):
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/bin/pytest", line 8, in <module>
  sys.exit(console_main())
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 187, in console_main
  code = main()
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 145, in main
  config = _prepareconfig(args, plugins)
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 324, in _prepareconfig
  config = pluginmanager.hook.pytest_cmdline_parse(
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/pluggy/_hooks.py", line 265, in __call__
  return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 80, in _hookexec
  return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 55, in _multicall
  gen.send(outcome)
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/helpconfig.py", line 102, in pytest_cmdline_parse    config: Config = outcome.get_result()
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/pluggy/_result.py", line 60, in get_result
  raise ex[1].with_traceback(ex[2])
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 39, in _multicall
  res = hook_impl.function(*args)
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1016, in pytest_cmdline_parse
  self.parse(args)
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1304, in parse
  self._preparse(args, addopts=addopts)
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/config/__init__.py", line 1187, in _preparse
  self.pluginmanager.load_setuptools_entrypoints("pytest11")
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints
  plugin = ep.load()
File "/usr/lib/python3.9/importlib/metadata.py", line 77, in load
  module = import_module(match.group('module'))
File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
  exec(co, module.__dict__)
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/plugin.py", line 17, in <module>
  from kubetest import errors, markers
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
  exec(co, module.__dict__)
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/markers.py", line 9, in <module>
  from kubetest import manager
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
  exec(co, module.__dict__)
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/manager.py", line 8, in <module>
  from kubetest import client, objects, utils
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
  exec(co, module.__dict__)
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/client.py", line 13, in <module>
  from kubetest import objects, utils
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
  exec(co, module.__dict__)
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/objects/__init__.py", line 6, in <module>
  from .clusterrolebinding import ClusterRoleBinding
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/_pytest/assertion/rewrite.py", line 168, in exec_module
  exec(co, module.__dict__)
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/objects/clusterrolebinding.py", line 12, in <module>
  class ClusterRoleBinding(ApiObject):
File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/objects/clusterrolebinding.py", line 30, in ClusterRoleBinding
  "rbac.authorization.k8s.io/v1alpha1": client.RbacAuthorizationV1alpha1Api,
AttributeError: module 'kubernetes.client' has no attribute 'RbacAuthorizationV1alpha1Api'

My test itself is trivial. Since it's failing before ever loading my test, I believe my test code is irrelevant.

I'm using python 3.9 and kubetest 0.9.5 on Debian 11 and the following python package versions installed via poetry:

Click to expand details about my environment 
$ poetry show
attrs              21.4.0    Classes Without Boilerplate
boto3              1.21.42   The AWS SDK for Python
botocore           1.24.42   Low-level, data-driven core of boto 3.
cachetools         5.0.0     Extensible memoizing collections and decorators
certifi            2021.10.8 Python package for providing Mozilla's CA Bundle.
cfgv               3.3.1     Validate configuration and produce human readable error messages.
charset-normalizer 2.0.12    The Real First Universal Charset Detector. Open, modern and actively maintained alternative to Chardet.
distlib            0.3.4     Distribution utilities
filelock           3.6.0     A platform independent file lock.
google-auth        2.6.5     Google Authentication Library
identify           2.4.12    File identification library for Python
idna               3.3       Internationalized Domain Names in Applications (IDNA)
iniconfig          1.1.1     iniconfig: brain-dead simple config-ini parsing
jmespath           1.0.0     JSON Matching Expressions
kubernetes         23.3.0    Kubernetes python client
kubetest           0.9.5     A Kubernetes integration test framework in Python.
nodeenv            1.6.0     Node.js virtual environment builder
oauthlib           3.2.0     A generic, spec-compliant, thorough implementation of the OAuth request-signing logic
packaging          21.3      Core utilities for Python packages
platformdirs       2.5.2     A small Python module for determining appropriate platform-specific dirs, e.g. a "user data dir".
pluggy             1.0.0     plugin and hook calling mechanisms for python
pre-commit         2.18.1    A framework for managing and maintaining multi-language pre-commit hooks.
py                 1.11.0    library with cross-python path, ini-parsing, io, code, log facilities
pyasn1             0.4.8     ASN.1 types and codecs
pyasn1-modules     0.2.8     A collection of ASN.1-based protocols modules.
pyparsing          3.0.8     pyparsing module - Classes and methods to define and execute parsing grammars
pytest             7.1.1     pytest: simple powerful testing with Python
python-dateutil    2.8.2     Extensions to the standard Python datetime module
pyyaml             6.0       YAML parser and emitter for Python
requests           2.27.1    Python HTTP for Humans.
requests-oauthlib  1.3.1     OAuthlib authentication support for Requests.
rsa                4.8       Pure-Python RSA implementation
s3transfer         0.5.2     An Amazon S3 Transfer Manager
six                1.16.0    Python 2 and 3 compatibility utilities
toml               0.10.2    Python Library for Tom's Obvious, Minimal Language
tomli              2.0.1     A lil' TOML parser
urllib3            1.26.9    HTTP library with thread-safe connection pooling, file post, and more.
virtualenv         20.14.1   Virtual Python Environment builder
websocket-client   1.3.2     WebSocket client for Python with low level API options

@timblaktu
Copy link

timblaktu commented Apr 20, 2022 •
edited
Loading

@MuraliK7hpe @tyxeron, it appears that the kubetest client object wrapping clusterrolebinding has a list of api versions, with specification for a "preferred" one:

class ClusterRoleBinding(ApiObject):
<snip>
    api_clients = {
        "preferred": client.RbacAuthorizationV1Api,
        "rbac.authorization.k8s.io/v1": client.RbacAuthorizationV1Api,
        "rbac.authorization.k8s.io/v1alpha1": client.RbacAuthorizationV1alpha1Api,
        "rbac.authorization.k8s.io/v1beta1": client.RbacAuthorizationV1beta1Api,
    }

Could there be a problem with this "preferred" mechanism? Why would we be getting this error if the client chose the preferred version first, i.e. rbac.authorization.k8s.io/v1?

EDIT: I found this description in the ApiObject.api_client comment:

A mapping of all the supported api clients for the API
object type. Various resources can have multiple versions,
e.g. "apps/v1", "apps/v1beta1", etc. The preferred version
for each resource type should be defined under the "preferred"
key. The preferred API client will be used when the apiVersion
is not specified for the resource.

This all appears to be the case for all the api objects in kubetest. I suspect we're just hitting clusterrolebinding first bc it's the first to be imported in the objects package.

@timblaktu
Copy link

timblaktu commented Apr 20, 2022 •
edited
Loading

For a second, I thought that these client api objects were choosing v1alpha1 because I have that specified in my kubeconfig file. I'm using EKS, and the aws eks update-kubeconfig command to generate the kubeconfig for me. In the generated file, it specifies to use v1alpha1 in the command to fetch the auth token for the generated user:

Click to expand user kubeconfig snippet generated by aws eks update-kubeconfig 
users:
- name: arn:aws:eks:<region>:<account>:cluster/<cluster name>
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - us-east-1
      - eks
      - get-token
      - --cluster-name
      - <cluster name>
      command: aws
      env:
      - name: AWS_PROFILE
        value: *****
      interactiveMode: IfAvailable
      provideClusterInfo: false

However, I'm still getting the same kubetest exception after changing the user-exec apiVersion in my kubeconfig file to v1. :-(

In fact, changing this api version in my kubeconfig also breaks kubectl. It appears that this api version is the only one supported in my cluster for the initial fetch of api versions that kubectl does:

error: couldn't get available api versions from server: Get "https://blahblahblah.eks.amazonaws.com/api?timeout=32s": getting credentials: exec plugin is configured to use API version client.authentication.k8s.io/v1beta1, plugin returned version client.authentication.k8s.io/v1alpha1

My cluster is v1.21, which supports the following api versions per-resource:

Click to expand kubectl api-versions 
~/src/infra2 530-implemen > kubectl api-versions                                                                                                                          admissionregistration.k8s.io/v1
admissionregistration.k8s.io/v1beta1
apiextensions.k8s.io/v1
apiextensions.k8s.io/v1beta1
apiregistration.k8s.io/v1
apiregistration.k8s.io/v1beta1
apps/v1
authentication.k8s.io/v1
authentication.k8s.io/v1beta1
authorization.k8s.io/v1
authorization.k8s.io/v1beta1
autoscaling/v1
autoscaling/v2beta1
autoscaling/v2beta2
batch/v1
batch/v1beta1
certificates.k8s.io/v1
certificates.k8s.io/v1beta1
coordination.k8s.io/v1
coordination.k8s.io/v1beta1
crd.k8s.amazonaws.com/v1alpha1
discovery.k8s.io/v1
discovery.k8s.io/v1beta1
events.k8s.io/v1
events.k8s.io/v1beta1
extensions/v1beta1
flowcontrol.apiserver.k8s.io/v1beta1
networking.k8s.io/v1
networking.k8s.io/v1beta1
node.k8s.io/v1
node.k8s.io/v1beta1
policy/v1
policy/v1beta1
rbac.authorization.k8s.io/v1
rbac.authorization.k8s.io/v1beta1
scheduling.k8s.io/v1
scheduling.k8s.io/v1beta1
secrets-store.csi.x-k8s.io/v1
secrets-store.csi.x-k8s.io/v1alpha1
storage.k8s.io/v1
storage.k8s.io/v1beta1
v1
vpcresources.k8s.aws/v1beta1

So, it appears that my cluster supports rbac.authorization.k8s.io api version v1 and v1beta1, but not v1alpha1.

I'm really confused about why kubetest is failing to access my v1.21 cluster.

@timblaktu
Copy link

timblaktu commented Apr 21, 2022 •
edited
Loading

OK, wading my way through this, I see:

  1. The error happens bc when populating the api_clients for ClusterRoleBinding class, the kubernetes client does not have a RbacAuthorizationV1alpha1Api member.
  2. The kubernetes client comes from the kubernetes package. On my system I have v23.3.0 of this package.
  3. Looking in kubernetes client v23.3.0 I see that it indeed only has RbacAuthorizationV1Api and no beta/alpha versions. THIS!

So, I think what's needed in my case is to downgrade the kubernetes package on my system.

Looking at the previous version of the kubernetes client package, I see that it indeed supports RbacAuthorizationV1alpha1Api as well as RbacAuthorizationV1Api.

EDIT: Ugh, I see that I cannot simply downgrade kubernetes client package because it is being installed as a dependency of kubetest. I will instead downgrade kubetest itself until I get a previous version of the kubernetes client that supports RbacAuthorizationV1alpha1Api.

@timblaktu
Copy link

Removing kubetest and reinstalling kubetest pinned to 0.9.4 and kubernetes pinned to 21.7.0 (the most recent version that appears to include v1alpha/beta1), I run into similar issue in DaemonSet Api versioning:

  File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/objects/daemonset.py", line 17, in <module>
    class DaemonSet(ApiObject):
  File "/home/tim/.cache/pypoetry/virtualenvs/ps-wheel-infrastructure--Poruhia-py3.9/lib/python3.9/site-packages/kubetest/objects/daemonset.py", line 35, in DaemonSet
    "apps/v1beta1": client.AppsV1beta1Api,
AttributeError: module 'kubernetes.client' has no attribute 'AppsV1beta1Api'

I believe this is what @tyxeron was referring to.

How do I find the version of kubetest (and accompanying kubernetes client package) that officially supports kubernetes v1.21?

@timblaktu timblaktu mentioned this issue Apr 21, 2022
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@timblaktu @tyxeron @MuraliK7hpe