Long-lasting job fails on EKS [kubernetes.client.rest.ApiException: (401)]

[TonyZZX]

Description

When a pod runs for a long time (~30min), the process will throw a 401 Unauthorized error.

Files

test.py

import os
import time


def test_deployment(kube):
    f = os.path.join(
        os.path.dirname(os.path.realpath(__file__)),
        'configs',
        'deployment.yaml'
    )

    d = kube.load_deployment(f)

    kube.create(d)

    d.wait_until_ready(timeout=20)
    d.refresh()

    pods = d.get_pods()
    assert len(pods) == 1

    p = pods[0]
    p.wait_until_ready(timeout=10)

    print('')
    for i in range(30):
        time.sleep(60)
        print(f'slept {i + 1} minutes')

    kube.delete(d)
    d.wait_until_deleted(timeout=20)

configs/deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-deployment
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:1.7.9
        ports:
        - containerPort: 80

Environment

Kubernetes is running on EKS

platform darwin -- Python 3.7.4, pytest-5.0.1, py-1.8.0, pluggy-0.12.0

plugins: kubetest-0.3.2

Error message

Unable to get pods for namespace "kubetest-test-deployment-1563912434" to cache logs ((401)
Reason: Unauthorized
HTTP response headers: HTTPHeaderDict({'Audit-Id': '...', 'Content-Type': 'application/json', 'Date': 'Tue, 23 Jul 2019 20:37:18 GMT', 'Content-Length': '129'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
...
ERROR at teardown of test_deployment 
item = <Function test_deployment>

    def pytest_runtest_teardown(item):
        """Run teardown actions to clean up the test client.
    
        See Also:
            https://docs.pytest.org/en/latest/reference.html#_pytest.hookspec.pytest_runtest_teardown
        """
        if item.config.getoption('kube_config'):
>           manager.teardown(item.nodeid)

venv/lib/python3.7/site-packages/kubetest/plugin.py:252: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
venv/lib/python3.7/site-packages/kubetest/manager.py:361: in teardown
    test_case.teardown()
venv/lib/python3.7/site-packages/kubetest/manager.py:203: in teardown
    self.namespace.delete()
venv/lib/python3.7/site-packages/kubetest/objects/namespace.py:91: in delete
    body=options,
venv/lib/python3.7/site-packages/kubernetes/client/apis/core_v1_api.py:9032: in delete_namespace
    (data) = self.delete_namespace_with_http_info(name, **kwargs)
venv/lib/python3.7/site-packages/kubernetes/client/apis/core_v1_api.py:9126: in delete_namespace_with_http_info
    collection_formats=collection_formats)
venv/lib/python3.7/site-packages/kubernetes/client/api_client.py:334: in call_api
    _return_http_data_only, collection_formats, _preload_content, _request_timeout)
venv/lib/python3.7/site-packages/kubernetes/client/api_client.py:168: in __call_api
    _request_timeout=_request_timeout)
venv/lib/python3.7/site-packages/kubernetes/client/api_client.py:400: in request
    body=body)
venv/lib/python3.7/site-packages/kubernetes/client/rest.py:256: in DELETE
    body=body)
...
    def request(self, method, url, query_params=None, headers=None,
                body=None, post_params=None, _preload_content=True, _request_timeout=None):
...
        if _preload_content:
            r = RESTResponse(r)
    
            # In the python 3, the response.data is bytes.
            # we need to decode it to string.
            if PY3:
                r.data = r.data.decode('utf8')
    
            # log response body
            logger.debug("response body: %s", r.data)
    
        if not 200 <= r.status <= 299:
>           raise ApiException(http_resp=r)
E           kubernetes.client.rest.ApiException: (401)
E           Reason: Unauthorized
E           HTTP response headers: HTTPHeaderDict({'Audit-Id': '8f3dba8d-a57e-4b19-af23-90f6d327c148', 'Content-Type': 'application/json', 'Date': 'Tue, 23 Jul 2019 20:37:18 GMT', 'Content-Length': '129'})
E           HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}

venv/lib/python3.7/site-packages/kubernetes/client/rest.py:222: ApiException

[sohil-babylist]

Were you able to find any resolution to this?