sudo yum install -y nvidia-container-toolkit failed - No such device

[howtoadd]

I am using AWS EC2(Tesla T4) , I think nvidia diver has been installed by default.

Run nvidia-smi get proper outputs.

Thu Nov 9 07:36:11 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03 Driver Version: 535.54.03 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla T4 Off | 00000000:00:1E.0 Off | 0 |
| N/A 30C P8 10W / 70W | 2MiB / 15360MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+

System Info
cat /etc/os-release

NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3⭕amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"

But When I install nvidia-container-toolkit got errors !!!!
install nvidia-container-toolkit following the guide

step 1 (success) : curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | > sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
outputs:

[nvidia-container-toolkit]
name=nvidia-container-toolkit
baseurl=https://nvidia.github.io/libnvidia-container/stable/rpm/$basearch
repo_gpgcheck=1
gpgcheck=0
enabled=1
gpgkey=https://nvidia.github.io/libnvidia-container/gpgkey
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

[nvidia-container-toolkit-experimental]
name=nvidia-container-toolkit-experimental
baseurl=https://nvidia.github.io/libnvidia-container/experimental/rpm/$basearch
repo_gpgcheck=1
gpgcheck=0
enabled=0
gpgkey=https://nvidia.github.io/libnvidia-container/gpgkey
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt

step 2(failed): sudo yum install -y nvidia-container-toolkit
Loaded plugins: dkms-build-requires, nvidia, priorities, update-motd, versionlock
neuron | 2.9 kB 00:00:00
nvidia-container-toolkit/x86_64/signature | 833 B 00:00:00

Retrieving key from https://nvidia.github.io/libnvidia-container/gpgkey
nvidia-container-toolkit/x86_64/signature | 2.1 kB 00:00:02 !!!
Traceback (most recent call last):
File "/bin/yum", line 29, in
yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 375, in user_main
errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 184, in main
result, resultmsgs = base.doCommands()
File "/usr/share/yum-cli/cli.py", line 584, in doCommands
return self.yum_cli_commands[self.basecmd].doCommand(self, self.basecmd, self.extcmds)
File "/usr/share/yum-cli/yumcommands.py", line 446, in doCommand
return base.installPkgs(extcmds, basecmd=basecmd)
File "/usr/share/yum-cli/cli.py", line 1016, in installPkgs
txmbrs = self.install(pattern=arg)
File "/usr/lib/python2.7/site-packages/yum/init.py", line 4827, in install
mypkgs = self.pkgSack.returnPackages(patterns=pats,
File "/usr/lib/python2.7/site-packages/yum/init.py", line 1074, in
pkgSack = property(fget=lambda self: self._getSacks(),
File "/usr/lib/python2.7/site-packages/yum/init.py", line 778, in _getSacks
self.repos.populateSack(which=repos)
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 347, in populateSack
self.doSetup()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 157, in doSetup
self.retrieveAllMD()
File "/usr/lib/python2.7/site-packages/yum/repos.py", line 88, in retrieveAllMD
dl = repo._async and repo._commonLoadRepoXML(repo)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1553, in _commonLoadRepoXML
result = self._getFileRepoXML(local, text)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1330, in _getFileRepoXML
size=102400) # setting max size as 100K
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1105, in _getFile
**kwargs
File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 448, in urlgrab
return self._mirror_try(func, url, kw)
File "/usr/lib/python2.7/site-packages/urlgrabber/mirror.py", line 425, in _mirror_try
return func_ref( *(fullurl,), opts=opts, **kw )
File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1216, in urlgrab
return self._retry(opts, retryfunc, url, filename)
File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1105, in _retry
r = apply(func, (opts,) + args, {})
File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1210, in retryfunc
_run_callback(opts.checkfunc, obj)
File "/usr/lib/python2.7/site-packages/urlgrabber/grabber.py", line 1073, in _run_callback
return cb(obj, *arg, **karg)
File "/usr/lib/python2.7/site-packages/yum/yumRepo.py", line 1802, in _checkRepoXML
self.gpg_import_func(self, self.confirm_func)
File "/usr/lib/python2.7/site-packages/yum/init.py", line 6420, in getKeyForRepo
self._getAnyKeyForRepo(repo, repo.gpgdir, repo.gpgkey, is_cakey=False, callback=callback)
File "/usr/lib/python2.7/site-packages/yum/init.py", line 6339, in _getAnyKeyForRepo
if hex(int(info['keyid']))[2:-1].upper() in misc.return_keyids_from_pubring(destdir):
File "/usr/lib/python2.7/site-packages/yum/misc.py", line 623, in return_keyids_from_pubring
for k in ctx.keylist():
gpgme.GpgmeError: (7, 32848, u'No such device')