Creating an AWS EKS cluster and solving kubectl connection issues (due to older awscli version)

Written by Claudio Kuenzler - 0 comments

Published on April 25th 2024 - Listed in AWS Cloud Kubernetes

After having just successfully created a Kubernetes cluster in AWS' managed Kubernetes service EKS (Amazon Elastic Kubernetes Service), my local kubectl was unable to communicate with the cluster. Time to find out why - but let's start at the beginning.

Creating a new EKS cluster

To create a new EKS cluster, it is recommended to use the eksctl command. This command can be downloaded from the GitHub releases:

ck@mint ~ $ ARCH=amd64
ck@mint ~ $ PLATFORM=$(uname -s)_$ARCH
ck@mint ~ $ curl -sLO "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_$PLATFORM.tar.gz"
ck@mint ~ $ curl -sL "https://github.com/eksctl-io/eksctl/releases/latest/download/eksctl_checksums.txt" | grep $PLATFORM | sha256sum --check
eksctl_Linux_amd64.tar.gz: OK
ck@mint ~ $ tar -xzf eksctl_$PLATFORM.tar.gz -C /tmp && rm eksctl_$PLATFORM.tar.gz
ck@mint ~ $ sudo mv /tmp/eksctl /usr/local/bin

The command should now be available:

ck@mint ~ $ eksctl version
0.176.0

To be able to communicate with the AWS account, we need to use the AWS cli and export the necessary keys and tokens (see How to use AWS CLI with MFA how to do this):

ck@mint ~ $ export AWS_ACCESS_KEY_ID="xxx"
ck@mint ~ $ export AWS_SECRET_ACCESS_KEY="xxx"
ck@mint ~ $ export AWS_SESSION_TOKEN="xxx="
ck@mint ~ $ aws ec2 describe-instances | head
{
"Reservations": []
}

The connection to the AWS accounts works (there are no EC2 instances created, yet), so now we can create the EKS cluster (eksctl uses the aws cli in the background to spin up the cluster's worker nodes as EC2 instances).

ck@mint ~ $ eksctl create cluster --name ck-test-cluster --version 1.28 --region eu-central-1 --nodegroup-name ck-test-nodes --nodes 3 --nodes-min 1 --nodes-max 4 --vpc-cidr 10.111.0.0/16 --tags Creator=CK,Org=Infiniroot,Environment=TEST --managed
2024-04-25 14:16:57 [i] eksctl version 0.176.0
2024-04-25 14:16:57 [i] using region eu-central-1
2024-04-25 14:16:57 [i] setting availability zones to [eu-central-1b eu-central-1a eu-central-1c]
2024-04-25 14:16:57 [i] subnets for eu-central-1b - public:10.111.0.0/19 private:10.111.96.0/19
2024-04-25 14:16:57 [i] subnets for eu-central-1a - public:10.111.32.0/19 private:10.111.128.0/19
2024-04-25 14:16:57 [i] subnets for eu-central-1c - public:10.111.64.0/19 private:10.111.160.0/19
2024-04-25 14:16:57 [i] nodegroup "ck-test-cluster" will use "" [AmazonLinux2/1.28]
2024-04-25 14:16:57 [i] using Kubernetes version 1.28
[...]
2024-04-25 14:32:55 [i] nodegroup "ck-test-nodes" has 3 node(s)
2024-04-25 14:32:55 [i] node "ip-10-111-16-180.eu-central-1.compute.internal" is ready
2024-04-25 14:32:55 [i] node "ip-10-111-42-190.eu-central-1.compute.internal" is ready
2024-04-25 14:32:55 [i] node "ip-10-111-67-91.eu-central-1.compute.internal" is ready
2024-04-25 14:32:55 [y] created 1 managed nodegroup(s) in cluster "ck-test-cluster"
2024-04-25 14:32:55 [x] getting Kubernetes version on EKS cluster: error running `kubectl version`: exit status 1 (check 'kubectl version')
2024-04-25 14:32:55 [i] cluster should be functional despite missing (or misconfigured) client binaries
2024-04-25 14:32:55 [y] EKS cluster "ck-test-cluster" in "eu-central-1" region is ready

This created a new EKS cluster named "ck-test-cluster" with Kubernetes version 1.28 in AWS region eu-central-1 (Frankfurt). A node group (the compute nodes) named ck-test-nodes with 3 nodes and a new VPC with internal network range 10.111.0.0/16 was created as well. You can also set some tags to help identify or describe a cluster.

Once the cluster is built, it should show up in eksctl:

ck@mint ~ $ eksctl get cluster
NAME REGION EKSCTL CREATED
ck-test-cluster eu-central-1 True

The next step is to use kubectl to communicate and deploy to the cluster.

Create a new standalone kubectl config file

The eksctl create command before should have added the new cluster configuration in .kube/config. However if you want to create a standalone config without other existing data/cluster, you can re-create the config from scratch:

ck@mint ~ $ mv .kube/config{,.bkp}
ck@mint ~ $ aws eks update-kubeconfig --region eu-central-1 --name ck-test-cluster
Added new context arn:aws:eks:eu-central-1:xxx:cluster/ck-test-cluster to /home/ck/.kube/config
ck@mint ~ $ export KUBECONFIG=~/.kube/config

Now with the kubectl config file written, we can start talking to the cluster. At least I thought so...

kubectl error: invalid apiVersion

My first check whether kubectl is able to communicate with the cluster is to list the nodes. But this already failed:

ck@mint ~ $ kubectl get nodes
error: exec plugin: invalid apiVersion "client.authentication.k8s.io/v1alpha1"

By taking a closer look into the config file it seems the mentioned invalid apiVersion was added under the user context:

ck@mint ~ $ cat .kube/config
apiVersion: v1
clusters:
[...]
users:
- name: arn:aws:eks:eu-central-1:xxx:cluster/ck-test-cluster
user:
    exec:
      apiVersion: client.authentication.k8s.io/v1alpha1
      args:
      - --region
      - eu-central-1
      - eks
      - get-token
      - --cluster-name
      - ck-test-cluster
      command: aws

My first guess was an incompatible kubectl version, but this turned out to be kubectl 1.29 which is just one version above the Kubernetes cluster (1.28) and should be compatible.

Maybe the v1alpha1 is the issue, as the v1alpha1 was removed in Kubernetes 1.24. Let's try v1beta1 instead.

After editing .kube/config and setting apiVersion: client.authentication.k8s.io/v1beta1, another error showed up:

ck@mint ~ $ kubectl get node
E0425 15:00:42.587159 76590 memcache.go:265] couldn't get current server API group list: Get "https://xxx.gr7.eu-central-1.eks.amazonaws.com/api?timeout=32s": getting credentials: decoding stdout: no kind "ExecCredential" is registered for version "client.authentication.k8s.io/v1alpha1" in scheme "pkg/client/auth/exec/exec.go:62"

Although I just changed v1alpha1 to v1beta1 in the kubectl config, the new error again mentions this outdated version tag. Where is this coming from?!

As I couldn't rule out an error during the cluster creation, I decided to delete the EKS cluster at this point:

ck@mint ~ $ eksctl delete cluster --name ck-test-cluster

It's an outdated aws cli!

After some research I finally came across a very helpful hint on a Stackoverflow question:

> You need to update your AWS CLI

The current version is installed as a deb package in version 1.22.34 - which seems dated.

ck@mint ~ $ dpkg -l awscli
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
+++-==============-============-============-==========================================
ii awscli 1.22.34-1 all Universal Command Line Environment for AWS

The AWS CLI documentation mentions that version 2 is the most recent version:

The AWS CLI version 2 is the most recent major version of the AWS CLI and supports all of the latest features. Some features introduced in version 2 are not backported to version 1 and you must upgrade to access those features.

On the other hand by looking at the PyPi project, the newest awscli available is currently version 1.32.91 from April 24th 2024 (one day old).

Install awscli from pypi project using pip3

Let's try using the newer awscli version from pypi first!

First remove the outdated system/deb package, then install the newest awscli version using pip3:

ck@mint ~ $ sudo apt-get remove awscli
ck@mint ~ $ sudo pip3 install awscli
Collecting awscli
[...]
Successfully installed awscli-1.32.91 botocore-1.34.91 docutils-0.16 rsa-4.7.2 s3transfer-0.10.1

Verify the new version is installed:

ck@mint ~ $ aws --version
aws-cli/1.32.91 Python/3.10.12 Linux/6.5.0-28-generic botocore/1.34.91

Re-create the cluster and new kubectl test

As I mentioned above, I already deleted the cluster. At this moment I'm actually not sure if a kubectl would have worked now or if the cluster itself was already created with outdated information from the old aws cli. So it would have been (likely) a wise idea anyway, to create a new EKS cluster from scratch.

Now re-create a new .kube/config file and then attempt the kubectl communication again:

ck@mint ~ $ rm .kube/config
ck@mint ~ $ aws eks update-kubeconfig --region eu-central-1 --name ck-test-cluster
Added new context arn:aws:eks:eu-central-1:xxx:cluster/ck-test-server to /home/ckadm/.kube/config

Let's check the contents of the kubectl config file:

ck@mint ~ $ cat /home/ckadm/.kube/config
apiVersion: v1
clusters:
[...]
users:
- name: arn:aws:eks:eu-central-1:xxx:cluster/ck-test-cluster
user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - eu-central-1
      - eks
      - get-token
      - --cluster-name
      - ck-test-cluster
      - --output
      - json
      command: aws

Hey, look at this! The apiVersion is now v1beta1!

What about kubectl now? Final attempt:

ck@mint ~ $ kubectl get node
NAME                                             STATUS   ROLES    AGE     VERSION
ip-10-111-29-59.eu-central-1.compute.internal    Ready    <none>   2m47s   v1.28.5-eks-5e0fdde
ip-10-111-40-114.eu-central-1.compute.internal   Ready    <none>   2m46s   v1.28.5-eks-5e0fdde
ip-10-111-64-119.eu-central-1.compute.internal   Ready    <none>   2m47s   v1.28.5-eks-5e0fdde

Heureka! The cluster's worker nodes are showing up; kubectl is now working!

Add a comment

Show form to leave a comment

Comments (newest first)

No comments yet.

Blog Tags:

AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Database Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Office PGSQL PHP Perl Personal PostgreSQL Postgres PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMWare VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Zoneminder