Manage Your AWS EKS Load Balancer Like a Pro
AWS Load Balancer advanced tips & tricks
My last article described setting up an Ingress Controller in an AWS-managed Kubernetes cluster (EKS). In this article, I want to highlight some advanced features that might otherwise be neglected but will significantly impact the level of automation inside your cluster.
The tips and tricks you see below are designed to adhere to DRY, one of the essential principles in software engineering.
What is an Ingress "Controller" Again?
Kubernetes has been designed and developed with a bunch of "controllers" responsible for bringing the current state to the desired state defined by the configurations and resources inside the cluster [source].
As a practical example, if you scale your Nginx deployment to 2 replicas, the responsible controller will talk to the Kubelet running on the worker node to spin up two replicas of the Nginx image, scaling up or down as needed.
This opens up a world of possibilities when it comes to adding your logic to the Kubernetes workload.
One of those possibilities we'll dive deep into today is the Ingress "Controller," responsible for routing traffic coming to your cluster to the proper Service with configured rules.
What is an Operator in Kubernetes?
Before we jump into the main discussion, defining an Operator is crucial, but if you already know the pattern, you can safely skip this section.
An Operator in a Kubernetes cluster is a set of Custom Resource Definitions and a "controller" that will be responsible for bringing the CRD's current state to the desired state, much like the same behavior as with other Kubernetes controllers defined above.
As a dummy example, consider you define the following CRD in your cluster.
This may not be the best sandwich on the planet. Still, once your controller is deployed in the cluster, every API call made to the Kubernetes api-server on the foo.bar endpoint, then by applying the above YAML file, you will have a burger sandwich in your cluster.
If you're interested in knowing more, jump into the docs for a complete reference.
Why Does it Matter to Know About Operators Here?
Kubernetes doesn't ship with any Ingress Controllers [source]. But it allows the community to write their Ingress Controllers, which may or may not have a set of their CRDs; hence the importance of knowing the Operator pattern!
AWS Load Balancer Controller is an implementation of the Operator pattern which listens for Ingress resources inside the cluster and, upon creation of such, will perform a bunch of API calls to create the right TargetGroupBinding resources.
The
TargetGroupBindingfills the gap between Kubernetes workload and AWS infrastructure because of the way AWS has designed and architected its Load Balancer & upstreams behind those load balancers. If you want to read more, here is the link to the reference guide.
There is a one-to-one mapping between Target Group and TargetGroupBindings.
Target Group is a group of one or more EC2 instances that sit behind an AWS Application Load Balancer [source].
You can see an example of a TargetGroupBinding resource below.
This is not a resource you'd typically create/manage on your own but will instead be maintained by the AWS Load Balancer Controller right when you create/modify an Ingress resource.
Knowing this here, you should remember one important note when you try to create a Service with an Ingress to receive traffic from the internet. Your Service resource should be of type NodePort for the Ingress Controller to be able to create the TargetGroupBindings. It makes sense in AWS's world because if the Service is only exposed as a ClusterIP, the AWS Load Balancer cannot talk to that since it's only exposed inside the cluster and is effectively inaccessible from even the host worker node itself.
You can read more about the discussion in my last article from the link below or head over to the reference guide to understanding the design and configurations.
Now that we have covered the basics and prerequisites, it's time to talk about the controller and CRD, which is how you'd want to configure your AWS Load Balancer.
What is the Role of an IngressClass?
An IngressClass is a resource that will delegate the responsibility of managing Ingress resources to a specified controller. To clarify the statement, let's bring up some code.
IngressClass will be created for you if you follow the installation guide.As you can see from above, two essential attributes are under the spec section. The controller is to determine who/what will handle the Ingress resources by this IngressClass, and the parameters are optional key values you pass to the same controller to customize its behavior.
Having
parametersallows having the samecontrollerwith different flavors, customized to handle different Ingress resources differently.
In plain English, if I create an Ingress resource with ingressClassName set to aws-alb, the controller named ingress.k8s.aws/alb will do what it has to in order to route the incoming traffic to the upstream Kubernetes Service.
The critical information we will need for the remainder of the discussion is placed under the spec.parameters.kind. The values of this attribute is IngressClassParams , a CRD that the AWS Load Balancer Controller will use to create the ultimate Load Balancer for you.
IngressClassParams is Your Best Friend
The AWS Load Balancer Controller has its own set of CRDs. The most crucial CRD that you should be aware of is named IngressClassParams and it has a bunch of beneficial configurable attributes that affect how the ultimate AWS Load Balancer will be created/configured.
These attributes allow you to define and customize your AWS Load Balancer that will act as a frontier to your applications.
NOTE: I’m talking about Application Load Balancer in this article. If you have a requirement to use the Network Load Balancer, you can use the same controller but be sure to check this link for the corresponding information.
Now, let us introduce the specification API provided by IngressClassParams [source]. Nothing works best without a good example, so let's bring the following to back up the argument.
The above configurations will lead to the following Load Balancer if you open your AWS Console.
Some of these attributes are critical when working in a production-grade environment, while others are just nice. We'll dive deeper into each of them below.
You can specify the following attributes on the IngressClassParams resource.
spec.group.name: This "grouping" is what makes it possible to have multiple Ingress resource sittings behind one single Load Balancer. The alternative, if not provided at theIngressClassParamslevel, is to annotate every one of your Ingress resources [source]. If you don't have "grouping" for the Ingress resources, the AWS will create one load balancer per Ingress resource, which will cost an awful lot.spec.scheme: Since it is possible to have internal and internet-facing Load Balancers in AWS, you can specify your choice here. The distinction is apparent, one being only available to your internal resources and the other exposed on the public internet.spec.loadBalancerAttributes: One of the most valuable specifications will give you enough flexibility to customize your Load Balancer according to your need. This spec is what this article is all about, so we'll get back to it again.spec.ipAddressType: You have the option to choose fromipv4anddualstack. In case the world runs out of IPv4 someday!
Load Balancer Attributes
Referring to the screenshot taken from the AWS Console in the last section, as you can see from the Attributes section, you modify quite a few attributes over here. All these attributes are documented properly on their corresponding webpage. So, be sure to check it out for a comprehensive understanding.
To give a brief intro, here are the attributes configured from the IngressClassParams resource.
deletion_protection.enabled: Whether or not removing the one last Ingress resource will terminate the Load Balancer. You will likely want to turn this off because every new Load Balancer will get a new DNS Name, so you'll have to reconfigure your DNS records in or outside AWS DNS entries.idle_timeout.timeout_seconds: This is the number of seconds the Load Balancer will keep the two communication channels open between itself and the two sides of the connection (user & upstream server). For file uploads and similar large operations, you'd want to increase this amount and program your client in a way that will send at least one byte before this amount [source].routing.http.drop_invalid_header_fields.enabled: According to IANA, HTTP header names should conform to alphanumeric and hyphen characters. If you want your Load Balancer to drop the headers that do not correspond, turn this config on.routing.http2.enabled: As the name indicates, this will switch on the HTTP/2 protocol for your Load Balancer. Default istrue.routing.http.preserve_host_header.enabled: If set totrueThe Load Balancer will pass the HTTP request's Host header to the upstream server. This won't change the behavior ofX-FORWARDEDheader, though.access_logs.s3.enabled: This will turn on logging Load Balancer's access logs to S3.
These settings will only need to be specified once per Ingress group. If you install the Ingress Controller using Helm, you can set them as values during the installation. As a practical guide, here's a sample values.yaml file you will use, passing it to Helm installation.
If you don’t annotate any of your IngressClass as
defaultand you don’t specify anyingressClassNamein your Ingress resources, it is as if you don’t have any Ingress at all because no controller will pick that up!
Upon creating such IngressClass, any Ingress controlled by such will fall under the same group.name and effectively sit behind a single Load Balancer, saving you costs and giving you more management & control into the inwards traffic coming to your cluster.
If you'd instead not specify the group.name in the IngressClassParams, you will be required to annotate every Ingress like the following.
Now that we have covered the IngressClass, it's time to talk about the Ingress resources and your power over the Load Balancer by specifying the right set of annotations.
Ingress Annotations
The Ingress resources inside the Kubernetes cluster don't have anything particular. They are just a specification of one or more rule(s) that has to be treated and sent to the specified Service(s). You specify the host and path, and the corresponding Ingress Controller will handle the rest.
What makes them unique, though, is the metadata.annotations that gives more control over how you want to customize your Ingress when it sits behind any vendor or non-vendor Load Balancer.
To keep up with the tradition, let us start this section with another practical example and explain each annotation using the sample.
group.order: This integer specifies the priority of a rule to be matched against when a packet arrives at the Load Balancer. The Load Balancer will evaluate the rules from the lowest number to the highest. You'd generally want your most accessed URLs to be on top so that matching doesn't take too long for your most hit rules.- The annotations related to health checks are pretty straightforward and speak for themselves, so that I won't spend more time on them here.
listen.ports: This will be the ultimate Listener in your Load Balancer. If you don't specify this annotation, AWS Load Balancer Controller will expose the Ingress on port 80 by default. You will most likely want to handle every traffic on the secure port 443, known as HTTPS while redirecting everything to it from HTTP (port 80). The last part of this article is about this redirection!target-type: The default value is theinstance, meaning the load balancer will send the traffic to the EC2's exposed NodePort. Another allowed value isipwhich will send the traffic to the Pod's IP instead, but this option requires that the VPC CNI supports it [source].success-codes: Can be a single value e.g.200, multiple comma-separated values e.g.200,300, or a range of values e.g.200-300.
The annotations discussed above are the ones that you need to specify for every Ingress resource you create. Other annotations need to be set once, and the Load Balancer will pick that up and use it for all the Ingress resources. I will discuss these in the next & last section.
One-Time Configurations of the Load Balancer
Redirect HTTP to HTTPS
As mentioned in the previous article, you won't get any Load Balancer just by deploying the AWS Ingress Controller. You will need to create your first Ingress resource, having the IngressClass either specified in the spec or being the default IngressClass and then the AWS will provision your Load Balancer in a minute or two.
One of the security best practices is configuring the Load Balancer to redirect non-HTTPS traffic to the secure HTTPS one. To do that with AWS Ingress Controller, you will need to create an Ingress resource with the right set of annotations.
The annotation we need for this task is: actions.${action-name}. The "action name" will be a dummy Service name that doesn't even exist as a Kubernetes Service resource, but to fill the gap between AWS and Kubernetes, it needs to be specified under the spec.defaultBackend.service.name.
Here's what it will ultimately look like for an Ingress resource inside the AWS EKS to redirect the HTTP traffic to the HTTPS.
The annotations specified under the actions.ssl-redirect is a special one, only understandable by the AWS Ingress Controller, and although the JSON format may look a lot to process, it is readable, and you get an idea of what it does.
The important note to mention here is that the spec.defaultBackend.service.port.name should exactly be use-annotation and you can't put anything else in there if you want it to work as explained.
Another vital thing to re-mention here is that there is no Kubernetes Service resource in the cluster with the name ssl-redirect. We're only specifying that name here because to create this resource using kubectl, we have to fill in all the required attributes & specifications since it is going to be validated before sending to the API server, and even if you pass --validate=falseThe api-server itself will yell back at you about the wrong specification.
Ultimately, the above Ingress resource will redirect all the HTTP traffic to the HTTPS port, and to make sure that's the case, here's what it looks like in the AWS Console.
This Load Balancer only listens on port 80 at the moment, so let's define a dummy Service to redirect all the unwanted traffic there in case all the other rules do not match the incoming packets.
Every Load Balancer has a default backend rule that will determine where to send the traffic if all the prior rules didn’t match the incoming packet. This allows you to redirect your user to a friendly page reminding them that there is no such host or path and they need to make sure they are heading to the right address.
Dummy Upstream for Default Backend
This part is elementary and straightforward since you can customize it to your use case as you please. However, I will try to send all the traffics without specific rules to a friendly maintenance page you can see below.
The code and resources for this rule, including the previous SSL redirection, are available below.
Upon creation of the resources above, every incoming traffic that is not destined for your configured Ingress rules will be sent to the friendly maintenance page.
You also have the option to audit these incoming packets to know what is mostly being requested and act accordingly if you see anything suspicious.
Other non-Critical But Annotations
As the last piece of the puzzle and a bonus, here's a list of all the other annotations that might be useful for your workload.
Don't forget that these last annotations are only specified once per Ingress group. If you set multiple with different values, it will overwrite the previous value, and you might get unexpected results.
Conclusion
If you have opt-in for a managed Kubernetes cluster in AWS, buckle up for some tough challenges to address when it comes to having a production-grade cluster.
Kubernetes is meant to be cloud-agnostic, allowing you to have your database, cache, message broker, and everything else as a self-managed solution inside your cluster. This gives you more performance since your application will benefit from the geography closeness, but this also means that you will spend less money on buying RDS, ElastiCache, and other products offered by AWS. I'm pretty sure they don't like the sound of that.
That means you will face challenges outside the Kubernetes domain to make your cluster work in an AWS infrastructure, giving you learning and growth opportunity. I recommend taking as much as you possibly can, and be sure to enjoy the ride!
Have an excellent rest of the day, stay tuned and take care!
Reference
If you liked this article, check out my other contents that you might also find helpful.
