How to configure Route53 Public Hosted Zone + ACM Certificate + External DNS in EKS for a custom domain
In the previous post, we deployed a sample application to a managed EKS environment in AWS.
In this article, we will,
- Create a Route53 Public Hosted Zone for a custom sub-domain (app.learnwithpras.xyz in this scenario)
- Generate a Public ACM certificate for the custom subdomain.
- Deploy External DNS to the EKS environment.
- Ensure application is reachable over HTTPS.
Without further ado, lets dive right into it!
Diagram
Route53 Public Hosted Zone
I own the domain learnwithpras.xyz
and I manage the domain via Cloudflare. Cloudflare manages all of my services, including other subdomains I might have and I only want DNS traffic for subdomain app.learnwithpras.xyz
to be directed to Amazon Route53 for the purpose of learning and this demo.
Navigate to Route53 in your AWS Account and click on Create Hosted Zone
create a public hosted zone for subdomain app.learnwithpras.xyz
.
For domain name, specify app.learnwithpras.xyz
and optionally fill description including addition of tags. Then, click on Create Hosted Zone
.
This is looking good but we want to leverage best practices as much as possible, so I am going to delete the hosted zone created manually and will use Cloudformation to deploy the public hosted zone. We will also store hosted zone ID in Systems Manager Parameter Store so that we can use it with other resources effectively. The code looks like,
---
AWSTemplateFormatVersion: 2010-09-09
Description: Create a public hosted zone for subdomain app.learnwithpras.xyz
Parameters:
DomainName:
Type: String
Description: Domain name to create public hosted zone for
Resources:
PrasDomainPublicHostedZone:
Type: AWS::Route53::HostedZone
Properties:
HostedZoneConfig:
Comment: !Sub 'Public hosted zone for subdomain ${DomainName}'
HostedZoneTags:
-
Key: Owner
Value: Pras
-
Key: Purpose
Value: Learning
Name: !Ref DomainName
PrasDomainPublicHostedZoneIDSsmParameter:
Type: AWS::SSM::Parameter
Properties:
Description: Parameter to store hosted zone ID of custom pras domain
Name: /pras/route53/app/public-hosted-zone/id
Type: String
Value: !GetAtt PrasDomainPublicHostedZone.Id
$ make deploy-hosted-zone
aws cloudformation deploy \
--s3-bucket pras-cloudformation-artifacts-bucket \
--template-file cloudformation/public-hosted-zone.yaml \
--stack-name pras-public-hosted-zone \
--capabilities CAPABILITY_NAMED_IAM \
--no-fail-on-empty-changeset \
--parameter-overrides \
DomainName=app.learnwithpras.xyz \
--tags \
Name='Route53 - Public Hosted Zone'
And we end up with the same result ๐
Now that we have our hosted zone setup on AWS side, we still need to tell Cloudflare to forward any DNS traffic for subdomain app.learnwithpras.xyz
to the Name Servers created by Route53 for our hosted zone.
I have prepared a small python script using Cloudflare library to add these NS records to Cloudflare zone learnwithpras.xyz
,
"""
Cloudflare API client.
"""
import CloudFlare
from dataclasses import dataclass
@dataclass
class PrasCloudflare:
"""
Cloudflare API client.
"""
api_key: str
base_url: str = "https://api.cloudflare.com/client/v4/"
def __post_init__(self):
self.cf = CloudFlare.CloudFlare(token=self.api_key)
def get_zones(self):
"""
Get Cloudflare zones.
"""
# add params = {'per_page':100} if you want to get more than 50 zones
return self.cf.zones.get()
def get_zone_id(self, zone_name):
"""
Get zone id.
"""
return self.cf.zones.get(params={"name": zone_name})[0]["id"]
def get_dns_records(self, zone_id):
"""
Get dns records for a cloudflare zone
"""
return self.cf.zones.dns_records.get(zone_id)
def add_dns_record(self, zone_id, dns_record):
"""
Add dns record for a cloudflare zone
"""
return self.cf.zones.dns_records.post(zone_id, data=dns_record)
if __name__ == "__main__":
zone_name = "learnwithpras.xyz"
dns_record_name = "app.learnwithpras.xyz"
init_cloudflare = PrasCloudflare(<api_token_does_here>)
zone_id = init_cloudflare.get_zone_id(zone_name)
# Add NS records to cloudflare
dns_records = [
{"name": f"{dns_record_name}", "type": "NS", "content": "ns-1084.awsdns-07.org."},
{"name": f"{dns_record_name}", "type": "NS", "content": "ns-1940.awsdns-50.co.uk."},
{"name": f"{dns_record_name}", "type": "NS", "content": "ns-457.awsdns-57.com."},
{"name": f"{dns_record_name}", "type": "NS", "content": "ns-952.awsdns-55.net."},
]
for dns_record in dns_records:
print(init_cloudflare.add_dns_record(zone_id, dns_record))
If we run the script, we can see that the NS records are successfully added to Cloudflare zone,
$ python3 src/cloudflare.py
{'id': '1ce28e7ef9e9c7e5e7976e917dc9246b', 'zone_id': '220cd9da0f4d9d90761523bfeac82643', 'zone_name': 'learnwithpras.xyz', 'name': 'app.learnwithpras.xyz', 'type': 'NS', 'content': 'ns-1084.awsdns-07.org', 'proxiable': False, 'proxied': False, 'ttl': 1, 'locked': False, 'meta': {'auto_added': False, 'managed_by_apps': False, 'managed_by_argo_tunnel': False, 'source': 'primary'}, 'created_on': '2021-09-09T21:08:59.312151Z', 'modified_on': '2021-09-09T21:08:59.312151Z'}
{'id': 'b086797c60b78cbd0149ee3b9efb877f', 'zone_id': '220cd9da0f4d9d90761523bfeac82643', 'zone_name': 'learnwithpras.xyz', 'name': 'app.learnwithpras.xyz', 'type': 'NS', 'content': 'ns-1940.awsdns-50.co.uk', 'proxiable': False, 'proxied': False, 'ttl': 1, 'locked': False, 'meta': {'auto_added': False, 'managed_by_apps': False, 'managed_by_argo_tunnel': False, 'source': 'primary'}, 'created_on': '2021-09-09T21:08:59.61448Z', 'modified_on': '2021-09-09T21:08:59.61448Z'}
{'id': 'ec4d60dbe20a5f355b57891d33ba675b', 'zone_id': '220cd9da0f4d9d90761523bfeac82643', 'zone_name': 'learnwithpras.xyz', 'name': 'app.learnwithpras.xyz', 'type': 'NS', 'content': 'ns-457.awsdns-57.com', 'proxiable': False, 'proxied': False, 'ttl': 1, 'locked': False, 'meta': {'auto_added': False, 'managed_by_apps': False, 'managed_by_argo_tunnel': False, 'source': 'primary'}, 'created_on': '2021-09-09T21:08:59.887883Z', 'modified_on': '2021-09-09T21:08:59.887883Z'}
{'id': 'df38fdfb41879e75b5203b9e7e7a8350', 'zone_id': '220cd9da0f4d9d90761523bfeac82643', 'zone_name': 'learnwithpras.xyz', 'name': 'app.learnwithpras.xyz', 'type': 'NS', 'content': 'ns-952.awsdns-55.net', 'proxiable': False, 'proxied': False, 'ttl': 1, 'locked': False, 'meta': {'auto_added': False, 'managed_by_apps': False, 'managed_by_argo_tunnel': False, 'source': 'primary'}, 'created_on': '2021-09-09T21:09:00.15107Z', 'modified_on': '2021-09-09T21:09:00.15107Z'}
(kubernetes)
If we run a dig command to fetch NS records for app.learnwithpras.xyz
,
$ dig app.learnwithpras.xyz ns
; <<>> DiG 9.10.6 <<>> app.learnwithpras.xyz ns
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54889
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;app.learnwithpras.xyz. IN NS
;; ANSWER SECTION:
app.learnwithpras.xyz. 172800 IN NS ns-1084.awsdns-07.org.
app.learnwithpras.xyz. 172800 IN NS ns-1940.awsdns-50.co.uk.
app.learnwithpras.xyz. 172800 IN NS ns-457.awsdns-57.com.
app.learnwithpras.xyz. 172800 IN NS ns-952.awsdns-55.net.
;; Query time: 139 msec
;; SERVER: 192.168.178.1#53(192.168.178.1)
;; WHEN: Fri Sep 10 07:17:52 AEST 2021
;; MSG SIZE rcvd: 190
Create Public ACM Certificate
Lets create an SSL certificate from AWS Certificate Manager. We will use this certificate later with NGINX controller's Network Load Balancer (created as part of previous article on application deployment ) and terminate TLS session at the NLB.
Cloudformation Template
---
AWSTemplateFormatVersion: 2010-09-09
Description: Deploy ACM Certificate to use with EKS application
Parameters:
DomainName:
Type: String
Description: Domain name to issue certificate for
HostedZoneId:
Type: AWS::SSM::Parameter::Value<String>
Description: Hosted Zone in use for the domain specified
Resources:
PrasAppAcmCertificate:
Type: AWS::CertificateManager::Certificate
Properties:
DomainName: !Ref DomainName
ValidationMethod: DNS
DomainValidationOptions:
- DomainName: !Ref DomainName
HostedZoneId: !Ref HostedZoneId
PrasAppAcmCertificateArnSsmParameter:
Type: AWS::SSM::Parameter
Properties:
Description: Eks cluster name parameter
Name: /pras/eks/app/certificate/arn
Type: String
Value: !Ref PrasAppAcmCertificate
Cloudformation parameters,
NOTE: AWS::SSM::Parameter::Value resolves an SSM parameter when we specify the path
- DomainName: app.learnwithpras.xyz - will be the domain you are using
- HostedZoneId: /pras/route53/app/public-hosted-zone/id - created with the hosted zone CFN stack earlier
aws cloudformation deploy \
--s3-bucket pras-cloudformation-artifacts-bucket \
--template-file cloudformation/acm-cert.yaml \
--stack-name pras-acm-cert \
--capabilities CAPABILITY_NAMED_IAM \
--no-fail-on-empty-changeset \
--parameter-overrides \
DomainName=app.learnwithpras.xyz \
HostedZoneId=/pras/route53/hosted-zone/id \
--tags \
Name='Kubernetes Cluster Resources - ACM Cert'
Cloudformation stack has been deployed successfully,
We used DNS validation method and so a CNAME record has been added to Route53 Hosted Zone by ACM,
Navigate to ACM Console to view the ACM certificate and its details,
Configure External DNS
External DNS makes Kubernetes resources discoverable over the public DNS servers. It maps Domain Names to Kubernetes resources dynamically by polling for changes in objects like Services and Ingresses.
We need to create an IAM Role for use with the service account mapped to External DNS pods which will have list and update permissions on Route53 Public Hosted Zone created earlier for subdomain app.learnwithpras.xyz
.
---
AWSTemplateFormatVersion: 2010-09-09
Description: Deploy Managed Kubernetes Resources - External DNS Consumable Role
Parameters:
HostedZoneId:
Type: AWS::SSM::Parameter::Value<String>
Description: Public hosted zone ID in use
OidcProvider:
Type: AWS::SSM::Parameter::Value<String>
Description: EKS Cluster's OIDC provider url
ExternalDnsNamespace:
Type: String
Description: Kubernetes External DNS Namespace
ExternalDnsServiceAccountName:
Type: String
Description: Name of the Service Account used by external dns to interact with Route53
Resources:
PrasClusterExternalDnsRole:
Type: AWS::IAM::Role
Properties:
AssumeRolePolicyDocument: !Sub
- |
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Federated": "${IamOidcProviderArn}"
},
"Action": "sts:AssumeRoleWithWebIdentity",
"Condition": {
"StringEquals": {
"${OidcProvider}:sub": "system:serviceaccount:${ExternalDnsNamespace}:${ExternalDnsServiceAccountName}"
}
}
}
]
}
-
IamOidcProviderArn: !Sub arn:aws:iam::${AWS::AccountId}:oidc-provider/${OidcProvider}
Policies:
- PolicyName: !Sub ${AWS::StackName}-pras-eks-cluster-externaldns-policy
PolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Action:
- route53:ChangeResourceRecordSets
Resource: !Sub arn:aws:route53:::hostedzone/${HostedZoneId}
- Effect: Allow
Action:
- route53:ListHostedZones
- route53:ListResourceRecordSets
Resource: "*"
PrasClusterExternalDnsRoleArn:
Type: AWS::SSM::Parameter
Properties:
Description: Parameter to store ARN of IAM Role used by External DNS pod to access Route53 resources
Name: /pras/eks/application/dns/iam-role/arn
Type: String
Value: !GetAtt PrasClusterExternalDnsRole.Arn
Cloudformation parameters,
NOTE: AWS::SSM::Parameter::Value resolves an SSM parameter when we specify the path
- HostedZoneId: /pras/route53/app/public-hosted-zone/id - created with the hosted zone CFN stack earlier
- OidcProvider: /pras/eks/cluster/oidc-provider/url - created in the cluster creation CFN template
- ExternalDnsNamespace: external-dns - remember this as we will need to create the namespace with this name
- ExternalDnsServiceAccountName: external-dns-svc - remember this as we will need to create the service account with this name
Run the aws cli command shown below to deploy the IAM role and an SSM parameter to store the ARN of the role (we will fetch this role arn and supply it to the external DNS service account's annotation),
aws cloudformation deploy \
--s3-bucket pras-cloudformation-artifacts-bucket \
--template-file cloudformation/external-dns-role.yaml \
--stack-name pras-external-dns-role \
--capabilities CAPABILITY_NAMED_IAM \
--no-fail-on-empty-changeset \
--parameter-overrides \
OidcProvider=/pras/eks/cluster/oidc-provider/url \
ExternalDnsNamespace=external-dns \
ExternalDnsServiceAccountName=external-dns-svc \
HostedZoneId=/pras/route53/hosted-zone/id \
--tags \
Name='Kubernetes Cluster Resources - External DNS Role'
As seen from the IAM console, the role only trusts OIDC Provider created for the EKS cluster. Even better, the condition ensures that the subject that assumes this IAM role must be from EKS namespace external-dns
and service account external-dns-svc
; external-dns containers in any pod running in the EKS cluster that uses the service account. In this way, we do not have to add permissions to the node group's EC2 Instances' IAM role or pass in credentials to the containers for the application to interact with AWS services.
A quick run of command kubectl get all -A
shows that the cluster as well as the sample eks application is up and running.
$ kubectl get all -A
NAMESPACE NAME READY STATUS RESTARTS AGE
boltdynamics pod/sample-eks-app-deployment-6c9fff7798-ckw92 1/1 Running 0 12h
ingress-nginx pod/ingress-nginx-controller-65c4f84996-9q6rg 1/1 Running 0 12h
kube-system pod/aws-node-6bph2 1/1 Running 0 12h
kube-system pod/coredns-6bfbc5f9f8-78sns 1/1 Running 0 12h
kube-system pod/coredns-6bfbc5f9f8-t4k95 1/1 Running 0 12h
kube-system pod/kube-proxy-5tflf 1/1 Running 0 12h
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
boltdynamics service/sample-eks-app-service ClusterIP 172.20.161.123 <none> 80/TCP 2d16h
default service/kubernetes ClusterIP 172.20.0.1 <none> 443/TCP 2d16h
ingress-nginx service/ingress-nginx-controller LoadBalancer 172.20.178.68 ab6361525ae524562852af72eb047dc4-d9886e3de31f1805.elb.ap-southeast-2.amazonaws.com 80:31910/TCP,443:31719/TCP 2d16h
ingress-nginx service/ingress-nginx-controller-admission ClusterIP 172.20.142.246 <none> 443/TCP 2d16h
kube-system service/kube-dns ClusterIP 172.20.0.10 <none> 53/UDP,53/TCP 2d16h
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system daemonset.apps/aws-node 1 1 1 1 1 <none> 2d16h
kube-system daemonset.apps/kube-proxy 1 1 1 1 1 <none> 2d16h
NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE
boltdynamics deployment.apps/sample-eks-app-deployment 1/1 1 1 2d16h
ingress-nginx deployment.apps/ingress-nginx-controller 1/1 1 1 2d16h
kube-system deployment.apps/coredns 2/2 2 2 2d16h
NAMESPACE NAME DESIRED CURRENT READY AGE
boltdynamics replicaset.apps/sample-eks-app-deployment-6c9fff7798 1 1 1 2d15h
boltdynamics replicaset.apps/sample-eks-app-deployment-7f6d79fb5c 0 0 0 2d16h
ingress-nginx replicaset.apps/ingress-nginx-controller-65c4f84996 1 1 1 2d16h
kube-system replicaset.apps/coredns-6bfbc5f9f8 2 2 2 2d16h
NAMESPACE NAME COMPLETIONS DURATION AGE
ingress-nginx job.batch/ingress-nginx-admission-create 1/1 6s 2d16h
ingress-nginx job.batch/ingress-nginx-admission-patch 1/1 7s 2d16h
If we navigate to http://ab6361525ae524562852af72eb047dc4-d9886e3de31f1805.elb.ap-southeast-2.amazonaws.com/
which is the external IP of service/ingress-nginx-controller from the browser, we can see that the sample application is also running,
Let's get the YAML required to deploy external DNS and modify a few values,
apiVersion: v1
kind: Namespace
metadata:
name: external-dns
labels:
name: external-dns
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: external-dns-svc
namespace: external-dns
annotations:
eks.amazonaws.com/role-arn: ${EXTERNAL_DNS_ROLE_ARN}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: external-dns-role
rules:
- apiGroups: [""]
resources: ["services","endpoints","pods"]
verbs: ["get","watch","list"]
- apiGroups: ["extensions","networking.k8s.io"]
resources: ["ingresses"]
verbs: ["get","watch","list"]
- apiGroups: [""]
resources: ["nodes"]
verbs: ["list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: external-dns-viewer
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: external-dns-role
subjects:
- kind: ServiceAccount
name: external-dns-svc
namespace: external-dns
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: external-dns-deployment
namespace: external-dns
spec:
strategy:
type: Recreate
selector:
matchLabels:
app: external-dns
template:
metadata:
labels:
app: external-dns
spec:
serviceAccountName: external-dns-svc
containers:
- name: external-dns
image: k8s.gcr.io/external-dns/external-dns:v0.9.0
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 300m
memory: 256Mi
args:
- --source=ingress
- --domain-filter=app.learnwithpras.xyz
- --provider=aws
- --policy=sync
- --aws-zone-type=public
- --registry=txt
- --publish-internal-services
Notice how the namespace and service account names match the ones specified in the IAM role created earlier. We will use envsubst to pass in the IAM role arn and replace ${EXTERNAL_DNS_ROLE_ARN}
in the yaml template.
Notice container argument source
set to ingress
which tells external-dns to look for dns hostnames in ingress objects. Argument domain-filter
is set to our custom domain app.learnwithpras.xyz
and policy
is set to sync
which tells external-dns to update/replace any records in Route53 to match the current state (you could use upsert-only which would prevent external-dns from deleting any records).
Run export EXTERNAL_DNS_ROLE_ARN=$(aws ssm get-parameter --name /pras/eks/application/dns/iam-role/arn --query Parameter.Value --output text)
to store the IAM Role's Arn value in the variable. Finally, run envsubst < kubernetes/external-dns/external-dns.yaml | kubectl apply -f -
to deploy external dns resources to the EKS cluster.
$ envsubst < kubernetes/external-dns/external-dns.yaml | kubectl apply -f -
namespace/external-dns created
serviceaccount/external-dns-svc created
clusterrole.rbac.authorization.k8s.io/external-dns-role created
clusterrolebinding.rbac.authorization.k8s.io/external-dns-viewer created
deployment.apps/external-dns-deployment created
$ kubectl get all -n external-dns
NAME READY STATUS RESTARTS AGE
pod/external-dns-deployment-778d4c597-l8nn2 1/1 Running 0 80s
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/external-dns-deployment 1/1 1 1 80s
NAME DESIRED CURRENT READY AGE
replicaset.apps/external-dns-deployment-778d4c597 1 1 1 80s
If we run kubectl describe pod external-dns-deployment-778d4c597-l8nn2 -n external-dns
, we can verify that the IAM role's arn has been passed to the external dns pod,
If we run kubectl logs external-dns-deployment-778d4c597-l8nn2 -n external-dns
, we can see that external-dns has synced up and is ready to add any new records to the hosted zone.
$ kubectl logs external-dns-deployment-778d4c597-l8nn2 -n external-dns
time="2021-09-11T08:38:44Z" level=info msg="config: {APIServerURL: KubeConfig: RequestTimeout:30s DefaultTargets:[] ContourLoadBalancerService:heptio-contour/contour GlooNamespace:gloo-system SkipperRouteGroupVersion:zalando.org/v1 Sources:[ingress] Namespace: AnnotationFilter: LabelFilter: FQDNTemplate: CombineFQDNAndAnnotation:false IgnoreHostnameAnnotation:false IgnoreIngressTLSSpec:false IgnoreIngressRulesSpec:false Compatibility: PublishInternal:true PublishHostIP:false AlwaysPublishNotReadyAddresses:false ConnectorSourceServer:localhost:8080 Provider:aws GoogleProject: GoogleBatchChangeSize:1000 GoogleBatchChangeInterval:1s GoogleZoneVisibility: DomainFilter:[app.learnwithpras.xyz] ExcludeDomains:[] RegexDomainFilter: RegexDomainExclusion: ZoneNameFilter:[] ZoneIDFilter:[] AlibabaCloudConfigFile:/etc/kubernetes/alibaba-cloud.json AlibabaCloudZoneType: AWSZoneType:public AWSZoneTagFilter:[] AWSAssumeRole: AWSBatchChangeSize:1000 AWSBatchChangeInterval:1s AWSEvaluateTargetHealth:true AWSAPIRetries:3 AWSPreferCNAME:false AWSZoneCacheDuration:0s AzureConfigFile:/etc/kubernetes/azure.json AzureResourceGroup: AzureSubscriptionID: AzureUserAssignedIdentityClientID: BluecatConfigFile:/etc/kubernetes/bluecat.json CloudflareProxied:false CloudflareZonesPerPage:50 CoreDNSPrefix:/skydns/ RcodezeroTXTEncrypt:false AkamaiServiceConsumerDomain: AkamaiClientToken: AkamaiClientSecret: AkamaiAccessToken: AkamaiEdgercPath: AkamaiEdgercSection: InfobloxGridHost: InfobloxWapiPort:443 InfobloxWapiUsername:admin InfobloxWapiPassword: InfobloxWapiVersion:2.3.1 InfobloxSSLVerify:true InfobloxView: InfobloxMaxResults:0 InfobloxFQDNRegEx: DynCustomerName: DynUsername: DynPassword: DynMinTTLSeconds:0 OCIConfigFile:/etc/kubernetes/oci.yaml InMemoryZones:[] OVHEndpoint:ovh-eu OVHApiRateLimit:20 PDNSServer:http://localhost:8081 PDNSAPIKey: PDNSTLSEnabled:false TLSCA: TLSClientCert: TLSClientCertKey: Policy:sync Registry:txt TXTOwnerID:default TXTPrefix: TXTSuffix: Interval:1m0s MinEventSyncInterval:5s Once:false DryRun:false UpdateEvents:false LogFormat:text MetricsAddress::7979 LogLevel:info TXTCacheInterval:0s TXTWildcardReplacement: ExoscaleEndpoint:https://api.exoscale.ch/dns ExoscaleAPIKey: ExoscaleAPISecret: CRDSourceAPIVersion:externaldns.k8s.io/v1alpha1 CRDSourceKind:DNSEndpoint ServiceTypeFilter:[] CFAPIEndpoint: CFUsername: CFPassword: RFC2136Host: RFC2136Port:0 RFC2136Zone: RFC2136Insecure:false RFC2136GSSTSIG:false RFC2136KerberosRealm: RFC2136KerberosUsername: RFC2136KerberosPassword: RFC2136TSIGKeyName: RFC2136TSIGSecret: RFC2136TSIGSecretAlg: RFC2136TAXFR:false RFC2136MinTTL:0s RFC2136BatchChangeSize:50 NS1Endpoint: NS1IgnoreSSL:false NS1MinTTLSeconds:0 TransIPAccountName: TransIPPrivateKeyFile: DigitalOceanAPIPageSize:50 ManagedDNSRecordTypes:[A CNAME] GoDaddyAPIKey: GoDaddySecretKey: GoDaddyTTL:0 GoDaddyOTE:false}"
time="2021-09-11T08:38:44Z" level=info msg="Instantiating new Kubernetes client"
time="2021-09-11T08:38:44Z" level=info msg="Using inCluster-config based on serviceaccount-token"
time="2021-09-11T08:38:44Z" level=info msg="Created Kubernetes client https://172.20.0.1:443"
time="2021-09-11T08:38:54Z" level=info msg="Applying provider record filter for domains: [app.learnwithpras.xyz. .app.learnwithpras.xyz.]"
time="2021-09-11T08:38:54Z" level=info msg="All records are already up to date"
time="2021-09-11T08:39:51Z" level=info msg="Applying provider record filter for domains: [app.learnwithpras.xyz. .app.learnwithpras.xyz.]"
time="2021-09-11T08:39:51Z" level=info msg="All records are already up to date"
time="2021-09-11T08:40:51Z" level=info msg="Applying provider record filter for domains: [app.learnwithpras.xyz. .app.learnwithpras.xyz.]"
time="2021-09-11T08:40:51Z" level=info msg="All records are already up to date"
time="2021-09-11T08:41:53Z" level=info msg="Applying provider record filter for domains: [app.learnwithpras.xyz. .app.learnwithpras.xyz.]"
time="2021-09-11T08:41:53Z" level=info msg="All records are already up to date"
time="2021-09-11T08:42:53Z" level=info msg="Applying provider record filter for domains: [app.learnwithpras.xyz. .app.learnwithpras.xyz.]"
time="2021-09-11T08:42:53Z" level=info msg="All records are already up to date"
time="2021-09-11T08:43:54Z" level=info msg="Applying provider record filter for domains: [app.learnwithpras.xyz. .app.learnwithpras.xyz.]"
time="2021-09-11T08:43:54Z" level=info msg="All records are already up to date"
We now need to update NGINX service to leverage the SSL certificate with NLB; terminate TLS connection at the load balancer.
Run kubectl edit -n
to edit the service resource and add an annotation, service.beta.kubernetes.io/aws-load-balancer-ssl-cert: ACM_CERT_ARN
where ACM_CERT_ARN is to be replaced with the ARN of certificate we created earlier.
This presents a problem, which is where the NGINX controller attempts to create a new listener on port 443 but one already exists. Run kubectl describe svc ingress-nginx-controller -n ingress-nginx
to read the events,
$ kubectl describe svc ingress-nginx-controller -n ingress-nginx
Name: ingress-nginx-controller
Namespace: ingress-nginx
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/managed-by=Helm
app.kubernetes.io/name=ingress-nginx
app.kubernetes.io/version=1.0.0
helm.sh/chart=ingress-nginx-4.0.1
Annotations: service.beta.kubernetes.io/aws-load-balancer-backend-protocol: tcp
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: true
service.beta.kubernetes.io/aws-load-balancer-ssl-cert:
arn:aws:acm:ap-southeast-2:[REDACTED]:certificate/91ab5da6-0800-4f19-a292-700c149ae1a5
service.beta.kubernetes.io/aws-load-balancer-type: nlb
Selector: app.kubernetes.io/component=controller,app.kubernetes.io/instance=ingress-nginx,app.kubernetes.io/name=ingress-nginx
Type: LoadBalancer
IP Families: <none>
IP: 172.20.178.68
IPs: 172.20.178.68
LoadBalancer Ingress: ab6361525ae524562852af72eb047dc4-d9886e3de31f1805.elb.ap-southeast-2.amazonaws.com
Port: http 80/TCP
TargetPort: http/TCP
NodePort: http 31910/TCP
Endpoints: 10.0.0.145:80
Port: https 443/TCP
TargetPort: https/TCP
NodePort: https 31719/TCP
Endpoints: 10.0.0.145:443
Session Affinity: None
External Traffic Policy: Local
HealthCheck NodePort: 32397
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning SyncLoadBalancerFailed 16m service-controller Error syncing load balancer: failed to ensure load balancer: error creating load balancer listener: "DuplicateListener: A listener already exists on this port for this load balancer 'arn:aws:elasticloadbalancing:ap-southeast-2:[REDACTED]:loadbalancer/net/ab6361525ae524562852af72eb047dc4/d9886e3de31f1805'\n\tstatus code: 400, request id: c92bf3a0-4e86-461a-9b3a-7ef63189702f"
Warning SyncLoadBalancerFailed 16m service-controller Error syncing load balancer: failed to ensure load balancer: error creating load balancer listener: "DuplicateListener: A listener already exists on this port for this load balancer 'arn:aws:elasticloadbalancing:ap-southeast-2:[REDACTED]:loadbalancer/net/ab6361525ae524562852af72eb047dc4/d9886e3de31f1805'\n\tstatus code: 400, request id: a10541de-047f-4211-99ad-f680d75e360c"
Warning SyncLoadBalancerFailed 15m service-controller Error syncing load balancer: failed to ensure load balancer: error creating load balancer listener: "DuplicateListener: A listener already exists on this port for this load balancer 'arn:aws:elasticloadbalancing:ap-southeast-2:[REDACTED]:loadbalancer/net/ab6361525ae524562852af72eb047dc4/d9886e3de31f1805'\n\tstatus code: 400, request id: 70e8514f-6de6-4378-82c6-bccc5fc6b72f"
Warning SyncLoadBalancerFailed 15m service-controller Error syncing load balancer: failed to ensure load balancer: error creating load balancer listener: "DuplicateListener: A listener already exists on this port for this load balancer 'arn:aws:elasticloadbalancing:ap-southeast-2:[REDACTED]:loadbalancer/net/ab6361525ae524562852af72eb047dc4/d9886e3de31f1805'\n\tstatus code: 400, request id: 3dc34d20-59b2-4c64-abcd-9e3ba99f9ed4"
Warning SyncLoadBalancerFailed 14m service-controller Error syncing load balancer: failed to ensure load balancer: error creating load balancer listener: "DuplicateListener: A listener already exists on this port for this load balancer 'arn:aws:elasticloadbalancing:ap-southeast-2:[REDACTED]:loadbalancer/net/ab6361525ae524562852af72eb047dc4/d9886e3de31f1805'\n\tstatus code: 400, request id: 2100fd6c-42d1-4aff-8434-9f64989240cd"
Warning SyncLoadBalancerFailed 13m service-controller Error syncing load balancer: failed to ensure load balancer: error creating load balancer listener: "DuplicateListener: A listener already exists on this port for this load balancer 'arn:aws:elasticloadbalancing:ap-southeast-2:[REDACTED]:loadbalancer/net/ab6361525ae524562852af72eb047dc4/d9886e3de31f1805'\n\tstatus code: 400, request id: 4d62de0a-3dae-4857-81ac-a2efb0525abd"
Warning SyncLoadBalancerFailed 10m service-controller Error syncing load balancer: failed to ensure load balancer: error creating load balancer listener: "DuplicateListener: A listener already exists on this port for this load balancer 'arn:aws:elasticloadbalancing:ap-southeast-2:[REDACTED]:loadbalancer/net/ab6361525ae524562852af72eb047dc4/d9886e3de31f1805'\n\tstatus code: 400, request id: 10d17d55-05b2-4e66-96d0-54dcae7e8dc4"
Warning SyncLoadBalancerFailed 5m55s service-controller Error syncing load balancer: failed to ensure load balancer: error creating load balancer listener: "DuplicateListener: A listener already exists on this port for this load balancer 'arn:aws:elasticloadbalancing:ap-southeast-2:[REDACTED]:loadbalancer/net/ab6361525ae524562852af72eb047dc4/d9886e3de31f1805'\n\tstatus code: 400, request id: e87dd113-7eb8-467f-9bcf-b184e7e8116c"
Normal EnsuringLoadBalancer 55s (x10 over 4d15h) service-controller Ensuring load balancer
Warning SyncLoadBalancerFailed 54s service-controller Error syncing load balancer: failed to ensure load balancer: error creating load balancer listener: "DuplicateListener: A listener already exists on this port for this load balancer 'arn:aws:elasticloadbalancing:ap-southeast-2:[REDACTED]:loadbalancer/net/ab6361525ae524562852af72eb047dc4/d9886e3de31f1805'\n\tstatus code: 400, request id: ea5d55f9-f6f8-4c1c-b5e6-54a961a28d79"
Run kubectl get svc ingress-nginx-controller -n ingress-nginx -o yaml > nginx-service.yaml
to save the existing nginx service state configuration in a file named nginx-service.yaml
Then, run kubectl delete svc ingress-nginx-controller -n ingress-nginx
to delete the nginx service.
$ kubectl delete svc ingress-nginx-controller -n ingress-nginx
service "ingress-nginx-controller" deleted
This will delete the existing NGINX controller service including the Network Load Balancer associated with it. Command kubectl get all -n ingress-nginx
verifies that the service is no longer present.
$ kubectl get all -n ingress-nginx
NAME READY STATUS RESTARTS AGE
pod/ingress-nginx-controller-65c4f84996-v54dk 1/1 Running 0 47h
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/ingress-nginx-controller-admission ClusterIP 172.20.142.246 <none> 443/TCP 4d16h
NAME READY UP-TO-DATE AVAILABLE AGE
deployment.apps/ingress-nginx-controller 1/1 1 1 4d16h
NAME DESIRED CURRENT READY AGE
replicaset.apps/ingress-nginx-controller-65c4f84996 1 1 1 4d16h
NAME COMPLETIONS DURATION AGE
job.batch/ingress-nginx-admission-create 1/1 6s 4d16h
job.batch/ingress-nginx-admission-patch 1/1 7s 4d16h
This time the listener on port 443 will be created and an SSL certificate associated with the NLB for TLS termination when we run kubectl apply -f nginx-service.yaml
to redeploy the service. Also, update the target port under spec to target http port 80 (ingress sample-eks-app-ingress
will listen on port 80),
- appProtocol: https
name: https
nodePort: 31719
port: 443
protocol: TCP
targetPort: http
$ kubectl apply -f nginx-service.yaml
service/ingress-nginx-controller created
$ kubectl get svc ingress-nginx-controller -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 172.20.178.68 a53c676577b9c4050a639d9a4372884e-0987ffe41fbacac9.elb.ap-southeast-2.amazonaws.com 80:31910/TCP,443:31719/TCP 32s
We can verify from the AWS Console that the NLB is being created with listeners and SSL certificate associated,
Apply the following ingress object with kubectl apply -f ingress.yaml
,
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: sample-eks-app-ingress
namespace: boltdynamics
annotations:
kubernetes.io/ingress.class: "nginx"
spec:
rules:
- host: app.learnwithpras.xyz
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: sample-eks-app-service
port:
number: 80
View external-dns pod logs to see if its attempted to add records to hosted zone,
$ kubectl logs external-dns-deployment-778d4c597-l8nn2 -n external-dns
time="2021-09-11T23:13:58Z" level=info msg="Applying provider record filter for domains: [app.learnwithpras.xyz. .app.learnwithpras.xyz.]"
time="2021-09-11T23:13:59Z" level=info msg="Desired change: CREATE app.learnwithpras.xyz A [Id: /hostedzone/Z07438702O3MLG3B1HLEU]"
time="2021-09-11T23:13:59Z" level=info msg="Desired change: CREATE app.learnwithpras.xyz TXT [Id: /hostedzone/Z07438702O3MLG3B1HLEU]"
time="2021-09-11T23:13:59Z" level=info msg="2 record(s) in zone app.learnwithpras.xyz. [Id: /hostedzone/Z07438702O3MLG3B1HLEU] were successfully updated"
And navigate to Route53 Console and check if records have been added to the hosted zone,
Navigate to a web browser and to app.learnwithpras.xyz
,
This brings us to the end of this article, thank you for reading all the way and I hope I was able to articulate the use of a custom domain to front an application deployed in an EKS environment and all the steps that are necessary to integrate it fully using different tools and techniques.
In the next article, we will implement TLS termination at the Nginx Controller using LetsEncrypt vs at the NLB as we have done as part of this demo.
Stay tuned and stay safe ๐