» Ambassador Integration with Consul Connect
In addition to enabling Kubernetes services to discover and securely connect to each other, Connect also can help route traffic into a Kubernetes cluster from outside, when paired with an ingress controller like DataWire's Ambassador.
Ambassador is a popular Kubernetes-native service that acts as an ingress controller or API gateway. It supports an optional integration with Consul that allows it to route incoming traffic to the proxies for your Connect-enabled services.
This means you can have end-to-end encryption from the browser, to Ambassador, to your Kubernetes services.
» Installation
Before you start, install Consul and enable Connect on the agents inside the cluster. Decide whether you will enable service sync or manually register your services with Consul.
Once you have tested and verified that everything is working, you can proceed with the Ambassador installation. Full instructions are available on the Ambassador site, but a summary of the steps is as follows:
If you are deploying to GKE, create a RoleBinding to grant you cluster admin
rights:
kubectl create clusterrolebinding my-cluster-admin-binding \
--clusterrole=cluster-admin \
--user=$(gcloud info --format="value(config.account)")
Install Ambassador and a LoadBalancer service for it:
kubectl apply -f https://getambassador.io/yaml/ambassador/ambassador-rbac.yaml
kubectl apply -f https://getambassador.io/yaml/ambassador/ambassador-service.yaml
Install the Ambassador Consul Connector:
kubectl apply -f https://getambassador.io/yaml/ambassador/pro/ambassador-consul-connector.yaml
Add TLSContext and Mapping annotations to your existing services, directing
HTTPS traffic to port 20000, which is opened by the Connect proxy. Here is an
example of doing this for the static-server example used in the documentation
for the Connect sidecar:
apiVersion: v1
kind: Service
metadata:
name: static-service
annotations:
getambassador.io/config: |
---
apiVersion: ambassador/v1
kind: TLSContext
name: ambassador-consul
hosts: []
secret: ambassador-consul-connect
---
apiVersion: ambassador/v1
kind: Mapping
name: static-service_mapping
prefix: /echo/
tls: ambassador-consul
service: https://static-server:443
spec:
type: NodePort
selector:
app: http-echo
ports:
- port: 443
name: https-echo
targetPort: 20000
Once Ambassador finishes deploying, you should have a new LoadBalancer service
with a public-facing IP address. Connecting to the HTTP port on this address
should display the output from the static service.
kubectl describe service ambassador
» Enabling end-to-end TLS
The Ambassador service definition provided in their documentation currently does not serve pages over HTTPS. To enable HTTPS for full end-to-end encryption, follow these steps.
First, upload your public SSL certificate and private key as a Kubernetes secret.
kubectl create secret tls ambassador-certs --cert=fullchain.pem --key=privkey.pem
Download a copy of the ambassador-service.yaml file from Ambassador. Replace
the metadata section with one that includes an Ambassador TLS configuration block,
using the secret name you created in the previous step. Then add an entry for port 443
to the LoadBalancer spec. Here is a complete example:
apiVersion: v1
kind: Service
metadata:
name: ambassador
annotations:
getambassador.io/config: |
---
apiVersion: ambassador/v1
kind: Module
name: tls
config:
server:
enabled: True
secret: ambassador-certs
spec:
type: LoadBalancer
externalTrafficPolicy: Local
ports:
- port: 80
targetPort: http
protocol: TCP
name: http
- port: 443
targetPort: https
protocol: TCP
name: https
selector:
service: ambassador
Update the service definition by applying it with kubectl:
kubectl apply -f ambassador-service.yaml
You should now be able to test the SSL connection from your browser.
» Troubleshooting
When Ambassador is unable to establish an authenticated connection to the Connect proxy servers, browser connections will display this message:
upstream connect error or disconnect/reset before headers
This error can have a number of different causes. Here are some things to check and troubleshooting steps you can take.
» Check intentions between Ambassador and your upstream service
If you followed the above installation guide, Consul should have registered a service called "ambassador". Make sure you create an intention to allow it to connect to your own services.
To check whether Ambassador is allowed to connect, use the intention check subcommand.
$ consul intention check ambassador http-echo
Allowed
» Confirm upstream proxy sidecar is running
First, find the name of the pod that contains your service.
$ kubectl get pods -l app=http-echo,role=server
NAME READY STATUS RESTARTS AGE
http-echo-7fb79566d6-jmccp 2/2 Running 0 1d
Then describe the pod to make sure that the sidecar is present and running.
$ kubectl describe pod http-echo-7fb79566d6-jmccp
[...]
Containers:
consul-connect-envoy-sidecar:
[...]
State: Running
Ready: True
» Start up a downstream proxy and try connecting to it
Log into one of your Consul server pods (or any pod that has a Consul binary in it).
$ kubectl exec -ti consul-server-0 -- /bin/sh
Once inside the pod, try starting a test proxy. Use the name of your service in place of http-echo.
# consul connect proxy -service=ambassador -upstream http-echo:1234
==> Consul Connect proxy starting...
Configuration mode: Flags
Service: http-echo-client
Upstream: http-echo => :1234
Public listener: Disabled
If the proxy starts successfully, try connecting to it. Verify the output is as you expect.
# curl localhost:1234
"hello world"
Don't forget to kill the test proxy when you're done.
# kill %1
==> Consul Connect proxy shutdown
# exit
» Check Ambassador Connect sidecar logs
Find the name of the Connect Integration pod and make sure it is running.
$ kubectl get pods -l app=ambassador-pro,component=consul-connect
NAME READY STATUS RESTARTS AGE
ambassador-pro-consul-connect-integration-f88fcb99f-hxk75 1/1 Running 0 1d
Dump the logs from the integration pod. If the service is running correctly, there won't be much in there.
$ kubectl logs ambassador-pro-consul-connect-integration-f88fcb99f-hxk75
time="2019-03-13T19:42:12Z" level=info msg="Starting Consul Connect Integration" consul_host=10.142.0.21 consul_port=8500 version=0.2.3
2019/03/13 19:42:12 Watching CA leaf for ambassador
time="2019-03-13T19:42:12Z" level=debug msg="Computed kubectl command and arguments" args="[kubectl apply -f -]"
time="2019-03-13T19:42:14Z" level=info msg="Updating TLS certificate secret" namespace= secret=ambassador-consul-connect
» Check Ambassador logs
Make sure the Ambassador pod itself is running.
$ kubectl get pods -l service=ambassador
NAME READY STATUS RESTARTS AGE
ambassador-655875b5d9-vpc2v 2/2 Running 0 1d
Finally, check the logs for the main Ambassador pod.
$ kubectl logs ambassador-655875b5d9-vpc2v
» Check Ambassador admin interface
Forward the admin port from the Ambassador pod to your local machine.
$ kubectl port-forward pods/ambassador-655875b5d9-vpc2v 8877:8877
You should then be able to open http://localhost:8877/ambassador/v0/diag/ in your browser and view Ambassador's routing table. The table lists each URL mapping that has been set up. Service names will appear in green if Ambassador believes they are healthy, and red otherwise.
From this interface, you can also enable debug logging via the yellow "Set Debug On" button, which might give you a better idea of what's happening when requests fail.
» Getting support
If you have tried the above troubleshooting steps and are still stuck, DataWire provides support for Ambassador via the popular Slack chat app. You can request access and then join the #ambassador room to get help.