Running Quay Enterprise behind a load balancer is often desired for large installations. However, simply putting a load balancer in front of the Quay Enterprise has some unintended consequences:
These issues can be avoided through the use of Proxy Protocol which is exposed by the container on port 8443. This requires the Quay Enterprise container be executed the with the -p 8443:8443
flag on the docker run
command to expose this port:
sudo docker run --restart=always -p 443:443 -p 80:80 -p 8443:8443 --privileged=true -v /local/path/to/the/config/directory:/conf/stack -v /local/path/to/the/storage/directory:/datastorage -d quay.io/coreos/quay
Load balancer requirements include:
Software | TCP Forwarding | Proxy Protocol |
---|---|---|
HAProxy | ✓ | ✓ |
ELB | ✓ | ✓ |
nginx | ✓ | ✕ |
Setting up an ELB with Proxy Protocol enabled requires an existing classic ELB and access to the aws cli tool. AWS documentation on creating an ELB can be found here.
ELB Name: quay-loadbalancer
Policy Name: quay-ProxyProtocol-policy
$ aws elb create-load-balancer-policy --load-balancer-name quay-loadbalancer --policy-name quay-ProxyProtocol-policy --policy-type-name ProxyProtocolPolicyType --policy-attributes AttributeName=ProxyProtocol,AttributeValue=true
Describe the quay-ProxyProtocol-policy
policy to ensure correctness.
$ aws elb describe-load-balancer-policies --load-balancer-name quay-loadbalancer --policy-names quay-ProxyProtocol-policy
Output should match the following:
{
"PolicyDescriptions": [
{
"PolicyName": "quay-ProxyProtocol-policy",
"PolicyAttributeDescriptions": [
{
"AttributeName": "ProxyProtocol",
"AttributeValue": "true"
}
],
"PolicyTypeName": "ProxyProtocolPolicyType"
}
]
}
$ aws elb set-load-balancer-policies-for-backend-server --load-balancer-name quay-loadbalancer --instance-port 8443 --policy-names quay-ProxyProtocol-policy
Confirm that the Proxy Protocol policy has been set to on instance port 8443 by looking for the output of BackendServerDescriptions
in the ELB description:
$ aws elb describe-load-balancers --load-balancer-name quay-loadbalancer
[...]
{
"LoadBalancerDescriptions": [
{
...
"BackendServerDescriptions": [
{
"PolicyNames": [
"quay-ProxyProtocol-policy"
],
"InstancePort": 8443
]
}
],
...
}
]
}
A TCP listener should be configured to route traffic from Load Balancer Port 443 to Instance Port 8443. This is most easily configured from the AWS console.
When properly configured, the Listeners tab for the ELB should appear like so:
After the listeners have been configured, the health checking endpoint needs to be configured to use the previously unused port 443. When properly configured, the Listeners tab for the ELB should appear like so:
If the ELB with Proxy Protocol is functioning properly curling the /v1/_ping
endpoint should return true
. Replace ${LoadBalancer} with the A
record of the quay-loadbalancer ELB..
$ curl -k https://${LoadBalancer}/v1/_ping
true
docker login
and docker push
should be operational.
Security group settings are often the culprit if the ELB is not resolving but the health check reports the instance as in service. Misconfiguration of the security group generally shows up as the connection to the ELB hanging during requests.
The security group for the ELB should be open for inbound traffic from port 80
, 443
, and 8443
.
Setting new ELB policy occasionally requires removing and reassociating the instance for the policy to be applied.