ingress-nginx

Kubernetes Ingress Controllers for HAProxy already exist. A nice one is at https://github.com/jcmoraisjr/haproxy-ingress Other examples of usage are spread around in the examples section of the ingress repo, like here https://github.com/kubernetes/ingress/tree/master/examples/d... The DNS-SD code will ship with HAProxy 1.8, and we will gladly help out by adding it to the above ingress controller after the release. A WIP of that code is up on github in case you’re interested. In fact, HAProxy is full of surprises and little known features (like the dynamic scaling via haproxy runtime api we already contributed to the controller) and we're happy to share them with everyone!

> TLS termination at the Ingress Controller and by default unencrypted from there to the service endpoint? We are doing TLS termination at the ELB (we're running on AWS). > Interesting discussion here: https://github.com/kubernetes/ingress/issues/257 Great, thanks! Regarding ways of updating of the NGINX upstreams without requiring a reload, I was just made aware of modules like ngx_dynamic_upstream[1]. I'm sure there are other ways to address this in a less disruptive way than reloading everything, so this is probably something that could be improved in the future. [1] https://github.com/cubicdaiya/ngx_dynamic_upstream

Thank you. So there is a DNAT to get to the Ingress Controller but from there at least it's direct routing to the service endpoint(s)? Does that mean the Virtual IP given to the Service is basically bypassed when using Ingress Controller? TLS termination at the Ingress Controller and by default unencrypted from there to the service endpoint? I found this useful: http://blog.wercker.com/troubleshooting-ingress-kubernetes Interesting discussion here: https://github.com/kubernetes/ingress/issues/257 It seems like a lot of overhead before even starting to process a request!

No. By default, the NGINX ingress controller routes traffic directly to pod IPs (the Service endpoints): https://github.com/kubernetes/ingress/tree/master/controller...

> Could you share which version of NGINX you found the issue with the reloads? Which version the fix was released? I'm using 0.9.0-beta.13. I first reported this issue in a NGINX ingress PR[1], so the last couple of releases are not suffering from the bug I reported in the blog post. > I find it interesting/brave that you use a single cluster for several environments. I'm not working for a big corporation, so dev/staging/prod "environments" are just three deployment pipelines to the same infrastructure. As of now, things are running smoothly as they are, but I might as well use different clusters for each environment in the future. [1] https://github.com/kubernetes/ingress/pull/1088

Note one significant gotcha with this approach: the Ingress does TLS termination, so the hop from the Ingress to your pod is unencrypted. That might be OK if 1) your data isn's sensitive or 2) you're running on your own metal (and so you control the network), but in GKE your nodes are on Google's SDN, and so you're sending your traffic across their DCs in the clear. There are a couple of pieces of hard-to-find config required to achieve TLS-to-the-pod with Ingress: 1) You need to enable ssl-passthrough on your nginx ingress; this is a simple annotation: https://github.com/kubernetes/contrib/issues/1854. This will use nginx's streaming mode to route requests with SNI without terminating the TLS connection. 2) Now you'll need a way of getting your certs into the pod; kube-lego attaches the certs to the Ingress pod, which is not what you wan

Please check https://github.com/kubernetes/ingress/pull/190

It's the same proxy_protocol keyword for both, which is why this is so confusing. In the listen line it means "remove proxy protocol from the inbound connection", as a top level directive "proxy_protocol on" on the server it means "add proxy protocol to the outbound connection" This commit should show the difference: https://github.com/kubernetes/ingress/commit/6fa461c2a7891b4... This is the nginx function AFAICT: https://github.com/nginx/nginx/blob/b580770f3afaeec48a15cb8c... Looking at that though, maybe it only works with SSL passthrough... but that is the typical use-case for using proxy protocol instead of X-Forwarded-For

> Kubernetes on the other hand needs atleast 12 yml files You don't need to split your manifests into multiple files. You can use a list: apiVersion: v1 kind: List items: - apiVersion: extensions/v1beta1 kind: Deployment ... - apiVersion: v1 kind: Service ... - apiVersion: v1 kind: ConfigMap ... Edit: Or YAML's "multiple document" syntax: apiVersion: extensions/v1beta1 kind: Deployment ... --- apiVersion: v1 kind: Service ... --- apiVersion: v1 kind: ConfigMap ... Then you just use "kubectl apply -f myapp.yml" to create or update. Re ingress, I agree that it's probably the weakest point of Kubernetes at this point. It's particularly weak when it comes to internal load balancing. When you have lots of services that should only need to

I'm familiar with proxy-protocol, and while I haven't injected it with nginx, it seems that it does support it: https://nginx.org/en/docs/stream/ngx_stream_proxy_module.htm... For reference, the feature was added in nginx 1.9.2, and the ingress-nginx controller appears to use nginx 1.11.9: https://github.com/kubernetes/ingress/blob/0.9.1/images/ngin... Regardless, would I be right in assuming that your issues are mostly related to the ingress of TCP streams? My usecase is almost exclusively HTTP, so I might be missing the worst of it. Thanks for the pointer to the #sig-onpremise channel, I'll be sure to check it out. Been meaning to look into the SIGs since reading https://coreos.com/blog/self-hosted-kubernetes.html