/ Management  

Fail-over and load-balancing; It's time to cover HAProxy

Hi there “Process Automation” fans,

Welcome to a new installment of “Process Automation” tips.

We move to our next step of hardening our VM. Two weeks ago, we did TomEE over HTTPS. Last week, we moved HTTPS to a reverse proxy over NGINX. For this week, we’ll put a load-balancer between the two applications. Why? because we can, because we’re curious, and because we want to understand what it takes to scale up. We’re not doing the real scaling as the laptop power isn’t sufficient, but still, we can do the setup for load-balancing via HAProxy and mimic some scenarios.

We played with load-balancing before with some fascinating insights you would like to read about when the topic is in your interest.

For this post we’ll build this scenario:

1
2
3
4
5
6
7
8
9
Client/Internet
    ↓
NGINX (TLS + HTTP/2)
    ↓
HAProxy (load-balancing)
    ↓
TomEE 1 (HTTP/1.1 internal)
TomEE 2 (HTTP/1.1 internal)
TomEE 3 (HTTP/1.1 internal)

FYI: HAProxy is a free, very fast and reliable reverse-proxy offering high-availability, load-balancing, and proxying for TCP and HTTP-based applications.

Let’s get right into it…

What do we have this far? Well, that’s a NGINX URL https://opa.mydomain.com/home/opa_tips/ over a secure/certificate HTTPS connection with h2 as a communication protocol; Exactly as we want…AND, blazing fast! This is all behind one TomEE instance. Let’s assume we have a second node up and running; read here on the HOW-part! For this post we only need one instance, but you’ll see in the configuration below it’s rather easy to extend to a second node. AND we now have the benefit of seeing if a missing node keeps the platform up and running…kind of! 🙈 🙉

FYI: We’ll do the load-balancing with software in this post; You also have hardware and cloud load-balancing solutions (not for now, but inform yourself here!)

We do a dive here with HAProxy on our RHEL VM. So, make sure you have your environment up-and-running with TomEE and NGINX to execute the first command to install it: sudo dnf install haproxy -y

Next, you can enable sudo systemctl enable haproxy and start it: sudo systemctl start haproxy

After this quick installation, you can update its configuration via sudo vi /etc/haproxy/haproxy.cfg where we first re-route the logging (comment the current log-statement):

1
2
3
4
global
    #log 127.0.0.1 local2
    log /dev/log local0 info
    log /dev/log local1 notice

You can check if the config is fine via: haproxy -c -f /etc/haproxy/haproxy.cfg

For further log-enablement do sudo vi /etc/rsyslog.d/haproxy.conf with this content:

1
2
3
$AddUnixListenSocket /var/lib/haproxy/dev/log
local0.* /var/log/haproxy-access.log
local1.* /var/log/haproxy-error.log

And create that socket directory: sudo mkdir -p /var/lib/haproxy/dev

After this change, restart ‘rsyslog’: sudo systemctl restart rsyslog

You can do a sanity check with these commands:

1
2
3
logger -p local0.info "test haproxy access log"
logger -p local1.notice "test haproxy error log"
ls -l /var/log/haproxy*

We can now monitor these log-files:

1
2
sudo tail -999f /var/log/haproxy-access.log
sudo tail -999f /var/log/haproxy-error.log

…

With the HAProxy up-and-running (including logging), we can now continue further configuration on sudo vi /etc/haproxy/haproxy.cfg. Add this at the end of the file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
#---------------------------------------------------------------------
# NGINX frontend which proxys to the backends
#---------------------------------------------------------------------
frontend nginx_front_tomee
   bind *:8081
   default_backend tomee_cluster
backend tomee_cluster
   option httpchk GET /cordysguest/com.eibus.web.tools.healthCheck.HealthCheckURL.wcp
   #http-check expect status 200
   http-check expect string "System is Up and Running"
   balance roundrobin
   server node1 192.168.56.103:8080 check
   server node3 192.168.56.105:8080 check
   server node5 192.168.56.107:8080 check
# The below quotes NGINX in front of Tomcat which is a task on your own!
frontend nginx_front_tomcat
   bind *:8082
   default_backend tomcat_cluster
backend tomcat_cluster
   balance roundrobin
   server node1 192.168.56.103:8181 check
   server node3 192.168.56.105:8181 check
   server node5 192.168.56.107:8181 check

In simple words…This config binds the frontend request (over NGINX on port 80) to the backend nodes (in our case only 1 TomEE on port 8080) in a “Round-robin” manner.

Restart HAProxy once more sudo systemctl restart haproxy and watch the logging on these entries:

1
2
3
4
Proxy nginx_front_tomee started.
Proxy tomee_cluster started.
Proxy nginx_front_tomcat started.
Proxy tomcat_cluster started.

So, we’re ready right? WRONG, we need to tell NGINX to communicate with HAProxy instead of directly with TomEE! Run sudo vi /etc/nginx/nginx.conf and make this change in port numbers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
# Location mappings
location /home/system {
    proxy_pass http://127.0.0.1:8081;
} # end location
location /home/opa_tips {
    proxy_pass http://127.0.0.1:8081;
} # end location
location /cordys {
    proxy_pass http://127.0.0.1:8081;
} # end location
location /cordysguest {
    proxy_pass http://127.0.0.1:8081;
} # end location
location /otdsws/login {
    proxy_pass http://127.0.0.1:8082;
} # end location
location /otdsws {
    proxy_pass http://127.0.0.1:8082;
} # end location
location /otds-admin {
    proxy_pass http://127.0.0.1:8082;
} # end location

Validate the NGINX config sudo nginx -t and restart sudo systemctl restart nginx!

For the grand final…Access https://opa.mydomain.com/home/opa_tips/ again, and check the haproxy-access.log! We have green flags with a working and validated setup. ✅

This was one of the main resources for starting this post.

A great load-balancing “DONE” where we had the chance to play with HAProxy in between NGINX and our TomEE. I agree, we only have one TomEE instance, but we can now easily expand to multiple TomEE instances with this setup. Next week, we’ll put our setup this far to a test with assessment software and a fascinating tool in the developer tools of the browser. “To measure is to know”…have a great weekend eXploring and improving your setup to a next level of performance and scale-level.

Don’t forget to subscribe to get updates on the activities happening on this site. Have you noticed the quiz where you find out if you are also “The Process Automation guy”?