Fail2ban To Block Break-In Attempts

While troubleshooting some initial configurations with my HAProxy setup, I noticed a few alarming things in my /var/log/auth.log.

Apr 17 07:17:00 hostname sshd[16969]: Failed password for invalid user admin from 14.160.56.206 port 20024 ssh2
Apr 17 08:23:55 hostname sshd[17206]: Failed password for invalid user hunter from 178.239.180.101 port 47259 ssh2
Apr 17 08:23:53 hostname sshd[17206]: Address 178.239.180.101 maps to host-101-180.239-178.enter.it, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Apr 17 08:44:28 hostname sshd[17325]: Address 113.160.158.43 maps to static.vdc.vn, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!

Are people trying to compromise my site?? I never thought I would need anything like this, but I decided to install Fail2ban for some protection.

Fail2ban Installation And Configuration

The installation was simple as most Ubuntu thing are:

sudo apt-get install fail2ban

The default configuration was almost all I needed since it already came with ssh protection enabled, but I also wanted email notifications and ssh-ddos protection. The man page was helpful in explaining actions, filters, and jails.

In /etc/fail2ban/jail.d/01_ssh.conf, I configured the following for email notifications:

[ssh]
destemail = christopher.baek@gmail.com
action = %(action_mwl)s

In /etc/fail2ban/jail.d/02_ssh-ddos.conf, I configured the following to enable ssh-ddos protection and email notifications:

[ssh-ddos]
enabled = true
destemail = christopher.baek@gmail.com
action = %(action_mwl)s

Testing

I actually started receiving email alerts almost immediately after I enabled Fail2ban, but for manual testing:

  1. List initial iptables rules
    iptables -L
  2. Trigger Fail2ban by trying to connect via SSH incorrectly for maxretry times within findtime seconds (from the /etc/fail2ban/jail.conf or overridden settings).
  3. Hopefully further SSH attempts should be blocked for bantime seconds, but listing iptables rules should show a new rule
Advertisements

Nullmailer For Send Only Mail

While still taking a break from planning the automated upgrade of this site from WordPress 4.4.2 to WordPress 4.5, I thought it would be a good opportunity to set up Nullmailer. I could use an MTA to write scripts or send alerts about my server.

This is my favorite MTA because it can be configured to send mail from a Gmail account without all the other hassles of fully setting up mail.

Nullmailer Installation And Configuration

The installation was simple as most Ubuntu thing are:

sudo apt-get install nullmailer

I usually skip the TUI wizard inputs since I know what configuration I want, which is just the following in
/etc/nullmailer/remotes:

smtp.gmail.com smtp --port=587 --auth-login --user=@gmail.com --pass= --starttls

Testing

After restarting Nullmailer to pick up the new configuration, testing is pretty simple but requires mailutils:

sudo apt-get install mailutils
mail -s "test" <email_address> < /dev/null

HAProxy Versus Apache Performance

After setting up HAProxy, I was curious how much performance was degraded by introducing an extra hop.

The Test

As with my last test, this was not meant to be scientific. I used ApacheBench again, sending 10,000 total requests and modifying the concurrency until I started getting broken responses.

I also put my site into maintenance mode so I would only be testing a static page and not introducing any database calls.

Apache Standalone Results

150 concurrent requests was about where my site could consistently serve up the maintenance page.

$ ab -n 10000 -c 150 http://www.christopherbaek.com/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.christopherbaek.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        Apache/2.4.7
Server Hostname:        www.christopherbaek.com
Server Port:            80

Document Path:          /
Document Length:        771 bytes

Concurrency Level:      150
Time taken for tests:   39.216 seconds
Complete requests:      10000
Failed requests:        0
Non-2xx responses:      10000
Total transferred:      10710000 bytes
HTML transferred:       7710000 bytes
Requests per second:    255.00 [#/sec] (mean)
Time per request:       588.243 [ms] (mean)
Time per request:       3.922 [ms] (mean, across all concurrent requests)
Transfer rate:          266.70 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       78  371 1115.1     89   10300
Processing:    79  148 111.6     91    1227
Waiting:       79  148 111.6     91    1227
Total:        160  519 1114.8    185   10387

Percentage of the requests served within a certain time (ms)
  50%    185
  66%    415
  75%    450
  80%    714
  90%    743
  95%    990
  98%   5526
  99%   5561
 100%  10387 (longest request)
$ ab -n 10000 -c 160 http://www.christopherbaek.com/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.christopherbaek.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
apr_socket_recv: Operation timed out (60)
Total of 9852 requests completed

HAProxy And Apache Results

Pretty surprisingly, the number of concurrent requests my site could maintain dropped by 50% to 75!

$ ab -n 10000 -c 75 http://www.christopherbaek.com/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.christopherbaek.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        Apache/2.4.7
Server Hostname:        www.christopherbaek.com
Server Port:            80

Document Path:          /
Document Length:        771 bytes

Concurrency Level:      75
Time taken for tests:   49.783 seconds
Complete requests:      10000
Failed requests:        0
Non-2xx responses:      10000
Total transferred:      10710000 bytes
HTML transferred:       7710000 bytes
Requests per second:    200.87 [#/sec] (mean)
Time per request:       373.370 [ms] (mean)
Time per request:       4.978 [ms] (mean, across all concurrent requests)
Transfer rate:          210.09 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:       76  260 759.9     87   11087
Processing:    78  101  53.5     88     960
Waiting:       77  101  53.5     88     960
Total:        160  360 761.8    175   11175

Percentage of the requests served within a certain time (ms)
  50%    175
  66%    179
  75%    184
  80%    379
  90%    642
  95%    656
  98%   4111
  99%   4185
 100%  11175 (longest request)
$ ab -n 10000 -c 85 http://www.christopherbaek.com/
This is ApacheBench, Version 2.3 <$Revision: 1706008 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking www.christopherbaek.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
apr_socket_recv: Operation timed out (60)
Total of 7691 requests completed

Conclusion

I was pretty surprised how much the performance dropped. Maybe this is why a lot of people use nginx as a proxy instead of HAProxy… Although this was some good information to discover, the database is still probably the bottleneck, so this shouldn’t become a problem anytime soon.

WordPress And Jenkins Behind HAProxy

I wanted to take a break from planning the automated upgrade of this site from WordPress 4.4.2 to WordPress 4.5 and decided to try setting up HAProxy. If the day comes where I need to scale this site, that would probably be the first thing I would set up so I could scale my Apache servers horizontally.

HAProxy Installation And Configuration

The installation was as simple as most Ubuntu things are:

sudo apt-get install haproxy

1.5 seems to be the latest version right now, but on Ubuntu 14.04, you get HAProxy 1.4.24. I just went with 1.4.24 because I didn’t want to add any apt repositories I’d end up forgetting about.

The configuration was pretty straightforward:

  1. Enable HAProxy to be started on reboot (in /etc/default/haproxy)
    ENABLED=1
    
  2. Configure a backend for Apache that checks on all its “load balanced” instances (in /etc/haproxy/haproxy.cfg)
    backend apache
    	mode http
    	option forwardfor
    	option httpchk HEAD / HTTP/1.1rnHost:localhost
    	server apache01 127.0.0.1:8000 check
    
  3. Configure a backend for Jenkins that checks on all its “load balanced” instances (in /etc/haproxy/haproxy.cfg)
    backend jenkins
    	mode http
    	option forwardfor
    	option httpchk HEAD /jenkins HTTP/1.1rnHost:localhost
    	server jenkins01 127.0.0.1:8080 check
    
  4. Configure a frontend (in /etc/haproxy/haproxy.cfg)
    frontend frontend 
    	bind 192.168.0.100:80
    	mode http
    	option http-server-close
    
    	acl url_jenkins path_beg /jenkins
    	use_backend jenkins if url_jenkins
    
    	default_backend apache
    

Apache Configuration Update

I had a configuration to have Apache act as a proxy to my Jenkins server, but now I was able to remove that since HAProxy was doing the work.

I also didn’t need Apache listening on all interfaces since HAProxy would be handling all incoming connections, so I updated that as well (in /etc/apache2/ports.conf).

Restart

rsyslog had to be restarted for HAProxy logs to start showing up since installing HAProxy just installs the configuration file. After restarting Apache and HAProxy to pick up their new configuration changes, everything worked!

Jenkins Behind An Apache Proxy

When I actually upgrade this site from WordPress 4.4.2 to WordPress 4.5, I think it would be pretty cool to automate as much of the process as possible. To try it out, I’m going to install a Jenkins server, and to make it a little more interesting, I’m going to try to set up access to the Jenkins server through my Apache server so I have a more unified experience access resources on my server (i.e,. I don’t have to type in any ports).

Installing Jenkins

The official documentation is here, but for brevity, here are the commands I ran to get Jenkins installed:

  1. Add the Jenkins apt key
    wget -q -O - http://pkg.jenkins-ci.org/debian-stable/jenkins-ci.org.key | sudo apt-key add -
  2. Add the stable apt repository
    sudo sh -c 'echo deb http://pkg.jenkins-ci.org/debian-stable binary/ > /etc/apt/sources.list.d/jenkins.list'
  3. Update
    sudo apt-get update
  4. Install
    sudo apt-get install jenkins

Configuring Jenkins To Sit Behind Apache

I wanted to access the Jenkins server via a subdirectory of my website (e.g., http://www.example.com/jenkins), so the only interface it needed to listen on was localhost. The server also had to have a context root configured since it would be behind an Apache proxy.

The PREFIX variable was already set in /etc/default/jenkins, so all I had to do was add these two parameters to JENKINS_ARGS:

  1. Listen only on localhost
    --httpListenAddress=localhost
  2. Serve the application from a context root
    --prefix=$PREFIX

I also removed the AJP connector.

Configuring Apache As A Proxy

To configure Apache as a proxy, I had to enable mod_proxy and mod_http and configure the proxy by adding the following to my virtual host:

ProxyPass         /jenkins  http://localhost:8080/jenkins nocanon
ProxyPassReverse  /jenkins  http://localhost:8080/jenkins
ProxyRequests     Off
AllowEncodedSlashes NoDecode
<Proxy http://localhost:8080/jenkins*>
    Order deny,allow   Allow from all
</Proxy>

Note that the proxy location matches the context root of the Jenkins server.

Restart

After restarting Jenkins and Apache to pick up the configuration changes, everything worked beautifully!

Creating A Simple Website Maintenance Page

I’m planning to upgrade this site from WordPress 4.4.2 to WordPress 4.5 soon. WordPress makes the upgrade process really easy for a small site like this, but I thought this would be a good opportunity to try some new things. I already tested a backup and restore of the database. Now I thought it would be a good opportunity to try creating a solution for putting a site into maintenance mode that I could use when I upgrade WordPress.

Maintenance Page Benefits

Aside from setting user expectations, a major technical benefit of serving a static page during maintenance windows is that backend resources will be shielded from live traffic, freeing them up for maintenance. A few maintenance examples:

  • Deploying new applications
  • Modifying configurations
  • Modifying infrastructure

Technical Requirements

The requirements for my ultimate solution were pretty standard:

  1. Simple to understand
  2. Simple to maintain
  3. As instantaneous as possible
  4. Frees up as many resources as possible
  5. Something that could potentially be automated

WordPress Maintenance Mode

The built in WordPress maintenance mode was the first potential solution on my list. It comes with WordPress, and there are even popular, actively supported plugins that make toggling maintenance mode a one-click operation. The problem with this solution is that it relies on the WordPress engine, which might interfere if I’m trying to upgrade or even replace WordPress itself. Also, while a plugin may help with manual administration, it doesn’t help as much for automated administration.

mod_rewrite, mod_headers, and .htaccess

These Apache features were what I ended up using in my final solution. Basically, the idea was to intercept requests via an .htaccess file and redirect them to a maintenance page. mod_rewrite is tried-and-true, the configurability is extensive, to say the least, and the WordPress engine doesn’t have to be triggered.

Separate Document Roots

I originally had a single DocumentRoot directory, but I renamed that directory, created a new one containing only my maintenance page and an  .htaccess file, and created a symlink from my configured DocumentRoot path to my WordPress directory. This change was transparent to the user, which was the goal.

.htaccess file

Header Set Cache-Control "max-age=0, no-store"
RewriteEngine On
RewriteCond %{SCRIPT_FILENAME} !maintenance.html
RewriteRule ^.*$ /maintenance.html [R=503,L]
ErrorDocument 503 /maintenance.html

This does a few things:

  • The Header directive hints to search engines that they should not cache this page if they happen to scrape my site while it is in maintenance mode
  • The remaining configurations takes any request and serves up the maintenance page with a 503 status code

Toggling

All I have to do to toggle maintenance mode for my site is point the DocumentRoot symlink from my WordPress directory to my maintenance directory and back!

Caveats

    • mod_rewrite and mod_headers must be enabled
    • AllowOverrides directive must be properly configured so the .htaccess file isn’t ignored
    • FollowSymLinks must be properly configured
    • Be mindful of the .htaccess file in the WordPress directory
    • The rewrite engine is not something I need all the time, but now it either has to be loaded all the time or it has to be toggled with maintenance mode
    • The rewrite rules could be a lot more sophisticated if something like controlled access is required

Backing Up And Restoring WordPress

When I migrated my blog from WordPress.com and on to my own DigitalOcean droplet, I installed WordPress 4.4.2. Only a few days later, WordPress 4.5 was released! WordPress definitely makes the upgrade process easy, especially for a simple setup like mine, but I thought this would be a good opportunity to test a backup and restore before actually upgrading.

Backup

Since I chose MySQL for my database on install, I planned on using mysqldump for the backup. To exercise the backup a little bit, I created a few published posts, a few scheduled posts, and a few drafts. A few things I noted:

  • The output is just a bunch of SQL statements, i.e., no secret sauce or funky formats
  • This command dumps all the tables and data from the database:
    mysqldump -u <username> -p <database_name> > <backup_filename>
  • Changing the command this way adds DROP and CREATE database statements, which might create more precise backups, but would then couple the backup with the WordPress configuration, users, permissions, etc.:
    mysqldump -u <username> -p --databases <database_name> > <backup_filename>
  • The --all-databases flag could be used to create a backup of the entire database
  • The mysql table could be dumped for user permissions
    mysqldump -u root -p mysql > <backup_filename>
  • For a mission critical scenario, the documentation should be consulted since there are a lot of different options, but I decided to just use the basic command

Restore

Restoring from the mysqldump backup went smoothly. The process depends on the type of backup (e.g., entire database, user permissions, etc.), but since I just went with the basic backup, it was just a couple of commands.

  1. Enter the mysql prompt:
    mysql -u <username> -p
  2. Read in the backup:
    mysql> source <backup_filename>

Gotchas

In a mission critical scenario, access to the site needs to be controlled.

This didn’t work:

  1. Verify website is working
  2. Create database backup
  3. Drop all tables
  4. Check what website looks like… Okay, it’s the install page
  5. Restore database
  6. Check what website looks like… It’s still the install page?

This worked:

  1. Verify website is working
  2. Create database backup
  3. Drop all tables
  4. Restore database
  5. Check what website looks like… looks the same!