It's About Time You Looked at More Than Just the Code

With my business partner, I ran ha.ckers.org, one of the most attacked websites on the planet. We averaged around one million attacks per year over its seven year lifespan. We didn’t have the luxury of ignoring the security threat or wishful thinking. As such we had to thoroughly research our options. The general advice on how to secure a web application falls into a few major buckets: scan, audit, and patch. It’s sound advice to recommend scanning a web application for vulnerabilities, and using human reviewers to ensure that nothing slips through the cracks. It’s also perfectly reasonable to recommend that people patch up to the most current and secure version. But there are several very large problems with this approach. The concept of scanning websites, and auditing code, assumes that they can find all of the vulnerabilities in a system. We’ve seen that to be untrue countless times. Although doing both is still a good idea, they are anything but foolproof techniques in identifying all the flaws in a web application. Secondly, patching assumes that the patch actually fixes the problem, doesn’t introduce new flaws, is available to be installed, and actually is installed faster than the attacker can take advantage of the flaw. Many times people don’t even know the patch is available or that they need to patch.

As a result, other technologies have been introduced to try to fill in the gaps. The most notable is the web application firewall (WAF) which is an inline technology designed to stop attacks as they come in. While a great concept that reduces the time to fix, the WAF has a similar problem to the web application vulnerability scan — it assumes that it can find any exploit and stop the threat with next to no false positives. There have been dozens of presentations and papers on why WAFs are only mostly effective, and shouldn’t be used as your only line of defense. Not to mention that some WAFs can actually introduce new vulnerabilities into the environment.

Like many others, I ran a website and was too busy to audit every line of code myself, too cheap to scan it using commercial scanners, and too untrusting of the patches which I knew would come with new problems and not just fix the old problems. Not that patching is bad, but it cannot be relied on to be the only solution and it often, counter intuitively, comes with extra “features” that can introduce additional security threats. Like others, we too looked at our options. The scan/audit/fix model didn’t work for us in lots of ways. A new approach was required.

Instead of focusing on fixing the code, we had to come up with a model that would ensure that even if the code was vulnerable, it wouldn’t matter and the site would be secure. We assumed anything we deployed would have at least one flaw. The issues we were most concerned with were command injection, local file includes and remote file includes (all three of which allow an attacker to take over the site), SQL injection (which allows an attacker to copy, modify and delete data in your database), man-in-the-middle attacks (which allows the attacker to sniff credentials) and brute force of the administrative user (which gives the attacker access to the admin console). Each of the aforementioned are typically considered “game over” by hackers — as that gives them what they need. In our environment, even these terrible exploits, if successful, should ideally give the attacker nothing. As such we came up with a list of 10 major tenants in our hosting model:

1. No web-application source code must be modified to take advantage of the new architectural security. Like everyone, we’re too busy for anything else.
2. All administrative functionality should be preserved, but at the same time limited in scope to administrators who should have access. This retained functionality should include the process of updating the code when dealing with third party CMSs/blogs/etc. which often have standardized patching processes.
3. All modifications to web application files and source code should be disallowed by anyone other than the administrator by default.
4. The effect of all command injection and SQL injection attacks should be limited to the scope of the application in question and should not affect other applications hosted on the system.
5. All security enhancements must be assumed to have at least one vulnerability; therefore, security enhancements should have a least one backup wherever feasible.
6. Web developers should not require root/system administrative access to perform their job functions (privilege separation and enforcement). This is embodied by the theory of “least privilege” — no user has access to more than they are required to have. The only exception to this rule is when updates to the webserver software or programming languages are required.
7. No extraneous services or applications should be installed and all installed services must be pre-configured to be more secure than distro-provided defaults. The list of removed applications includes but is not limited to games, superfluous drivers, and unneeded services.
8. The system should assess itself for performance/stability issues, security flaws, and unauthorized access attempts.
9. All logs should be guaranteed to be forensically secured from an attacker, even post-compromise. Ensuring that your logs haven’t been tampered with is incredibly important when doing investigations and pursuing legal action, for instance.
10. Information about — and source code of — one web application should not leak to another application on the same device through this architecture, even post-compromise.

A typical LAMP stack was not an option as it did not offer the level of security we needed. Among many other problems, a default LAMP stack has no separation of administrative functionality for the web server; it often encourages webserver owned files (e.g., the .htaccess file, upload directories, and so on), it logs to local disc, it allows modification to system binaries after a local root exploit, unrestricted access to the Internet, etc. None of which is good from a security perspective.

For ha.ckers.org, we chose to use FreeBSD as the operating system of choice for several reasons. FreeBSD has a great security track record, has standard deployment schedules for security patches, and — most importantly for us — it has the concept of jails, which are similar to a chroot under Linux, only with security built in. Effectively, it is very similar to a virtual machine, only much more lightweight.

Utilizing the concept of jails, an application can dramatically reduce its attack surface area through a series of cross mounts (Image at right). In the simplest example, consider an environment with a parent operating system that hosts three child jails. The first jail is an “administrative jail” that the website administrator uses. The second jail is the “public jail” that is utilized by all public Internet traffic. A third jail (or separate physical machine) contains a database and is called the “DB jail.” It is reasonable to assume that the public jail will come under attack by both robotic attacks and determined and sophisticated adversaries. Each jail has a dramatically different security profile.

For instance, an administrator may need to upload images, text, movies or even source code. The administrator may also need to read personally identifiable information about their users, including credit cards. A user from the Internet, on the other hand, may only need to log in, leave comments in the database, or pay for items using a credit card. If the Internet user were able to perform any task that is normally reserved for the administrator, it could be devastating to the system.

The varying security requirements of each user are important to understanding why the concept of jails is important to a partitioned security model. However, there is a risk of incurring significant extra work if there are simply two copies of the same application, which may just double the work for an administrator, and leaves significant margin for error. To get around this, a series of file mounts are created to cross mount the application in a very particular manner as to allow all of the code to be duplicated, but still allow certain files and directories to be excluded from the list of direct copies.

This exclusion list may contain an administrative user and password to the database enabling an administrator to read credit card data. Meanwhile the settings file within the public jail can contain only limited privileged account access to the database, enabling the public user to submit credit cards only. As a redundant security control, the public jail cannot log into the administrative DB account — even if the attacker manages to extract or guess the correct administrative database password. This prevents command injection and SQL injection attacks from leveraging their exploit to perform any more tasks than are already allowed for that database user or local web user.

Although the database is still writable, the web directories as well as the root file system of the public jail are marked as read-only to prevent attackers from modifying the code. One apparent disadvantage of a read-only file system is that logs cannot be written to the local jail, but that’s actually a good thing. First, logs contain sensitive information that attackers could possibly read — including personally identifiable information, IP addresses of users and so on. Second, logs can sometimes contain cookie data which, when replayed in the attacker's browser, can allow them to authenticate without knowing the password. Third, logs can be leveraged in local file include attacks (as can any writable file that has a known location). Lastly, logs can be tampered with by an attacker to cover their tracks after a successful compromise.

With that said, it would seem that logs are almost more trouble than they are worth in some respect. However, logs are still extremely important for forensics and auditing. Therefore all logs still exist, but instead of being written to the local jail they are shuttled out of the public jail from which they originate, and sent to either the administrative jail (in the case of webserver logs) or to the parent (in the case of security event logs). In this way, logs are still available but remain forensically secured. It is important to provide real-time web server logs to administrators who often need them for debugging purposes.

Jail separation provides another valuable resource to limit dumping of data that must be at least partially accessible from the public jail. Good examples of this are passwords and credit card numbers, but any sensitive data could theoretically be substituted. Stored procedures are typically meant to stop SQL injection and to improve performance; they are not typically designed to prevent data from being copied and pulled off the machine after a compromise, but that is unimportant. Stored procedures can be granted privileges to return information from a database table that is otherwise inaccessible to other queries. Therefore, in the simplest example, a stored procedure could be created that submits the following query: “for X username and Y password hash combination, return the username and password hash if found.” As such, even if an attacker does gain command execution, sensitive data must still be brute forced in-place, as opposed to offline cracking. This makes it significantly more difficult for an attacker to extract any meaningful amount of information without submitting millions of database requests, which can be detected. This method could protect nearly any kind of sensitive data from being dumped, including but not limited to passwords, credit cards, session tokens, secret question/answers, and so on.

Because the public jails are essentially unaware of what is occurring in both the administrative jail and the parent operating system, this gives the parent some distinct advantages. For instance, content integrity monitoring is typically known to an attacker, and may even be vulnerable to direct attack — allowing the attacker to disable or modify the monitoring application in a normal compromise. But because content integrity monitoring can reside in the parent operating system, it can monitor the public jail for modifications to disc without the user in the public jail being aware that the jail is under surveillance. This monitoring also includes, but is not limited to, process auditing, login attempt monitoring, and so on.

Another powerful feature is that all jails are mounted with their base OS marked as read-only. This preserves the jail integrity while still allowing site administrators to have root access to their own jail to upgrade programming language libraries, web servers, and so on. Maintaining the environment in this way has other advantages too — removing insecure transport mechanisms, like FTP and forcing administrative traffic over a secured SSH/VPN/HTTPS tunnel. It effectively becomes IP based access control without additional work in the code (Image at left).

The following chart shows an example of the delta between a typical LAMP stack, the Armored Stack and a version of the Armored Stack where the website has been modified to take advantage of additional security controls provided.

Ha.ckers.org allowed us to test these theories out in a very hostile real-world environment. People are often wary of throwing out old technology in favor of new, but in this case, re-writing the security model for hosting our own applications was critical for our own survival. We took these lessons learned and created a LAMP stack replacement called the Armored Stack™ which provides significantly more security than a default LAMP stack. Customers have already successfully deployed Drupal, WordPress, and Expression Engine. This environment, while great from a security perspective, does require some work. We partner with consulting companies to help with the installation and maintenance of the applications.

And how effective was all of this for ha.ckers.org? After seven years and north of seven million attacks, the site was still never compromised. No security is perfect, but the Armored Stack gave us an acceptable level of risk.