Chapter 12: Web Site Architectures

• Apply redundancy consistently. Don’t just address some single points of failure. Begin with the most vulnerable systems and components.
• Don’t waste money by applying redundancy at both the component level and across multiple servers within the same architectural tier.
• Include an analysis of the reliability of your web-hosting service’s infrastructure when developing your plans for redundancy.
• All other things being equal (and except in the case of database servers - an exception that’s discussed in more detail later in this chapter), it’s better to use a larger number of redundant inexpensive servers than a smaller number of high-reliability ones.
• Consider software license fee peculiarities when trying to optimize your web-site architecture. Expensive licenses may favor the use of a smaller number of larger servers.
• Web-hosting service and MSP fees for rackspace and server management may also favor the use of a smaller number of larger servers.
• If your site will be hosted on a shared server, but you need increased reliability, look for a web-hosting service that offers clustered servers. It’s usually only slightly more expensive than using non-clustered servers.
• Add a load balancer (or use load-balancing services from your web-hosting service) as a cost-effective way to increase reliability.
• An architecture is not fully redundant unless it includes redundant switches, load balancers, and links to your web-hosting service’s routers.
• Database servers are the one case where it’s best to build redundancy into a single server before going to a multi-server architecture.
• A hot-standby database configuration is a good intermediate step between a single-server design and a fully-redundant architecture.
• Use a database cluster to achieve the highest levels of single-location database redundancy.
• Unless you need the very high availability that can be achieved only through fully-distributed architectures, you can probably get the other benefits of such a configuration at a much lower cost by using a content delivery network.

Chapter 13: Caching and Content Delivery Networks

• If you need to deliver more static content than your current web server(s) can handle, consider adding a reverse-proxy cache instead of another web server.
• CDNs using the centralized model (4-30 edge servers) tend to work best for web sites with low to medium traffic volumes. High-volume sites may do better with a CDN based on the distributed model.
• For guaranteed performance levels, look for a vendor that offers pre-loading of your content.
• Vendors that use URL re-writing allow you to get up and running on their CDNs with very little investment and little risk of lock-in.
• Create separate virtual servers for images and other types of static objects (e.g., audio and video files). This will give you the option of moving these objects to dedicated servers or having them managed by a CDN without making any changes to your HTML or applications.
• Create aliases with CNAME records or delegate subdomains to your CDN rather than give it control over all of your DNS.
• Be wary of CDNs that require you to modify your URLs in ways that are unique to one vendor. If you decide to use a CDN that requires modification of your URLs, try to make the changes in a manner that’s automatically reversible, for instance by changing an environment variable or configuration parameter.
• Look for a CDN that supports ESIs (edge-side includes) to deliver dynamically-generated content directly from its edge servers.
• If you’re considering using a CDN for the delivery of on-demand streaming content, look for one that offers pre-loading as an option.
• Be aware of the effects of third-party caching on your log files, particularly if your web site generates revenues from advertising.
• Ask prospective CDN vendors if they participate in any content peering initiatives or if they plan to do so in the future.
• Running your own tests, using your own content, is the only good way to evaluate the performance of CDNs.
• When evaluating CDNs, use third-party measurement services that measure from locations as close as possible to the visitor’s browser. Ask the measurement-service provider how it addresses the problem of the proximity of its monitoring probes to the CDN’s edge servers.
• Don’t use published comparisons of CDNs, or vendor-supplied test data as the basis for comparing content delivery networks in lieu of running your own tests.
• Do some snooping on web sites that appear to have content and traffic volumes similar to yours. Find out what CDNs they use, then contact the site owners to find out their levels of satisfaction with those CDNs.
• Make sure prospective CDNs support the file types and streaming formats you intend to use, and that they operate edge servers capable of handling those content types in the regions from which your visitors will access your web site.
• If consumers are important to your business, ask prospective CDNs how they optimize content delivery to AOL users.
• Evaluate the financial stability of prospective CDNs in the same way as you would a web-hosting service or MSP.
• Include in your evaluation of vendors any CDNs that have relationships with your web-hosting service or MSP. But don’t limit your tests to only those CDNs; you may find another that’s better for your site.

Chapter 14: Connectivity Practices

• Don’t use a web-hosting service that doesn’t purchase transit service from at least two major ISPs unless the web-hosting service is, itself, one of the major ISPs.
• If a vendor has multiple, linked data centers, ask how those connections are used. The best answers are “never, ” “always,” and “for distributed architectures.” Stay away from “sometimes.”
• Ask a prospective vendor if it has more than one data center in the area. If so, investigate how the data centers are connected to one another, and where the links to the major ISPs enter each of the buildings. Try to locate your servers in a facility that is as independent as possible from problems in another facility.
• Ask your vendor what its policies are for upgrading the speed of its links. They should have such a policy in writing. Better yet, get an SLA that states that any link that exceeds one third of its capacity when measured using a monthly 95th percentile rule will promptly be upgraded.
• Don’t use a web-hosting service that has any link slower than a DS-3.
• Evaluate vendors’ connectivity by capacity/load ratio, not by capacity alone.
• Don’t use a web-hosting service that makes use of any bandwidth-choking technologies.
• Be aware of the market pricing for bandwidth. It’s a commodity and prices are continuing to drop.
• Plan for (or at least consider) success. Calculate what you’ll pay for bandwidth if your web site is successful beyond your wildest dreams.

Chapter 15: Storage

• Use DAS (direct attached storage) unless you (a) have more than 200GB of data, (b) need access to a single file system from multiple servers, or (c) need very high performance.
• Use NAS (network attached storage) if you have between 200GB and 10TB of data, or you need access to a single file system from both Unix and Windows operating systems.
• Use SAN-based storage for database servers and clusters if you need (a) maximum performance and reliability, (b) more than 200GB of database storage, or (c) a significant volume of write/update operations. Stick with DAS and NAS for the storage of static content, and DAS for smaller databases.
• Don’t use a storage service provider (SSP) for your web site unless you have truly extraordinary storage or uptime requirements. Even then, consider that management of your storage may be a core competency that you must maintain in house.
• If your web site includes more than 300GB of combined static and streaming content, consider using a GSS (global storage service), perhaps in combination with a CDN.

Chapter 16: Backup and Recovery

• For maximum protection against the propagation of corrupted data, make full archives daily and keep them forever.
• Make sure your archive strategy addresses undetected data corruption. If you can’t afford to retain all your full archives in perpetuity, adopt a plan for media rotation. But be aware that any time you re-use or rotate media you have some risk of loss due to the propagation of corrupted data.
• Ask prospective vendors how long it will take them to recover a single file from tape. Check their data-recovery SLAs, if they have them.
• If your site is small and simple, consider using your development system to back up your content in lieu of some of your vendor’s extra-cost services, such as archiving, tape rotation, and off-site storage.
• Use a revision control system in your development environment as part of your complete backup and recovery strategy.
• Use magnetic disk backup as intermediate storage between your live site and your web-hosting vendor’s tape archiving system.
• Unless you have intermediate magnetic disk backups, make sure you have copies of your most recent data and content nearby, and another almost-as-recent copy off site at all times.
• Review your backup and archive logs daily. Check that any recently added files and directories are included.
• Run data recovery tests before servers are brought on line, and on a regular basis thereafter. Test for both complete recovery to a virgin server as well as recovery of individual files and databases.

Chapter 17: Security

• Consider a good backup and recovery plan as the starting point for any security defense. Anything more should be justified on a cost/benefit basis.
• There’s only one good way to deploy firewalls: Configure your web site as though you didn’t have a firewall, and then add one anyway.
• Don’t use a firewall in front of your web servers unless access to your static content is restricted to authenticated visitors and unauthorized access would cause a substantial loss.
• In three-tier architectures, install a firewall between your web servers and your application servers.
• Always use packet filtering in the switches, routers, and load balancers that connect your web site to the Internet.
• Don’t use shared firewalls under any circumstances.
• If you want to deploy a network-based intrusion detection system, consider placing it in front of your application servers rather than in front of your entire web site.
• If you decide to use a network-based IDS, consider outsourcing the reaction services to an MSS (managed security service).
• Use a host-based IDS or data-integrity monitoring package as a low-cost substitute for a network-based IDS.
• Never re-use the same password on more than one server or account.
• Ask prospective vendors how they manage passwords. Give the highest marks to those that use token-based authentication systems.
• As a way to evaluate a vendor’s security policies, telephone the vendor’s NOC, pretending to be an existing customer. If you’re able to get a password to a customer’s server or to the vendor’s administrative web site, there’s a serious problem.
• Be wary of vendors’ shared administrative networks. Configurations in which the network itself is not shared are safer.
• If you have an administrative network, secure it with a firewall.
• Never store credit card numbers in your web-site database, even if they’re encrypted. Use a third-party credit-card processing service that will store your customer’s credit-card numbers for use in future transactions.
• Ask vendors what security services they monitor or subscribe to, how promptly they install security-related patches, and if they test patches for compatibility with applications before installing them.
• If your total annual web-site budget is greater than $1 million and your risks are substantially greater than just the loss of your static content, use an MSS (managed security service) to add knowledge, experience, and objectivity beyond what an MSP can provide.

Got another tip you think we should add to the list? Email it to tips.