In Search of... A better, faster, stronger Web

This is a great presentation by Marissa Mayer (Google) on optimizing for speed, and evaluating how your users respond to different user experience experiments.

The Google search business generates $10 billion/year.  In 1999, Google indexed 30 million pages.  Today they index over 300 billion pages, and their concern about optimizing speed and the end user experience is more important than ever.

'Tis the season for..... Load Testing

Holiday retail madness is coming at us fast.  Is your site ready?    Over the years I have load tested hundreds of sites for capacity planning and deployment readiness.  Below are my top 10 tips for success in accurately measuring performance of your site.

1. Start off testing a single server, not the farm/cluster.  Originate the load from the same network switch to remove all possible network latency.   I prefer Silk Performer for this.
   
2. Test the application pages first.  No assets, no images, no CSS, no .JS includes.   Typically page generation by the application server bottlenecks long before the web server and static content.  When doing this, hits/sec should equal pages/sec.
3. Initially make each test transaction a new user.  This will aggressively show you what impact your session persistence has on server memory growth.
   
4. Test site content and order processing pages individually at first, then combine the tests.   Order processing pages will always be more process intensive than site content that is most often cached by the app server, CDN, or on the client.  Testing individually and then combined will show you how they impact each others page performance and define how you optimize going forward.
5. Stress test by eliminating simulated "Think Time".   In early testing you need to focus on levels of concurrency, not total users on the site.  Concurrency translates directly to maximum pages/sec, transactions/sec, and data throughput. 
   
6. Under load/stress pay close attention to 3rd party resources (database).  Understand what is responsible for increasing page generation response times.  Allowed concurrency can be tuned on the app server to manage memory growth, CPU utilization, and 3rd party resource saturation.  Request queuing is not bad and needs to be used to optimize and throttle request threading rather than letting unlimited requests eventually crush the server.
7. Endurance test.  Test with high concurrency/load for 8 to 12 hours.  Small memory leaks not previously seen will be found here.  This is your stability certification.
   
8. Re-test with all page assets loading.  Hits/sec will no longer equal pages/sec, note changes to CPU levels and page response times with all assets loading.   Use these results to make client and CDN caching decisions.
9. Increase load and re-test with all servers in the farm/cluster.  Pay special attention to what happens to the performance of 3rd party resources (database) now with all web servers under heavy load.
10. Test in the wild.   Get outside the firewall/DMZ and test with Gomez Reality Load.  Gomez Reality Load can send MASSIVE amounts real load from the internet to your site allowing you to fully test all components and hosts in your site and report performance on each host and asset.   Your site pages and assets might run as expected under load, but will your 3rd party hosts?  CDN providers, advertisers, page analytics plug-ins?  At this point, take the opportunity to test your fail-over scenarios.   Don't take anyone's "Word" for it.  Fail-over strategies are like data backups.  If you have not actually tested them in the wild, they don't work.
    

This is a very compressed top 10 with a lot of missing detail.   If you would like to know more, feel free to contact me.

Do Small Changes in Performance Really Matter?

The Velocity 2009 "Web Performance and Operations" conference took place in San Jose this year on June 22.   Attending the conference were developers, engineers, and managers from companies of all sizes, focusing on web performance.   So, do even the smallest changes in performance matter?  Lets have a look at what some of the big players had to say.

• Microsoft (Bing) found that 2 additional seconds in response time reduced the number of searches per user by 1.8% and decreased revenue per user by 4.3%.
  
  
• Google found that a 400 millisecond delay resulted in 0.59% fewer searches per user and that even after the delay was removed, these users still had 0.21% fewer searches, indicating that a slower user experience affects long term behavior.
  
  
• Shopzilla conducted a year-long performance redesign reducing response times by 5 seconds (from ~7 seconds to ~2 seconds) resulting in a 25% increase in page views, a 7-12% increase in revenue, and a 50% reduction in hardware.
  
  

Source:  Steve Souders @Velocity Conference 2009
http://radar.oreilly.com/2009/07/velocity-making-your-site-fast.html

Brand Reputation. Who's Watching ?

Assuming that brand reputation is important, every company must maintain some level of brand reputation management.   Although this might be the least concern of your IT operations manager and your developers, I guarantee its near and dear to the heart of your business owners, C level executives, and shareholders.

So who is watching?  Who's keeping score ?     Gomez is!
I'll eventually do a complete Gomez breakdown, but for now you should know that Gomez Benchmarking is keeping score.   The old saying goes "You can't improve what you can't measure".   You might not be measuring, but somebody else surely is.

According to a report by the Aberdeen Group entitled “The Performance of Web Applications — Customers Are Won or Lost in One Second” one second delays in response time significantly impact top business goals.

One second delays can reduce:

• page views by 11%
• conversions by 7%
• customer satisfaction by 16%

Who knows, maybe nobody will care that last quarter on-line sales were down by 2%, and that you ended up being the worst performer on this list.   Still think you don't need a Performance Management Strategy ?

Do you have a Web Performance Management Strategy ?

I have worked with internet technologies since long before the WWW gained household adoption.   I've watched the internet and its usage evolve from VT100 emulation, WAIS, MUDS, and Gopher to the WWW as we all know it today.   In 1994 the first online retailers appeared, and today this retail B2C market is responsible for $225 billion in revenue in the United States alone for 2009.

As with everything that changes or evolves, many themes stay the same.  One constant challenge through this evolutionary time line has been developing and managing a better "end user" experience for website visitors.   This topic can be broken into two categories.  One category addresses the DESIGN of the actual user interface that provides the navigation and features of the end user experience.  The other category addresses the PERFORMANCE experience, or in other words, how fast and reliable the end user experience is.

While working for Universal Mind Inc, and previously for Allaire and Macromedia.   I have conducted hundreds of performance related engagements on web applications and retail sites ranging from performance and stability tuning to capacity planning and load testing.   While both design and performance are critical to the end user experience, and ultimately translate into revenue for eCommerce and iBusiness stakeholders,  I'll be focusing on performance management strategies and technology, because not even the coolest or most intuitive rich featured web applications will retain customers or sell products if the site is unreliable or painfully slow.   Further more, speed (response time) is directly related to revenue conversion, capture and/or loss.   Because your site was performing great today doesn't mean it was performing great yesterday, or last week, and it certainly doesn't mean it will perform well tomorrow, on Black Friday, or on Cyber Monday.  Even a 1 second difference in the user experience can be connected to revenue.

You might have technical tools that measure performance points along your network layers,  and you might have user analytics to tell you what parts of the site your users visit most, but if you are responsible for a business critical web application and don't have a comprehensive performance management strategy that provides meaningful visibility, trending, and instrumentation into site, page, and service level performance, your visibility into the end user experience, capacity planning, and associated revenue gain or loss is severely limited.

I'll address these concepts here in my blog, topic by topic.   Enjoy!