others comments on othersites for reference only

1. Do you have numbers representing how the same status codes break down by User-Agent? e.g. are people using IE more likely to have an empty cache than Firefox?
   Were the logs cleaned first to remove non-browser access (i.e. crawlers, etc)?
   
   Comment by Evan — January 4, 2007 #
2. When you say users does that mean “users logged in to Yahoo!”? Does the experiment do any filtering of traffic to exclude bots,scrapers,etc?
   
   Comment by Joe Preston — January 4, 2007 #
3. I believe this result may be a side-effect of the cache size and replacement algorithms of the browsers. (My Firefox cache size is listed as 200MB, and IE7 shows 1GB, but no mention is made of an upper limit on number of individual files.)
   I’ve observed this on my own applications. Where I have thumbnails set to expire a month in the future, quite often the browser will re-fetch the images if I don’t visit the page for a couple of days. The large number of tiny images and scripts my browser has seen in the meantime have pushed my cacheable images out of its cache.
   
   Comment by Martin Kenny — January 4, 2007 #
4. @Evan, @Joe: The experiment contained a filter for removing traffic from robots, crawlers and internal ips. To further strengthen the validity of the data, we even applied additional filters such as removing users generating more than 40 page views per day (since generating that many page views in a single day was highly unlikely for a real user).
   
   Comment by Tenni Theurer — January 4, 2007 #
5. Has there been any research on the total time taken for the 304 handshakes, especialy given that DSL lines are much slower in the request direction, and that the request can be quite large (500-700) bytes.
   
   Comment by Tim Hawkins — January 5, 2007 #
6. Tha’s logical:
   The cache is a danger for privacity (try Picasa!) and spend many sectors in HD (1 byte file = 32 kb in many modern HD)
   Then many people (with ADSL conections) goes to:
   IE: Tools/Internet Options/Advanced Options and X the last one: Empty temp folder…
   Firefox: Tools/Options/Privacity/Clean always…
   
   Comment by Yano — January 5, 2007 #
7. Great article. What tool(s) did you use to filter the logs?
   
   Comment by Scott Hovey — January 5, 2007 #
8. [...] A month ago, I linked to the first article in a ‘Performance Research’ series by the YUI folks in which they explored how the number of HTTP requests impacts application performance. The second part of the series looks at caching. [...]
   
   Pingback by Ajax Performance » Real world caching numbers from the YUI blog — January 5, 2007 #
9. @Tenni: Do you have any way of knowing that the “users” generating more than 40 page views a day were not just a group of users behind a corporate proxy server?
   
   Comment by Joe Pearson — January 5, 2007 #
10. [...] Tenni Theurer has posted a second part of performance research on Browser Cache Usage – Exposed!. [...]
    
    Pingback by Ajaxian » Browser Cache Usage: Only 40-60%? — January 5, 2007 #
11. What was used to create the graphs for figure 1? I looked for ages to find something like that and ended up writing a very crude version using curl. But it didn’t make nice graphs for me like that.
    t
    
    Comment by Toby — January 5, 2007 #
12. Tenni, very nice to see a very identical graph of your experiement. I did a little bit different test some months ago but came to a very similar result. On my test the page views percentage is higher because I think of less page count at all.
    
    Comment by Michael Schwarz — January 5, 2007 #
13. Very interesting article. I find this really interesting information to see.
    However, I this Martin Kenny could have an interesting point in his comment.
    I’m going to use this information in a recommended im making. Thanks
    
    Comment by Dougal Matthews — January 5, 2007 #
14. Performance Research, Part 2: Browser Cache Usage – Exposed!…
    This is the second in a series of articles describing experiments conducted to learn more about optimizing web page performance. You may be wondering why you?re reading a performance article on the YUI Blog. It turns out that most of web page performan…
    
    Trackback by Func. News — January 5, 2007 #
15. It would be very interesting to find a way to see what the reasons for the higher % of cache misses that were seen from the client’s side of things.
    That might require some sort of quick survey to determine either cache settings or how people use their cache (as one other person mentioned, for security some people have it set to clean always)
    
    Comment by Dan Smith — January 5, 2007 #
16. We use a transparent Squid proxy (http://www.squid-cache.org/) in our office as browsers don’t seem to do a good job of managing or using their own caches. It makes for a much more pleasant and zippy browsing experience! Unfortunately, it does not work with SSL requests, and I’ve found that some browsers (notably Safari) are particularly bad with using its cache for secure sites.
    
    Comment by Patrick — January 5, 2007 #
17. I am new to doing fancy Javascript development, but I read recently that caching proxies will pretty much ruin your AJAX experience.
    That makes sense, but what about caching in the browser? What are the downsides to browser caching?
    
    Comment by Chris Blow — January 5, 2007 #
18. Squid is a pretty intelligent proxy server, and it does look at the various headers sent by the real web server. For an application that uses AJAX, it’s not going to cache the dynamic responses sent back from the server; what it does cache are images, style sheets, and script files which are the real culprits in slowing down an overall page request.
    
    Comment by Patrick — January 5, 2007 #
19. I’m looking at the graphs here, but I don’t agree with your conclusions. I think what you’re really seeing is that 50% of your traffic is made up of frequently-returning users, while the other 50% are people who seldom visit — you know grandma who dials up once a month to check her email. The steep initial drop is representative of the frequent visitors, while the slow-sloped line is representative of the other group. Had you run your experiment longer, I think you would have found that the line for unique users would have continued a downward trend until it converged with the line representing page views without caching. This also IMHO better explains the giant gap between the views and unique visitors.
    
    Comment by Dave Bialac — January 5, 2007 #
20. Just to supplement my previous comment and support my case a little more, note the 2 small bumps on the users curve that do NOT appear on the pages curve. These bumps fall exactly 7 days apart — a spike likely caused by more users viewing on that particular page.
    
    Comment by Dave Bialac — January 5, 2007 #
21. @Chris: One downside to browser caching is that your users might not be using the latest version of a file. You can get around this by adding a version number to the filename at the cost of having to rev the filename and update all references to that file. Since most components are fairly static, caching them with a far future date helps save on HTTP requests (for full cache page views).
    
    Comment by Tenni Theurer — January 5, 2007 #
22. The metrics will not clearly say anything about users with empty cache. Letz say I visit one of the pages where this tracking is done for the first time, my browser will do a empty cache request to the server. Then from the next time my browser will recieve a 304 (not modified) response. So the first 200 response should not be considered in the metric, therefore
    The percentage of users with an empty cache is:
    # of unique users with at least TWO 200 response / total # of unique users
    Whatever metrics you guys have put up is invalid. It doesn’t depict anything about users who clears the cache everytime or browsers with settings that ignores cached files. Moreover the percentage of users with empty cache who are first time visitors to a website is 100%. The data that you guys have put up now is based on the total number of visitors to the website on that day of experiment. The user who has visited the website the first day is not required to visit the website on the second day. The graph depicts on how many new visitors are visiting the tracking pages every day. Not the users who ignore cached files.
    
    Comment by prathap — January 6, 2007 #
23. Keep in mind that full page requests are done when a page is force-reloaded by the user as well, and all elements on the page will be requested again regardless of cache status of the reloaded elements.
    I suspect that with a portal site such as Yahoo that forced page refreshes are a non-trivial number of requests.
    
    Comment by Scott Anderson — January 7, 2007 #
24. [...] Performance Research, Part 2: Browser Cache Usage – Exposed! This is the second in a series of articles describing experiments conducted to learn more about optimizing web page performance. You may be wondering why you’re reading a performance article on the YUI Blog. It turns out that most of web page performanc (tags: performance user-experience webperf) No TagsLicenseThis work is published under a Creative Commons Attribution-NonCommercial-ShareAlike 2.5 License.   « links for 2007-01-05 |   [...]
    
    Pingback by links for 2007-01-06 » dougmcclure.net — January 8, 2007 #
25. @Toby: Microsoft Excel has a feature under Chart Wizard > Custom Types > Floating Bars that can generate the graph like you see in Figure 1 and 2. For showing the components of an HTML page, I use IBM Page Detailer.
    @Dave: I think you make an interesting point. We do run this experiment on different Yahoo! pages to determine correlations with usage patterns. Next time we’ll run it over a longer period of time and share our findings here.
    @prathap: The purpose of the experiment was to find the percentages of users and page views visiting Yahoo! pages with an empty cache each day. Whatever reason why the image was not in the browser’s cache, the fact is the browser had to download the entire image again and that was what we were measuring. When the percentages dropped to a steady state, the returning users (with a full cache) reached a constant.
    
    Comment by Tenni Theurer — January 8, 2007 #
26. [...] Tenni Theurer has posted a second part of performance research on Browser Cache Usage – Exposed!. [...]
    
    Pingback by Browser Cache Usage: Only 40-60%? — January 9, 2007 #
27. [...] Performance Research, Part 2: Browser Cache Usage—Exposed! A fascinating look at the impact of client-side caches on the real-world performance on web sites. In particular, we see that Yahoo! sees 40-60% of its visitors arriving with no cached content, so the uncached load time of your site is very important. (tags: http) [...]
    
    Pingback by SitePoint Blogs » News Wire: How’s Your Uncached Performance? — January 9, 2007 #
28. [...] Daha önce bu makalenin birinci kısmının linkini vermiştik. Web sitelerinde performansı arttırmanın yolarını üzerine Link [...]
    
    Pingback by Fatih Hayrioğlu’nun not defteri » 12 Ocak 2007 Web’den seçme haberler — January 11, 2007 #
29. [...] I finally got around to reading the second article on browser performance optimisation on the YUI blog. While the techniques aren’t new to me (I work on the Yahoo! front pages for Europe remember), some of the data was. “40-60% of Yahoo!’s users have an empty cache experience and ~20% of all page views are done with an empty cache” [...]
    
    Pingback by Kid666 Blog » Blog Archive » Browser-Cache-A-Go-Go — January 15, 2007 #
30. Let’s not forget, even a cache-lookup takes time.
    If you could guarantee that most/all of your users had a full cache all the time, it would STILL be better and faster to concatenate static files wherever possible. Images should be sprite-ified, and CSS and JS should be minified.
    In fact, according to my rough measurements, having a single script tag instead of several can significantly reduce the time required to parse and execute the code, even if we’re talking about inline scripts, where cache has no effect.
    In short, caching is great and all, but don’t count on it to solve any problems. Build for the empty-cache situation, and you’ll usually benefit all of your users.
    
    Comment by Isaac Z. Schlueter — January 16, 2007 #
31. 40-60% users with an empty cache seems pretty high. I could understand that for Firefox users as they are considered more tech savvy to know how to delete private data. but I doubt that for the majority of IE users and normally they stand for the bigger piece of cake in the browser game…
    
    Comment by 信用卡, 計算器 — January 17, 2007 #
32. [...] This week’s feature article on the 10 Rules to Code By generated a lot of comments (as well as 800+ guest visitors and 30+ new subscribers). One of the hottest discussions was about my claim that “there is never a valid need to add inline styling” to your markup. In the course of that debate a guest reader (named Matt) offered up some links from the internal Yahoo team about their performance research: The 80/20 rule and the truth about browser cache usage. [...]
    
    Pingback by Browser and Page Performance » Professional Web Software Development — January 25, 2007 #
33. [...] “A single large file is fastest.” (Nate Koechley) That’s why Yahoo! apparently has such an amount of inline CSS. They found out browser caching is not as effective as they thought, in particular not on a user’s start page. So they deliver “inline” CSS. Actually writing inline CSS is a maintenance nightmare, but delivering CSS content inline doesn’t mean the files can’t have separate lives on the server: concatenate the files with a server side technique of your choice. [...]
    
    Pingback by Learning the World » Website Performance Tweaks — January 25, 2007 #
34. CSS- und WebSite-Performance mit CSS-Sprites und Sliding-Doors…
    Heute mal zum Thema, wie bekomme ich müde bzw. langsame Webseiten munter.Zuerst der schöne Artikel »Website Performance Tweaks« (s. 3. und 4.):learningtheworld.eu/2007/performance/CSS-Sprites: was ist das, was bringt das, wie geht das:yuiblog.com/bl…
    
    Trackback by 11com7 Knowledge-Blog — January 26, 2007 #
35. [...] Of course with Yahoo’s performance research about browser cache and HTTP requests you could discuss if external files load faster with several server requests through script src="foo.js" or just one request and internal HTML code, but either way you should use central, non-redundant files and include them in one way or another. [...]
    
    Pingback by Learning the World » Best Practices in Web Development — January 26, 2007 #
36. Really cool study. I second Evan request to see the differences of cache behaviour between browsers.
    
    Comment by Monkeyget — January 27, 2007 #
37. [...] Our approach in the past has been to just cache everything and forget about it. Unlike many public websites that have to operate under the assumption that the user’s cache will be empty, we can depend on a fairly high percentage of populated caches. And, because we have a single page application design, even if they don’t have the JavaScript cached, they only have to download the files once and they’ll be available the entire time the application is being used. [...]
    
    Pingback by Spider Bytes » Blog Archive » Smart JavaScript Caching — February 9, 2007 #
38. [...] Wer mag kann inzwischen auch mal bei meinem Bruder, CSSImport, CSSBeauty, CSSGuy ,F-LOG-GE oder Alistapart vorbeigucken, für Zwecke der Seitenoptimierung empfehlen sich noch diese, diese und diese Seite – seeehr interessant. ersteres bei Interesse an Bergen, schönen Bildern und Tourberichten, letztere für CSS/Webwork-Interessierte. Und weil es immer iweder Spass macht: Beetlebum. [...]
    
    Pingback by Mal was anderes… - Cywhale.de:CoreBlog — February 12, 2007 #
39. [...] The last one is the topic I had in mind with the this blog’s title. Since Yahoo will be using the same locations for the libraries you need and since Yahoo is the most popular site, chances are your visitors have already checked their Y! mail or their Y! finance page and searched or done anything on the Yahoo network of sites. This means they have already requested and (hopefully) cached these .js files. And as proved before, lowering the number of HTTP requests is top 1 performance optimization you can ever do. [...]
    
    Pingback by phpied.com » Blog Archive » YUI good for your performance — April 2, 2007 #
40. High Performance Web Sites: Rule 1 – Make Fewer HTTP Requests…
    [Steve Souders is Yahoo!’s Chief Performance Yahoo!. This is one in a series of blogs describing the best practices he’s developed at Yahoo! for improving performance. This article is based on Chapter 3, Rule 1: Make Fewer HTTP Requests from……
    
    Trackback by Yahoo! Developer Network blog — April 4, 2007 #
41. [...] Not everyone has cache turned on and many people who do have it turned on don’t let files fester in there forever, so there’s no guarantee the files you dump onto the user’s hard drive will actually be there the next time they visit. Yahoo published some numbers on this subject in January: “40-60% of Yahoo!’s users have an empty cache experience and ~20% of all page views are done with an empty cache.” [...]
    
    Pingback by No Mod Required » Blog Archive » Don’t plan around a users second visit or page view… — April 10, 2007 #
42. We can use apache deflate_module to compress the content before sending it to the client.
    http://httpd.apache.org/docs/2.0/mod/mod_deflate.html
    
    Comment by Anand Muthu — April 11, 2007 #
43. 14 rules for fast web pages…
    Steve Souders of Yahoo’s “Exceptional Performance Team” gave an insanely great presentation at Web 2.0 about optimizing website performance by focusing on front end issues. Unfortunately I didn’t get to see it in person but the Web 2.0 talks……
    
    Trackback by Skrentablog — May 9, 2007 #
44. [...]     http://yuiblog.com/blog/2007/01/04/performance-research-part-2/ [...]
    
    Pingback by » How to Build Insanely Fast Web Pages · Tech Blog · Techasaur.us Rex - Tech News Encyclopedia | Yesterday, Today & Tomorrow’s Tech News — May 12, 2007 #
45. [...] Performance Research, Part 2: Browser Cache Usage – Exposed! [...]
    
    Pingback by Top methods for faster, speedier web sites — May 23, 2007 #
46. [...] http://yuiblog.com/blog/2007/01/04/performance-research-part-2/ Tags: performance, optimization, cache, browser, http, web, programming, caching(del.icio.us history) [...]
    
    Pingback by Performance Research, Part 2: Browser Cache Usage - Exposed! » Yahoo! User Interface Blog — June 10, 2007 #
47. [...] for improving performance for first time visitors. As described in Tenni Theurer’s blog Browser Cache Usage – Exposed!, 40-60% of daily visitors to your site come in with an empty cache. Making your page fast for these [...]
    
    Pingback by Yahoo! представи YSlow - Netlabs Bulgaria Blog — July 26, 2007 #
48. [...] for improving performance for first time visitors. As described in Tenni Theurer’s blog Browser Cache Usage – Exposed!, 40-60% of daily visitors to your site come in with an empty cache. Making your page fast for these [...]
    
    Pingback by Thirteen Simple Rules for Speeding Up Your Web Site at think room — July 26, 2007 #
49. [...] Performance Research, Part 2: Browser Cache Usage - Exposed! Performance Research, Part 2: Browser Cache Usage – Exposed! [...]
    
    Pingback by Performance Research, Part 2: Browser Cache Usage - Exposed! « ilab — July 29, 2007 #
50. [...] with including version numbers in the file names) saves basic HTTP calls for component data in about 3/4 of all page visits. Bad Penguin explains how to cache from the code [...]
    
    Pingback by BlogSchmog » TuningSchmuning — August 5, 2007 #
51. [...] Browser Cache Usage – Exposed! [...]
    
    Pingback by Client side web site performance — onenaught.com — August 6, 2007 #
52. nice post.. btw: did you tried YSlow against the blog site ? eheh – Performance Grade: D (68)
    
    Comment by Felipe Gaucho — August 8, 2007 #
53. [...] an empty cache experience and about 20% of all page views are done with an empty cache (see this article for more information on browser cache usage) This fact outlines the importance of keeping web pages [...]
    
    Pingback by Julien Lecomte’s Blog » Introducing the YUI Compressor — August 13, 2007 #
54. [...] for improving performance for first time visitors. As described in Tenni Theurer’s blog Browser Cache Usage – Exposed!, 40-60% of daily visitors to your site come in with an empty cache. Making your page fast for these [...]
    
    Pingback by Howto: Create Faster Web Sites (13 rules to speed up website) « ****Linux rules the heart of its users. This is enough a reason why one should embrace Linux******* — August 15, 2007 #
55. [...] Glaubt man den Ingenieuren von Yahoo, haben 40% bis 60% der Yahoo Benutzer Erfahrungen mit leerem Cache und 20% aller Seitenaufrufe erfolgen gar ohne Cache (mehr zu diesem Thema in diesem Blogeintrag). [...]