How the Vary HTTP Header Can Be Bad

Vary is а powerful HTTP header that plays a significant role in how your website cache is working. When this header is set correctly, it ensures that your site visitors see the right content, regardless of the caching applied. However, if this header is set up incorrectly, it can totally overrule even the benefits of the best caching system and cause resource overusage. In this blog post, I want to shed some light on how the Vary header affects your SiteGround hosted site and specifically our caching system, and to show you what's the recommended use of Vary header for your site.

How Does the Vary Header Work?

The role of the Vary header is to indicate in which cases a different version of your site should be served. If your Vary header has one of the following values: User-Agent, Cookie, Referrer or even * (wildcard) it can significantly reduce the effect of our caching system. This happens because these values indicate that a different version of your pages should be served based on the browser type used (user-agent), the presence of unique cookie or referral URL or, in the worse scenario when * is used, this means that a fresh content will be served for every single visit, which will completely disable our cache.

Vary: User-Agent

Let’s take a look how the most commonly used value, User-Agent, works. It tells our Dynamic caching: “Hey, you should store different caches for different OS and browsers”. This is done in order to avoid serving a cached desktop version to a mobile visitor for example. Note, however, that nowadays most of the sites don’t actually serve different HTML for their mobile versions. It’s the responsive CSS that does all the heavy lifting and shows your site in a different way on mobile and on desktop. So, unless you’re using a plugin like WP Touch for example, or you know for sure you have a difference in the HTML output of your site, based on visitors browsers, your site should not be sending the Vary: User-Agent header.

Without the Vary: User-Agent set, our caching system will cache your site on the first load and then serve the cached version to the following requests unless you update your content or purge the cache manually. With the User-Agent enabled, the system will keep different copies for each combination of OS and browser version visiting your site. This means that you will have a dynamic request for the first person loading your site on a Desktop Safari, one for the first person loading a mobile Chrome, desktop Chrome and so on.

Let’s say you clear your cache roughly every 100 hits. Without Vary: User-Agent sent by your site, your site will have only 1 dynamic request in 100. On the other hand, with the header enabled, depending on your visitors profile, you will have 5-30 dynamic requests for the same 100 visits. So the very same site will use 5-30 times the resources it would need without the header being set.

Bad, Bad Bots

We hate malware traffic and bad bots. It’s been an ongoing effort to fight them in every possible way. I am mentioning bots here, because if you get spammed by them, the Vary: User-Agent header may determine how successful their attack would be and how being crawled by bots will affect your site performance.

You can think of a bot as a browser that does automated actions on your site. It may collect data for their own index, search for vulnerabilities, try to submit forms, etc. Bot’s User-Agent is simply a piece of text added by its creator. Reputable crawling machines like the Google bot have theirs set properly, while others may try to mimic the Bing bot, or the Google bot or even randomly generate a user agent.

If such bot starts crawling your site and you have the Vary: User-Agent header, each request it makes to your site would be a dynamic one and will eat up your resources. On the other hand, if the header is not used, you would serve cached requests to the bots directly from our Dynamic caching.

How to See Your Response Headers

First of all, you should check the headers your site is returning to the visitors. Usually, I use the Network tab of FireBug on my browser but there is a handy online checker (http://www.webconfs.com/http-header-check.php) that you can use too. There are multiple values of the Vary header and user-agent is just one of them. If you do not have a Vary header or it includes only Accept-Encoding value (that helps gZIP compression to work properly) you should not worry. However, if you see User-Agent or any of the other values (Cookie, Referrer or *) we recommend that these are removed from your header.

How to Configure the Vary Header

Generally, the best approach would be to understand which part of your site is generating it and reconfigure it. However, this might require some troubleshooting skills and may not be a very easy task. So if you’re on WordPress, you can check your headers and then replace the response with only what you want by adding the following lines to your functions.php file:

function replace_wp_headers($headers) {
$headers['Vary'] = 'Accept-Encoding';
return $headers;
}
add_filter('wp_headers', replace_wp_headers);

Alternatively, you can try using an .htaccess rules to unset the unnecessary Vary headers:

<ifModule mod_headers.c>
Header unset Vary
Header set Vary “Accept-Encoding”
</ifModule>

Doing this will remove all headers passed from your site and leave only the one specified in your code - Accept-Encoding. Of course, you can add more rules but always make sure you really need them and there's no other way to achieve the result you're after!

Product Development - Technical

Enthusiastic about all Open Source applications you can think of, but mostly about WordPress. Add a pinch of love for web design, new technologies, search engine optimisation and you are pretty much there!

22 Comments

  1. Reply June 21, 2017 / 08:39 BadanSiteGround Team

    Hello,

    Can we do the same for Joomla ?

    Regards.

    Alain.

    • Reply June 23, 2017 / 01:51 Hristo PandjarovSiteGround Team

      Our caching system will behave the same for Joomla sites. So you need to check it first, and if you see a header that's not necessary to remove it. Either from the extension that's generating it or using the provider .htaccess code.

  2. Reply June 21, 2017 / 16:38 Brian ProwsSiteGround Team

    I use CloudFlare. Is this correct?

    HTTP/1.1 200 OK =>
    Date => Wed, 21 Jun 2017 22:32:55 GMT
    Content-Type => text/html; charset=UTF-8
    Connection => close
    Set-Cookie => wpSGCacheBypass=0; expires=Wed, 21-Jun-2017 21:32:55 GMT; Max-Age=0; path=/
    Cache-Control => max-age=2592000
    Cf-Railgun => direct (waiting for pending WAN connection)
    Expires => Fri, 21 Jul 2017 22:32:55 GMT
    Host-Header => 192fc2e7e50945beb8231a492d6a8024
    Link => ; rel="https://api.w.org/"
    Vary => Accept-Encoding,User-Agent
    X-Cache-Enabled => True
    X-Proxy-Cache => MISS
    Server => cloudflare-nginx
    CF-RAY => 372a6d2fa8a92573-ORD

    • Reply June 23, 2017 / 01:53 Hristo PandjarovSiteGround Team

      That header shows that this is a dynamic request (X-Proxy-Cache => MISS) and you indeed have User-Agent added to the Vary header. It is correct but can be optimized for better caching ratios if the site permits it.

  3. Reply June 27, 2017 / 13:33 VaneSiteGround Team

    Hi,

    why doesn't SG add this setting feature to SG Optimizer?
    ( To modify .htaccess:

    Header unset Vary
    Header set Vary “Accept-Encoding”

    )
    ... and probably some other useful presets...

    BTW I have site speed issues which - I suppose - are caused by SG caching system, so your hint above might help to improve them - I hope.

    But doing it from SG Optimizer would be more convenient...

    Best,
    V.

    • Reply June 28, 2017 / 08:59 Hristo PandjarovSiteGround Team

      We can't add every optimization technique into the SG Optimizer plugin. Plus, the Vary header is used properly by many plugins and themes so it does not need to be removed completely in every case. Last but not least, the PHP service may add it after the .htaccess file is processed. So the way it can be removed...well varies 🙂

      • July 6, 2017 / 09:26 MarkSiteGround Team

        So does this mean that we shouldn't do what you've suggested because it is used in many plugins and themes? Will adding the code to the functions.php file break anything with the themes and plugins that use the Vary header? Example: Woocommerce

      • July 7, 2017 / 02:05 Hristo PandjarovSiteGround Team

        No, I would suggest that you try it and test if something breaks. In most of the cases unless you're using a special plugin for a separate mobile html output you don't need that header.

  4. Reply July 5, 2017 / 19:46 HassanSiteGround Team

    Hello Hristo Pandjarov,

    Informative article. I run the test and below is what i got. Could you suggest its all good or i shall change the header? if need to change, shall i just head to .htaccess and remove the header? Appreciate your help!

    Vary => User-Agent

    • Reply July 7, 2017 / 02:04 Hristo PandjarovSiteGround Team

      You can try using the functions.php code and keep it if nothing breaks.

  5. Reply July 6, 2017 / 00:09 MattSiteGround Team

    I got the following - is this ok?

    HTTP/1.1 200 OK =>
    Server => nginx
    Date => Thu, 06 Jul 2017 06:07:27 GMT
    Content-Type => text/html; charset=UTF-8
    Connection => close
    X-Pingback => https://mgpestcontrol.co.uk/xmlrpc.php
    Link => ; rel="https://api.w.org/", ; rel=shortlink
    Host-Header => 192fc2e7e50945beb8231a492d6a8024
    X-Proxy-Cache => MISS

    • Reply July 6, 2017 / 04:46 Hristo PandjarovSiteGround Team

      Yes 🙂

  6. Reply July 6, 2017 / 10:13 LoriSiteGround Team

    Hello,

    This is what was returned by the tool at webconfs.com - Do I need to make any changes?
    Thank you,

    Lori

    • Reply July 7, 2017 / 01:57 Hristo PandjarovSiteGround Team

      No, you're good 🙂

  7. Reply July 6, 2017 / 10:22 Dr. Hava CohenSiteGround Team

    my site Header

    My site is very slow. I'm not a programmer and I'm trying to figure out how to make it quicker, so your article came just on time. However, I don't see the "User-Agent" add, yet I see "X-Proxy-Cache=> Miss.
    Is this a problem?

    • Reply July 7, 2017 / 02:00 Hristo PandjarovSiteGround Team

      The miss says that your page is not cached Dynamically. Please, post a ticket in your Help Desk and ask our support to configure your Dynamic caching, your site will become much faster 🙂

  8. Reply July 6, 2017 / 10:33 Jon SymonsSiteGround Team

    Hi, could you take a look at mine and let me know if I have things set correctly. Thank you.

    • Reply July 7, 2017 / 01:59 Hristo PandjarovSiteGround Team

      Yes, you don't need to worry about your caching config 🙂

  9. Reply July 6, 2017 / 10:52 ChristianSiteGround Team

    I got the following - is this ok?

    • Reply July 7, 2017 / 01:58 Hristo PandjarovSiteGround Team

      Yes, you're good

  10. Reply July 8, 2017 / 22:18 LaurieSiteGround Team

    This is what I have. Is it okay?

    Vary => Accept-Encoding

    • Reply July 10, 2017 / 02:14 Hristo PandjarovSiteGround Team

      Yes 🙂

Reply

* (Required)