User:Frank.farmer/Blog/Implementing Server-Side Detection for WebP

=Implementing Server-Side Detection for WebP= MAR 20, 2017 #performanceThere are several solutions that help reduce bandwidth usage. We compress the HTML pages during the transfer, minify CSS and Javascript files or use lazy-loading to fetch resources on demand. All of those methods allow us to reduce network usage and result in faster page loads. But when looking at the details of the pages we serve at Fandom, we can easily notice that the biggest offender in terms of network usage are images, which take around 80-90% of the whole data transfer. Can anything be done about it?

WebP image format to the rescue
In recent years a lot of work has been put into media compression and we could see a lot of improvements on this field. One that can help us with the image bandwidth usage is the WebP format. Here are some quick facts about it: While desktop utilities often struggle with WebP format, it has a pretty good coverage in Web Browsers. It is supported by Chrome and Opera, which is around 70% of the market share. Firefox is actively working on the WebP support, which means, even more users will be able to benefit from it in the future.
 * it was developed by Google back in 2010 (and is being improved – the latest release is from January 2017)
 * it supports both, lossless and lossy compression so it can replace JPEG and PNG images
 * on average WebP image files are smaller by 25-35% than their corresponding JPEG and PNG images

WebP: the client-side solution
At Fandom the WebP support was implemented back in 2014. It was a JS-based solution. In the case of WebP support was detected in the browser, JS would append a “format=webp” parameter to an image thumbnail URL. This would trigger a format converter on the server-side and as a result, the browser would get the WebP version of the image.

In January 2017 we took a closer look at our WebP usage. Each request comes with all the necessary info: we have the Accept browser header that can tell us whether the browser supports WebP and also we have the “format=webp” url parameter that is set in case browser asked for WebP. This allowed us to measure our WebP image format utilization: WebP usage – JavaScript solution This graph shows image requests sampled from our CDN and split by the WebP support and utilization. Requests from browsers without WebP support are shown in red. The green color is used for requests that ask for WebP. And blue color is for requests that could handle WebP, but the “format=webp” parameter was not specified so we served the image in other format. Quick math shows that we were serving WebP to 17% of the clients. And that we could do much better than that, as additional 45% of the requests could also handle WebP, but we didn’t take advantage of it. The main reason for low WebP usage was: As you can see the JavaScript-based solution resulted in some severe limitations. Luckily nowadays we can use the…
 * JavaScript WebP detection was enabled only for lazy-loaded images that make sense. Top 4 images on our pages are not lazy-loaded, as the are almost always visible and we want to start downloading them as soon as possible, without waiting for the lazy-loading JavaScript code to kick in. But this also means the top images didn’t have the WebP support.
 * Due to our image serving backend restrictions, the WebP conversion was supported only for images that were requested as thumbnails. Full-size images were never served as WebP, which is a pity as obviously, they are bigger in size.
 * Our custom JavaScript was required for the images to be served as WebP.
 * Without it, the format parameter required to ask for WebP was missing. This means that images requested outside of our main app stack (like Wiki background referenced from CSS, images linked from external sites or shown in Google images search) were never served as WebP.

Server-side solution
When a browser sends a request, it comes with an Accept header, which gives us the information about the content types and can be understood by it. At first, Chrome was simply sending the */* value in this header, and you couldn’t really tell if a browser supports WebP or not. Luckily this issue was addressed and nowadays browsers explicitly include the image/webp value in the Accept header. This means we can implement the whole WebP support negotiation on the server. Without the need for a custom JavaScript, which had the limitations mentioned above.

The idea is really straightforward – in case a browser asks for an image that can be converted to WebP, simply check the Accept header and do the conversion in case WebP is supported by the client. The most important thing to keep in mind was caching. We would serve the JPEG/PNG and WebP under the same image url, which could create caching issues. But as we know the response content type depends on the value of the Accept request header, we can send the Vary: Accept response header along with the image. This tells the caching layers that our response will differ depending on the value of the Accept header sent by the browser. So the cache keeps multiple responses under the same url and serves the correct one based on the client browser request.

The last minor detail was the request Accept header normalization – as you may imagine browsers send dozens of different variations of the Accept header. This would mean the cache layer would have a separate image copy for each value of the Acceptheader, which would decrease our cache hit ratio. That’s why we normalize the Acceptheader value on our CDN layer – in a case of image requests we either set the header value to image/webp or an empty string. This way we are ensuring that each image will have at most two copies in the cache – an original image format and a WebP version for browsers that support it.

Rolling out the changes
In mid-February, we pushed our changes for image thumbnails. The server-side WebP detection was live on production and we disabled the JavaScript solution. During the rollout we were monitoring the incoming image requests: WebP thumbnails rollout The decreasing orange bars correspond the JavaScript WebP detection that we disabled – it was dropping fairly quickly, although there were still 3% of leftovers – URLs cached in various places. We can either wait for them to expire or purge them. WebP images served by our new server-side solution are shown in green. As the images are cached in our CDN up to one year, the clients were still receiving non WebP images from the cache. This was something we expected and it was actually good, as we didn’t want to be hit by millions of new WebP requests immediately after the rollout. But as the images were dropping out of the cache very slowly, the new solution wasn’t able to quickly replace the disabled JavaScript. As a result, our thumbnails WebP support dropped down from initial 26% to around 17%. To speed the things up a little bit, we started purging frequently used images from our CDN which increased the WebP thumbnails usage to over 40%. And right now, after purging most of the images, it is around 62%%.

In the meantime we also implemented the WebP support for full-size images – as their average size was much bigger than in the case of thumbnails, we were expecting more gains there. We released this change at the end of February (and had a second round of image cache purging). To get the confirmation, we compared the bandwidth requests details from before and after the release. This is an image request split by content type at the beginning of February: Before the release: Requests by content type The WebP usage was around 17%. Here is how it looked shortly after the release: After the release: Requests by content type 54% confirmed our solution was working as expected. Right now, as we were able to refresh our image cache, it is over 60%. To make sure we were able to reduce the size of the image responses, we monitoring the chart showing the total response size from our servers: Daily bandwidth usage The periodic four spikes each month correspond to weekend peeks when we serve more traffic. Our two WebP releases are marked with vertical green lines. After releasing WebP thumbnail changes we were able to bring the bandwidth usage down to the levels from Q4 2016, which is pretty good itself given that we serve more requests than 4 months ago. After the second release and additional cache purge, the network usage went down by additional 12%.

The big spike on the graph in the middle of February was related to another project at Fandom, so WebP changes don’t take credit for neither of its slopes. Overall we are estimating that on average increasing the WebP usage allowed us to reduce the average image size by over 20%. That is noticeable to us and also to our users, who can see the images being downloaded quicker.

Can we do even better than that?
Although the result we saw after the release were very good (and corresponded to the estimates we made before the project), there are still few extra options we may consider:
 * So far we’re generating WebP from static images in JPEG and PNG formats. Our Fandom users like to include animated GIF files in their articles. A number of those GIF images is smaller than in the case of other formats, but because the GIFs are not really the best way to share short scenes from your favorite TV series or games, they make up for 12-15% of our traffic. We will consider using animated WebP or other, more suitable formats, for serving this content.
 * Recently Google released Guetzli, which can reduce the size of JPEG images by 35% and still keep their original quality. This new tool is quite resource heavy, which is something to keep in mind. But as we are able to encode each image only once and keep the result in our cache, this is a very interesting option to consider. It uses a format that has been around here for quite w while and all the browsers support it. So this can benefit the users without WebP support and also reduce our storage size in case we decided to recompress some of the images.

Summary
Our WebP project was not very innovative. The WebP format is not new and we cannot even tell we introduced it to Fandom, as it has been here for over 2 years. Still were able to predict (and achieve) 20% gains.

Optimization is a fun challenge. As in many other areas, you have to be up-to-date with new solutions and their pros and cons. It also requires to you has quite a good understanding of your whole stack, as you’re likely to go through several layers of if when looking for opportunities and applying changes. But at the end, it is very rewarding when done properly and the benefits are clearly visible. My best advice is – do your homework before you start coding. Doing the research, gathering metrics and having a good monitoring will help you a lot along the way!

Author: Jacek Woźniak