Notes on Adaptive Images (yet again!)

All the cool kids are doing responsive design, it seems. This means fluid designs, having some hot media query action to reformat your layout depending on screen size, and ensuring your images are flexible so they don’t break out of container, generally by setting them to {max-width:100%;}.

Having images scaling down presents a problem though, if you’re a performance-minded sort of chap(ette). Why send a 300K 400 x 800 image that would look lovely on tablet device attached to wifi, but which takes ages to download on a 3G phone, and which gets resized by the browser to fit a 320px wide screen? Not only are you wasting your user’s bandwidth and time, but not every browser is created equal and not every resize algorithm makes pleasing end results. The CSS 4(!) image-rendering property can help with this but it only hints to the browser.

Sending the right-sized image to devices without wasting bandwidth is one of the knottiest problems in cross-device and responsive design at the moment. In the 24ways advent calendar series of articles, the subject has been discussed twice in eight articles (by Matt Wilcox and Jake Archibald). There are numerous other techniques, as well, such as tinySrc and the Filament Group’s Responsive Images.

All these are very clever, different solutions to solve the same problem, and they all rely on scripts, or cookies, or server-side cleverness or (in the case of Jake’s ideas) dirty hacks and spacer GIFs. I’m reminded of Image Replacement techniques from more than 6 years ago, which were over-engineered solutions to the problem better solved by CSS web fonts.

Let’s recap. We have several images, of different sizes, and want to send only the most appropriately-sized image to the browser. The circumstances differ. The canonical use case is to send smaller, lower-resolution images to smaller screen-sizes on the assumption that connection speed is slow and they have low-resolution displays. This isn’t the case, though. Some people are using retina displays on fast wifi networks. SO, while currently CSS Media Queries allow us to detect screen width and pixel density, we need new media features such as network speed/ bandwidth.

The DAP is working on a Network Information API, there’s a Phonegap version for native apps, and a Modernizr detect, but using script for this seems much harder than being able to access it via Media Queries, and if you just want to rearrange CSS layout, CSS is the place to detect it. (Purists may argue that network connection isn’t a characteristic of device, but in your face, purists!)

Once you have a media query, you can swap images in and out using the CSS content property:

<img id=thingy src=picture.png alt="a mankini">
@media all and (max-width:600px) {
 #thingy {content: url(medium-res.png);}

@media all and (max-width:320px) {
 #thingy {content: url(low-res.png);}

@media all and (network-speed:3g) {
 #thingy {content: attr(alt);}

A browser that doesn’t support Media Queries, or doesn’t report “true” for any of the Media Queries shows the picture.png, which is the src of the img. A browser that is less than 600px replaces picture.png with medium-res.png. A 320px or narrower browser replaces picture.png with low-res.png. A browser that is only connected via 3g replaces the image with its alt text.

I first researched this technique in 2008 as a way of doing image replacement without extra markup (ironically enough). The first two media queries only works in Opera and Chrome at the moment, as they’re the only browsers that fully support content without :before or :after. (The network-speed media query works nowhere as I just made it up).

Recently, Nicolas Gallagher experimented with generated content for responsive images, and unfortunately discovered that there are no advantages to the technique because browsers always download picture.png, even if they never show it because they immediately replace it. Perhaps this could be optimised away by browsers, but there would still be the double-download problem with current browsers.

My mind turned to HTML5 video. Here we have an element that has the ability to specify multiple sources (because of different codecs) and can also switch sources according to media characteristics. It borrows syntax from Media Queries but puts them in the HTML:

<source src=high-res.webm media="min-width:800px"> 
<source src=low-res.webm> 
<!-- fallback content --> 

I firmly believe that if something as new-fangled as video can be responsive in a declarative manner (without needing script, APIs, cookies or server magic) then responsive images should be similarly simple.

Previously, on the mailing list, a new <picture> element was proposed. I mashed up that idea with the already-proven video code above and came up with a strawman:

<picture alt="angry pirate">
<source src=hires.png media="min-width:800px">
<source src=midres.png media="network-speed:3g">
<source src=lores.png>
   <!-- fallback for browsers without support -->
   <img src=midres.png alt="angry pirate"> 

This would show a different source image for different situations. Old (=current) browsers get the fallback img element. As I said, this is a strawman; it ain’t pretty. But it would work. (It’s discussed in my recent Smashing Magazine article HTML5 Semantics.)

I prefer the media queries to be in the HTML for two reasons: you don’t need to have ids or complex selectors to target a particular image, and (more importantly) many content authors use a CMS and have no ability to edit the CSS. (Although the nasty <style scoped> could solve this.)

On the other hand, I might be over-engineering the whole problem. I chatted informally with my colleague Anne van Kesteren, glamorous Dutch WHATWG inner circle member. There’s a school of thought that says everything will be 300ppi and networks will be fast enough, so this is really an intermediate problem until everyone starts using highres graphics and all displays go from 150 to 300. Standards are long term, and by the time we have a standardised version, the problem might have gone away.

What do you think?

(Matt Machell pithily pointed out “if only browsers hadn’t forgotten the old school lowsrc attribute for images“.)

Looks like our chums at WHATWG are discussing this too.

79 Responses to “ Notes on Adaptive Images (yet again!) ”

Comment by Matt Wilcox

I think we really need to question our assumed best-practices with regard to header information. Yes, the argument that it will add bytes to all network requests is true – but is that a worse thing than continuing along this dumb-network path?

I really want to avoid the “here and now” mentality that landed us with CSS not accepting % as valid units for border-width. Just because something isn’t used *right now* does not mean that it isn’t a good idea to implement.

We’ve seen the need for additional headers for years and years – instead we abuse the UA string. It has become excessively bloated, arcane to use, and increasingly un-reliable because browser vendors are abusing it. Send headers. It’ll add bytes. Are a few bytes a problem measured to:

a) the potential that becomes available when the server is aware of the client’s capabilities.
b) the massive saving of bandwidth and file size content negotiation can make.

Bare in mind that while it won’t work mean an automatic net gain for desktop, it’ll almost always result in a net gain for mobile, and the internet is headed for mobile.

I wonder if SPDY could work around the supposed “oh my god no never the internet will fall to it’s knees” horror of sending a couple more bytes per request?

Comment by Matt Wilcox

Or, what’s to stop the browser sending a header on first connection to a domain that says “I can send additional data per request if you like” and the server sending a “yes please, for the following data types: jpg, gif, png”.

Default to sending nothing extra, and when you get a response that says it’d like some more details flip the client behaviour and re-request the original resource.

Comment by Carsten

Just my two cents.

Standard image tag:
<img src=”path/to/image.png” alt=”Alternative Text” width=”960″ height=”520″ />

Standard video tag:
<video width=”640″ height=”480″ controls=”controls”>
<source src=”path/to/movie.mp4″ type=”video/mp4″ />
<source src=”path/to/movie.ogg” type=”video/ogg” />

Meta Tags:
<meta name=”resolution” size=”320″ src=”{data[mediaurl]}{*-320}.{data[extension]}” />
If screen resolution is 320px or lower: Browser loads “path/to/image-320.png”, “path/to/movie-320.mp4”, “path/to/movie-320.ogg”

<meta name=”resolution” size=”480″ src\”{data[mediaurl]}{*-480}.{data[extension]}” />
If screen resolution is between 321px and 480px: Browser loads “path/to/image-480.png”, “path/to/movie-480.mp4”, “path/to/movie-480.ogg”

<meta name=”resolution” size=”800″ src=\”{data[mediaurl]}{*-800}.{data[extension]}” />
If screen resolution is between 481px and 800px: Browser loads “path/to/image-800.png”, but still loads “path/to/movie.mp4”, “path/to/movie.ogg”

Comment by Marc Brooks

Why not make the browser tell us?
Since we’re talking about changing the browser’s behavior anyway, why not actually make it offer up the information to properly solve this problem server-side? Let’s have the browser send the information that is available to the css media queries in the HTTP GET request headers caused by the one-and-only whatever.png tag or CSS url(“whatever.png”).

To wit:
X-Media: “screen color:24 resolution:120dpi orientation:portrait device:640×480 viewport:500×300”

(obviously, the aspect-ratio and device-aspect-ratio can be computed, color-index seems meh)

The nice thing about this is that it is really up to the user-agent’s current needs to determine what should be displayed, thus what should be requested. If someone zooms in, the user-agent can ask for a bigger version. If they switch to print view, we can ask for the monochrome version in higher DPI and all is warm and fuzzy. If the server doesn’t care, it serves up the one-true-image. If the server cares, it can serve specific versions. Cache logic can be driven by the standard content-negotiation logic (and left non-cached until it is).

The best part of ALL of this is that the CSS can just reference an image, the html can just include a img tag, etc… and the browser’s request can be driven by what it knows already about the size/placement/device of the image.

Comment by What The New iPad’s Retina Display Means for Web Developers | TransAlchemy

[…] Notes on Adaptive Images (yet again!) — Opera’s Bruce Lawson rounds up problems and solutions facing anyone trying to serve up different images based on screen size. Related Posts:What The New iPad’s Retina Display Means for Web DevelopersAnalyst Stands by Report: Samsung Only iPad Display Maker Shipping En MasseNew iPad Shows Up 3 Days Early in Vietnam ? Benchmarks, Unboxing EnsueCorporate Types Are iPad-Crazy, Survey FindsTED-Ed, Khan Academy Enable Flipped Classrooms This entry was posted in Wired and tagged community, html, iPads, Jason Grigsby, Responsive Image Working Group, Vector Graphics, web developers by TransAlchemy. Bookmark the permalink. […]

Comment by {Best Souvenirs} |

[…] Notes on Adaptive Images (yet again!) — Opera’s Bruce Lawson rounds adult problems and solutions confronting anyone perplexing to offer adult opposite images formed on shade size. Thanks for passing by: ↓ Name* […]

Comment by Rani

It seems like there is a thoughts overload here. I also agree about having media queries in HTML so that the worry about editing the CSS will not be a fuss. What’s more important for me is that there’s always the versatility in images to keep their quality as expected.

Comment by Interview With Bruce Lawson of Opera | JTB Productions

[…] I think the web stack is in pretty good shape these days. There’s work to be done making sure that sites can work offline (Appcache-done-right, in whatever guise it comes back) and with web payments. The lack of any useful way for developers to deal with responsive images is a problem, 18 months after it was flagged up. […]

Comment by Interview With Bruce Lawson of Opera

[…] I think the web stack is in pretty good shape these days. There’s work to be done making sure that sites can work offline (Appcache-done-right, in whatever guise it comes back) and with web payments. The lack of any useful way for developers to deal with responsive images is a problem, 18 months after it was flagged up. […]

Comment by Interview With Bruce Lawson of Opera | GMancer

[…] I think the web stack is in pretty good shape these days. There’s work to be done making sure that sites can work offline (Appcache-done-right, in whatever guise it comes back) and with web payments. The lack of any useful way for developers to deal with responsive images is a problem, 18 months after it was flagged up. […]

Leave a Reply

HTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong> . To display code, manually escape it.