1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
|
<p>The web is going to get faster in the very near future. Sadly, this is rare enough to be news.</p>
<p>The speed bump won't be because our devices are getting faster, though they are. Nor will it be because some giant company created something great, though they probably have.</p>
<p>The web will be getting faster very soon because a small group of web developers saw a problem and decided to solve it for all of us.</p>
<p>The problem is images.</p>
<p>As of August 2014 the <a href="http://httparchive.org/interesting.php?a=All&l=Aug%2015%202014&s=Top1000">size of the average page in the top 1,000 sites on the web</a> is 1.7MB. Images account for almost 1MB of that 1.7MB.</p>
<p>If you've got a nice fast fiber connection that image payload isn't such a big deal. But, if you're on a mobile network, that huge image payload is not just slowing you down, it's using up your limited bandwidth and, depending on your mobile data plan, might well be costing you money.</p>
<p>What makes that image payload doubly annoying when you're using a mobile device is that you're getting images intended for giant monitors and they're being loaded on a screen little bigger than your palm. It's a waste of bandwidth delivering pixels you don't need.</p>
<p>Web developers recognized this problem very early on in the growth of what was called the "mobile" web back then.</p>
<p>More recently a few of them banded together to do something that web developers have never done before -- create a new HTML element.</p>
<h2 id="in-the-beginning-was-the-mobile-web">In the Beginning Was the "Mobile Web"</h2>
<p>Browsing the web on your phone hasn't always been what it is today. Even browsing the web on the first iPhone, one of the first phones with a real web browser, was still pretty terrible.</p>
<p>Browsing the web on a small screen back then required constant tapping to zoom in on content that had been optimized for much larger screens. Images took forever to load over the iPhone's slow EDGE network connection and then there was all that Flash content, which didn't load at all. And that was the iPhone. Browsing the web using Blackberry and other OSes crippled mobile browsers was even worse.</p>
<p>It wasn't necessarily the devices' fault, though mobile browsers did, and in many cases still do, lag well behind their desktop brethren. Most of the problem though was the fault of web developers. The web is inherently flexible, but web developers had made it fixed by optimizing sites for large desktop monitors.</p>
<p>To fix this a lot of sites started building a second site. It sounds crazy now, but just a few years ago the going solution for handling new devices like the Blackberry, the then-new iPhone and some of the first Android phones was to use server-side device detection scripts and redirect users to a dedicated site for mobile devices, typically a URL like m.domain.com.</p>
<p>These dedicated mobile URLs -- often referred to as M-dot sites -- typically lacked many features found on their "real" desktop counterparts and often didn't even redirect properly, leaving you on the homepage when you wanted a specific article.</p>
<p>M-dot websites are a fine example of developers encountering a problem and figuring out a way to make it even worse.</p>
<p>Luckily for us, most web developers did not jump on the m-dot bandwagon because something much better came along.</p>
<h2 id="responsive-design-killed-the-m-dot-star">Responsive Design Killed the M-Dot Star</h2>
<p>In 2010 web developer Ethan Marcotte wrote a little article about something he called <a href="http://alistapart.com/article/responsive-web-design">Responsive Web Design</a>.</p>
<p>Marcotte suggested that with the proliferation of mobile devices and the pain of building these dedicated m-dot sites, it might make more sense to embrace the inherently fluid nature of the web and build websites that were flexible. Sites that used relative widths to fit any screen and worked well no matter what device was accessing it.</p>
[image="pic-marcotte-example.png" caption="Ethan Marcotte's original responsive demo site at roughly phone, tablet and desktop screen sizes"]
<p>Marcotte's vision gave web developers a way to build sites that flex and rearrange their content based on the size and characteristics of the device in your hand.</p>
<p>Responsive web design isn't perhaps a panacea, but it's pretty close.</p>
<p>Responsive design started with a few more prominent developers making their personal sites responsive, but it quickly took off when Marcotte and the developers at the Filament Group redesigned the <a href="http://www.bostonglobe.com/">Boston Globe</a> website to make it responsive. The Globe redesign showed that responsive design worked for more than developer portfolios and blogs. The Globe redesign showed that responsive design was the way of the future.</p>
<p>While the Globe redesign was successful from a user standpoint, Marcotte and the Filament Group did run into some problems behind the scenes, particularly with images.</p>
<p>Marcotte's original article dealt with images by scaling them down using CSS. That made them fit smaller screens and preserve the layout of content, but it also means mobile devices were loading huge images that would never be displayed at full resolution.</p>
<p>For the most part this is still what happens on nearly every site you visit on a small screen. Web developers know, as the developers building the Globe site knew, that this is a problem, but solving it is not as easy as it seems at first glance.</p>
<p>In fact solving this problem would require adding a brand new element to HTML.</p>
<h2 id="introducing-the-picture-element">Introducing the Picture Element</h2>
<p>The Picture element story begins with the developers working on the Boston Globe, including Mat Marquis, who would eventually co-author the HTML specification.</p>
<p>In the beginning though, no one working on the Globe site was thinking about creating new HTML elements. Marquis and the other developers just wanted to build a site that loaded faster on mobile devices.</p>
<p>As Marquis explains, they thought they had a solution. "We started with an image for mobile and then selectively enhanced it up from there. It was a hack using cookies and JavaScript. It worked up until about a week before the site launched."</p>
<p>Around this time both Firefox and Chrome were updating their prefetching capabilities and the new image prefetching tools broke the method used on the Globe prototypes.</p>
<p>Browser prefetching was more than just a problem for the solution originally planed for the Globe site. It's actually the crux of what's so difficult about responsive images.</p>
<p>When a server sends a page to your browser the browser first downloads all the HTML on the page and then parses it. Or at least that's what used to happen. Modern web browsers attempt to speed up page load times by downloading images <em>before</em> parsing the page's body. The browser starts downloading the image long before it knows where that image will be in the page layout or how big it will need to be.</p>
<p>This is simultaneously a very good thing -- it means images load faster -- and a very tricky thing -- it means using JavaScript to manipulate images can actually slow down your page even when your JavaScript is trying to load smaller images (because you end up fighting the prefetcher and downloading two images).</p>
<p>Marquis and the rest of the developers working on the site had to scrap their original plan and go back to the drawing board. "We started trying to hash out some solution that we could use going forward... but nothing really materialized." However, they started <a href="http://blog.cloudfour.com/responsive-imgs/">writing about the problem</a> and other developers joined the conversation. The quickly learned they were not alone in struggling with responsive images.</p>
<p>"By this time," Marquis says, "we have 10 or 15 developers and nobody has come up with anything."</p>
<p>The Globe site ended up launched with no solution -- mobile devices were stuck downloading huge images.</p>
[image="pic-globe-responsive.png" caption="The Boston Globe website at phone, tablet and desktop sizes"]
<p>Soon other prominent developers outside the Globe project started to weigh in with possible solutions, including Google's Paul Irish and Opera's Bruce Lawson. Still, no one was able to craft a solution that covered <a href="http://usecases.responsiveimages.org/">all the possible use cases</a> developers had identified.</p>
<p>"We soon realized," says Marquis, "that, even if we were able to solve this with a clever bit of JavaScript we would be working around browser-level optimizations rather than working with them." In other words, using JavaScript meant fighting the browser's built-in image prefetching.</p>
<p>Talk then moved to lower-level solutions, including a new HTML element that might somehow get around the image prefetching problems in a way that JavaScript never would. It was Bruce Lawson of Opera who first suggested that a new <code><picture></code> element might be in order. Though they did not know it at the time, a picture element had been proposed once before, but it never went anywhere.</p>
<h2 id="welcome-to-standards-jungle">Welcome to Standards Jungle</h2>
<p>It is one thing to decide a new HTML element is needed. It's quite another thing to actually navigate the stratified, labyrinthine world of web standards. Especially if no one on your team has ever done such a thing.</p>
<p>Perhaps the best thing about being naive though is that you tend to plow forward without the hesitation that attends someone who <em>knows</em> how difficult the road ahead it will be.</p>
<p>And so the developers working on the picture element took their ideas to the WHATWG, one of two groups that oversee the development of HTML. The WHATWG is made up primarily of browser vendors, which makes it a good place to gauge how likely it is that browsers will actually ship your ideas.</p>
<p>To paraphrase Tolstoy, every standards body is unhappy in its own way, but, as Marquis was about to learn, the WHATWG is perhaps most unhappy when people outside it make suggestions about what it ought to do. Suffice to say that Marquis and the rest of the developers involved did not get the WHATWG interested in a new HTML element.</p>
<p>Right around this time the W3C, which is where the second group that oversees HTML, the HTML WG, is based, launched a new idea -- community groups. Community groups are the W3C's attempt to get outsiders involved in the standards process, a place to propose problems and work on solutions.</p>
<p>After being shot down by the WHATWG, someone suggested that the developers start a community group and the <a href="http://responsiveimages.org/">Responsive Images Community Group</a> (RICG) was born.</p>
[image="pic-ricg-site.png" caption="The Responsive Images Community Group website."]
<p>The only problem with community groups is that no one in the actual working groups pays any attention to community groups. Or at least they didn't in 2011.</p>
<p>Blissfully unaware of this, Marquis and hundreds of other developers hashed out a responsive image solution in the community group.</p>
<p>Much of that effort was thanks to Marcos Caceres, now at Mozilla, who, unlike the rest of the group members, had some experience with writing web standards. That experience allowed Caceres to span the divide between two worlds -- web development and standards development. Caceres organized the RICG's efforts and helped the group produce the kind of use cases and tests that standards bodies are looking for. As Marquis puts it, "Marcos saw us flailing around in IRC and helped get everything organized."</p>
<p>"I tried to herd all the cats," Caceres jokes. And herd he did. He set up the Github repos to get everything in one place, set up a space for the responsive images site and helped bring everything together into the first use cases document. "This played a really critical role for me and for the community," says Caceres, "because it forced us to articulate what the actual problem was... and to set priorities."</p>
<p>After months of effort, the RICG brought their ideas to the WHATWG IRC. This also did not go well. As Caceres puts it, "standards bodies like to say 'oh we want a lot of input for developers', but then when developers come it ends in tears. Or it used to."</p>
<p>If you read the WHATWG IRC logs from that time you'll see that the WHATWG members fall into a classic "not invented here" trap. Not only did they reject the input from developers, they turned around and, without considering the RICG's work at all, <a href="http://www.w3.org/community/respimg/2012/05/11/respimg-proposal/">proposed their own solution</a>, something called <code>set</code>, an attribute that solved only one of the many use cases Marquis and company had already identified.</p>
<p>Developers were, understandably, miffed.</p>
<p>With developers pushing Picture and browser makers and standards bodies favoring the far more limited and very confusing (albeit still useful) <code>set</code> proposal, it looked like nothing would ever actually come of the RICG's work.</p>
<p>As Paul Irish put it in the <a href="http://krijnhoetmer.nl/irc-logs/whatwg/20120510#l-747">WHATWG IRC channel</a>, "[Marquis] corralled and led a group of the best mobile web developers, created a CG, isolated a solution (from many), fought for and won consensus within the group, wrote a draft spec and proposed it. Basically he's done the thing standards folks really want "authors" to do. Which is why this this feels so defeating."</p>
<p>Irish was not alone. The developer outcry surrounding the WHATWG's counter proposal was quite vocal, vocal enough that some entirely new proposals surfaced, but browser makers failed to agree on anything. Mozilla killed the WHATWG's idea of <code>set</code> on <code>img</code>. And Chrome refused to implement Picture as it was defined at the time.</p>
<p>If this all sounds like a bad soap opera, well, it was. This process is, believe it or not, how the web you're using right now gets made.</p>
<h2 id="invented-here.">Invented Here.</h2>
<p>To the credit of the WHATWG, the group did eventually overcome their not-invented-here syndrome. Or at least partially overcame it.</p>
<p>Compromises started to happen. The RICG rolled support for many of the ideas in<code>set</code> into their proposal. That wasn't enough to convince the WHATWG, but it got some members working together with the Marquis and the RICG. The WHATWG still didn't like Picture, but they didn't outright reject it anymore either.</p>
<p>To an outsider the revision process looks a bit like a game of Ping Pong, except that every time someone hits the ball it changes shape.</p>
<p>The big breakthrough for Picture came from Opera's Simon Pieters and Apple's Tab Atkins. They made a simple, but powerful, suggestion -- make picture a wrapper for <code>img</code>. That way there would not be two separate elements for images on the web (which was rightly considered confusing), but there would still be a new way to control which image the browser displays.</p>
<p>This is exactly the approach used in the final version of the Picture spec.</p>
<p>When the browser encounters a Picture element, it first evaluates any rules that the web developer might specify. Opera's developer site has a good article on <a href="http://dev.opera.com/articles/native-responsive-images/">all the possibilities Picture offers</a>. Then, after evaluating the various rules, the browser picks the best image based on its own criteria. This is another nice feature since the browser's criteria can include your settings. For example, future browsers might offer an option to stop high-res images from loading over 3G, regardless of what any Picture element on the page might say. Once the browser knows which image is the best choice it actually loads and displays that image in a good old <code>img</code> element.</p>
<p>This solves two big problems -- the browser prefetching problem -- prefetching still works and there's no performance penalty -- and the problem of what to do when the browser doesn't understand picture -- it falls back to whatever is in the <code>img</code> tag.</p>
<p>So, in the final proposal, what happens is Picture wraps an <code>img</code> tag and if the browser is too old to know what to make of a <code><picture></code> element then it loads the fallback <code>img</code> tag. All the accessibility benefits remain since the alt attribute is still on the <code>img</code> element.</p>
<p>Everyone is happy and the web wins.</p>
<h2 id="nice-theory-but-show-me-the-browser">Nice Theory, but Show Me the Browser</h2>
<p>The web only wins if browsers actually support a proposed standard. And at this time last year no browser on the web actually supported Picture.</p>
<p>While Firefox and Chrome had both committed to supporting it, it might be years before it became a priority for either, making Picture little more than a nice theory.</p>
<p>Enter Yoav Weiss, a rare developer who spans the worlds of web development and C++ development. Weiss was a independent contractor who wanted Picture to become a part of the web. Weiss knew C++, the language most browsers are written in, but had never worked on a web browser before.</p>
<p>Still, like Caceres, Weiss was able to bridge a gap, in this case the world of web developers and C++ developers, putting him in a unique position to be able to know what Picture needed to do and how to make it happen. So, after talking it over with other Chromium developers, Weiss started hacking on Blink, the rendering engine that powers Google's Chrome browser.</p>
<p>Implementing Picture was no small task. "Getting Picture into Blink required some infrastructure that wasn't there," says Weiss. "I had two options: either wait for the infrastructure to happen naturally over the course of the next two years, or make it happen myself."</p>
<p>Weiss, who, incidentally, has three young children and, presumably, not much in the way of free time, quickly realized that working night and weekends wasn't going to cut it. Weiss need to turn his work on Picture into a contract job. So he, Marquis and others involved in the community group, set up a <a href="https://www.indiegogo.com/projects/picture-element-implementation-in-blink">crowd funding campaign on Indiegogo</a>.</p>
<p>On the face of it it sounds like a doomed proposition -- why would developers fund a feature that will ultimately end up in a web browser they otherwise have no control over?</p>
<p>Then something amazing happened. The campaign didn't just meet its goal, it went way over it. Web developers wanted Picture bad enough to spend their money on the cause.</p>
<p>It could have been the t-shirts. It could have been the novelty of it. Or it could have been that web developers saw how important a solution to the image problem was in a way that the browser makers and standards bodies didn't. Most likely it was some combination of all these and more.</p>
<p>In the end enough money was raised to not only implement Picture in Blink, but to also port Weiss' work back to WebKit so WebKit browsers (including Apple's iOS version of Safari) can use it as well. At the same time Marcos Caceres started work at Mozilla and has helped drive Firefox's support for Picture.</p>
<p>As of today the Picture element will be available in Chrome and Firefox by the end of the year. It's available now in Chrome's dev channel and Firefox 34+ (in Firefox you'll need enable it in <code>about:config</code>). Here's a test page showing the new <a href="https://longhandpixels.net/blog/2014/02/complete-guide-picture-element">Picture element in action</a>.</p>
<p>Apple appears to be adding support to Safari though the backport to WebKit wasn't finished in time for the upcoming Safari 8. Microsoft has likewise been supportive and is considering Picture for the next release of IE.</p>
<h2 id="the-future-of-the-web">The Future of the Web</h2>
<p>The story of the Picture element isn't just an interesting tale of web developers working together to make the web a better place. It's also a glimpse at the future of the web. The separation between between those who build the web and those who create web standards is disappearing. The W3C's community groups are growing and sites like <a href="http://movethewebforward.org/">Move the Web Forward</a> aim to help bridge the gap between developer ideas and standards bodies.</p>
<p>There's even a site devoted to what it calls "<a href="http://specifiction.org/">specifiction</a>" -- giving web developers a place to suggest tools they need, discuss possible solutions and then find the relevant W3C working group to make it happen.</p>
<p>Picture may be almost finished, but the RICG isn't going away. In fact it's renaming itself and taking on a new project -- <a href="http://responsiveimagescg.github.io/eq-usecases/">Element Queries</a>. Coming soon to a browser near you.</p>
|
