Installable web apps and interoperability

One (of many) super outcomes of the HTML5 evolution is that browser vendors are striving towards interoperability – even Microsoft is talking of “same markup”.

However, the dedication to interoperability hasn’t spilled over into the world of installable browser applications. Installable apps (in W3C parlance, Widgets – which is a terrible name) allow authors to write apps using HTML(5), CSS, JavaScript, SVG etc, and package them up into a glorified Zip file with some configuration details which can then be installed on a computer. They then use the browser’s rendering engine, but don’t show browser chrome. To all intents and purposes, they’re indistinguishable from native apps.

The W3C defines them as

full-fledged client-side applications that are authored using technologies such as HTML and then packaged for distribution. Examples range from simple clocks, stock tickers, news casters, games and weather forecasters, to complex applications that pull data from multiple sources to be “mashed-up” and presented to a user in some interesting and useful way.

Opera (disclosure: my employer) uses the W3C standard for its packaging format – so the assets are bagged up in a zip file, which is renamed .wgt The configuration information is an XML file that specifies things like required features (eg, camera access), default size of the Widget (full-screen, minimised etc) and its network access permissions. (Learn more or download some widgets and have a play).

W3C Widgets work in Opera browsers (and we re-use the format for Opera Extensions), Apache Wookie, Nokia Series 40 phones [PDF], some Vodafone phones, Windows Mobile 6.5 (although in Win Phone 7 you have to use Silverlight, yay Same markup!) and other places, such as forthcoming interactive classroom whiteboards and TVs.

Chrome has a widget format called .crx that is so similar to W3C Widgets that Scott Wilson wrote a Chrome apps to W3C Widget converter, saying

I was intrigued to find out what sort of formats the Chrome Web Store was using for its “installed web applications”, and discovered somewhat to my surprise it uses yet another packaging and manifest format! This is in addition to the two manifest formats Google already uses (for Gadgets/OpenSocial/Wave and Google App Market).

The format uses the file extension “.crx” and consists of a 7-Zip archive and a manifest file in JSON. The actual contents however are almost identical to W3C Widgets, and so I created a proof-of-concept set of conversion code.

My question to chums in Chrome and Mozilla (which has its own Mozilla Install Manifest format): what’s missing from the standard W3C Widgets specification that you need? Wouldn’t it be better for us all to collaborate on one interoperable specification, as we’re doing so successfully with HTML5?

My chum Marcos Caceres is the editor of the Widgets spec and would doubtless leuurve your feedback.

Update 24 July 2012. I never got an answer from Chrome, but they have announced their own non-standards-based proprietary Packaged Apps format. Hurray for interoperability.

38 Responses to “ Installable web apps and interoperability ”

Comment by jpvincent

that would be nice and if possible it would be nice to include other people that go to the Web languages make their app. I’m thinking to Microsoft who seems to say one can develop applications in the next windows, or to television makers like samsung (I’m working on one those days), who has its own format of zip and .xml file.

Comment by Mo

You know, so badly named is the W3C’s Widgets spec that I assumed that it wasn’t at all the same thing as Chrome’s “apps”, and things of its ilk and so it got shuffled well into the depths of my “pay attention to, one day” pile.

I mean, I can sort of understand the rationale: for a while widgets and gadgets were the hot thing, and there’s a fair amount of commonality between what we think of as “widgets” and what we think of as HTML-based “apps”, but really, if you were to ask somebody what the spec would be called which describes the workings of installable applications built using Web technologies, “Widgets” would be pretty low down the list.

Comment by AFBarstow

I would also like to understand why the W3C’s web application packaging format (“widgets”) was rejected by others. Was it primarily an XML versus JSON issue; was it mainly about NIH Syndrome; something else? If people have some feedback on this, please consider sending it to “public-webapps@w3.org”.

(BTW, I agree the title of the spec isn’t particularly descriptive and I take some blame for that. Bruce hints at Web Application Installation Format (WAIF) which is certainly more descriptive but “waif” itself is perhaps too disparaging. The group originally used Web Application Packaging Format but succumbed to marketing pressure to use “widget”.)

Comment by Trygve Lie

Great initiative and I’ve been confused about this for a very long time.

One interesting actor using the W3C Widget spec which you do not mention are PhoneGap. In their build service you can upload W3C widgets and it will spit out browser wrapped applications. Same goes for apparat.io. A very interesting approach to the web vs native mobile application mambo jambo.

Tough: the difference in this are more fragmented than being FireFox and Chrome going solo.

Desktop widgets on Mac can also be build the same way as W3C Widgets, but they does have another structure on their config and the zip file have another extension name. A good intro to Mac desktop widgets can be found here.

Even more fragmented does it become when we look at Windows. Windows also have a desktop widget system which are written in the same way as W3C widgets but with their own config and extension on the zip package. Microsofts documentation on this are here.

In other word; HTML, CSS and JS as packaged applications are not a new idea and yes I would love to see this standarized. It’s really sad to see both Chrome and FireFox not taking part in the W3C standard but going their own way with a non standard format.

And yes: I totally agree that the name “Widgets” are horrible!

Comment by Matt Wilcox

While I have little to say on this matter specifically as I know far too little about it, I can say you should ignore marketing if that’s the kind of crap they’ve pushed you into.

WAPF is both far more descriptive and far more useful than “widget” What’s the extensions for a Web Application Package Format? .wapf What’s the extension for a Widget? .wgt? .widg? .wdgt?

Never let marketing dictate the name of something engineers and coders are going to use. We’re from different worlds, and they don’t mix well. Marketing is all fluff, and frankly widget is utterly non descriptive, and doesn’t even conjour up the idea of an application or web technology. Horrid, possibly harmful, naming.

Comment by Matt Wilcox

To explain it to the Marketing people:

This is a web/os technology that is aimed at coders. To gain traction and recognition with coders it must be descriptive and obvious. Not vague, fluffy, or “recognised by a more generic group of people”. End user’s “name recognition” is irrelevant.

Comment by Doug Schepers

Regarding the putrescent name… I believe it comes from Opera! Robin Berjon and I tried to convince the original editor, Anne van Kesteren, to call it something else (we suggested WRAP: Web Resource Application Packaging), and I’ve since come up with many clever names (how does Local Applications for Web Standards, Offline and Networked, or LAWSON, strike you?), but a rose by any other name…

What is marginally more important than the name is widespread adoption and deployment. I totally agree that this is a real missed opportunity. There is a fundamental set of overlapping use cases for all of the browser vendors to work with, and good market reason for most of them to do so.

Certainly, mobile application developers and Web developers want this, and I’ve talked with big-name content creation companies, social networking sites, and mobile carriers that really want this to happen, so people can use a single universal app format for mobiles, tablets, desktops and laptops, and televisions. It seems that the only ones who haven’t wanted this to happen so far are the browser vendors and those invested in their own proprietary installable application market.

There have been some mistakes and missteps along the way, with Widgets, and some bruises. It may be that the whole Installable WebApps effort needs to be rebooted, learning what we can from Widgets, but getting as many of the right stakeholders at the table as possible to get quick consensus on the necessary features and functionality, with no preconceptions but with a predisposition to existing running code (and of course, with a rebranding). Speaking for myself, I would welcome any suggestion that will help get everyone on the same page.

Comment by Scott Wilson

@Doug I don’t think the spec needs “rebooting” – its done. Maybe just some marketing push, maybe even renaming. But there isn’t anything wrong with the actual spec functionality.

Comment by Bruce

Doug said “marginally more important than the name is widespread adoption and deployment”. I’d go a step further and say that it’s “quite a bit more important”.

I’ve pinged a couple of people in the Firefox and Chrome teams to see what their colleagues say to the questions “what’s missing from the standard W3C Widgets specification that you need? Wouldn’t it be better for us all to collaborate on one interoperable specification, as we’re doing so successfully with HTML5?”

Comment by John Allsopp

Bruce,

not sure if you and I have discussed this the last couple of years over sushi, a barbie, or bangers and mash, but I couldn’t agree more.

Keep in mind too other widget platforms like Yahoo Widgets (there are plenty others out there), all more or less identical, except for the packaging formats. It’s ridonkulous.

A huge advantage of web over specific platforms is of course interoperability. This is different an area where things are rather missing.

Comment by Mark Nottingham

I’ve always been curious as to why the widgets folks conflated the manifest (i.e., what’s important to the app) with the package (the zip file containing the actual assets). Doing it this way effectively puts all of that content off the web.

It also requires some duplication of effort / overlap with appcache; these things are way too similar. I.e., it’d be a lot more sensible if the appcache manifest were extended with configuration information, rather than reinventing it as a zipfile.

I see the utility of *sometimes* having an app package format as well (e.g, for distribution on a DVD), but why enforce the relationship? The manifest should define the app.

Anyway, I suspect that this confusion / overlap is hurting both appcache and widgets.

Comment by Peter Gasston

Should we re-name W3C Widgets as “Packaged web apps” (short-hand term ‘web package’.)?

Yes.

Would it be better to work on one interoperable standard (regardless of the name)?

Of course.

Comment by Robin Berjon

As Doug says, we’d initially proposed a much better name: WRAP! If we do v.next, we can certainly rebrand!

And if we do do v.next, then adding a JSON alternative to the configuration format really shouldn’t be that big a deal. We can split packaging and configuration as Mark suggests, in fact that’s been on the shopping list for a while. This also means splitting localisation for the configuration (where it’s good) and for the content (where it isn’t so hot). The good news is that even in the face of a large restructuring of the specification, a lot of the more bothersome parts of the work (e.g. figuring out what part of Zip and UTF-8 is interoperable) have already been done. So it’s not that much work.

Comment by Marcos Caceres

Renaming was shot down. Will probably rewrite a preface Note with Scott showing how everything slots together and title the document appropriately (Packaged Web Apps). Wow! bike-shedding is fun :)

PS: Robin, WRAP rhymes with CRAP. I won’t have it.

Comment by Henri Sivonen

Disclaimer: I don’t work on the relevant Mozilla subprojects. The following is my personal view.

The link to “Mozilla Install Manifest” points to a legacy RDF/XML format. It is an unfortunate artifact of Netscape dabbling with RDF ages ago. It’s probably not a good idea to use it as an indication of where things are going.

I think the W3C widget packaging has 3 problems:

1) It uses XML. Meh. Chrome and upcoming work at Mozilla use a JSON-based manifest. The manifest case seems like something where JSON is adequate and XML is needless complexity. I see the W3C strategy tax at work.

2) W3C widget packing reinvents jar signing using XML Signature. First, what’s the point of reinventing jar signing? Mozilla uses the jar signing mechanism (albeit with different crypto algorithms) for xpi signing. Second, XML Signature is just gratuitous complexity. XML Signature is one of those W3C specs that you shouldn’t depend on if you are aiming at deployment in the browser space instead of the enterprise boondoggle space. Again, W3C strategy tax at work.

3) It takes the app off the Web and stuffs it into a zip file. It would be more Webby to use the HTML5 app cache and make “installation” mean that the app is guaranteed to stay in the app cache and that the app can use as much cache space as it wants. (Note that developing for the app cache isn’t without problems, but it’s still more Webby than HTML/CSS/JS in a zip file.)

Comment by Marcos Caceres

Henri’s claims are wrong.

1) It uses XML. Meh. Chrome and upcoming work at Mozilla use a JSON-based manifest. The manifest case seems like something where JSON is adequate and XML is needless complexity. I see the W3C strategy tax at work.

JSON and XML are almost identical to parse and process. Both are as restrictive as each other. It’s a matter of what’s trendy, not a technical issue. I have already proposed on a number of occasions to rewrite the thing in JSON (has been a proposal for years, see the requirements document) – see also the name of the current packaging spec: it deliberately says “XML Configuration” because I was ready to make the JSON fork. If Moz was to get on board, there would be enough momentum to actually make a new spec.

2) W3C widget packing reinvents jar signing using XML Signature. First, what’s the point of reinventing jar signing? Mozilla uses the jar signing mechanism (albeit with different crypto algorithms) for xpi signing. Second, XML Signature is just gratuitous complexity. XML Signature is one of those W3C specs that you shouldn’t depend on if you are aiming at deployment in the browser space instead of the enterprise boondoggle space. Again, W3C strategy tax at work.

It does not reinvent anything. It just uses XML Digital signatures as one possible format. There is no reason it cannot support an alternative format for signatures.

3) It takes the app off the Web and stuffs it into a zip file. It would be more Webby to use the HTML5 app cache and make “installation” mean that the app is guaranteed to stay in the app cache and that the app can use as much cache space as it wants. (Note that developing for the app cache isn’t without problems, but it’s still more Webby than HTML/CSS/JS in a zip file.)

I think you are confusing the use cases (or being “too Webby”, whatever that means). Sometimes you want to keep things you make. Also, it works great for developing extensions: see Opera extensions which use W3C widgets.

Comment by Marcos Caceres

@bruce, the working group did not reject anything. To this date the working group and W3C keeps begging Moz and Google to come and discuss their requirements. The W3C and I have bent over backwards to make sure we can accomodate all their requirements (make the XML format optional, do JSON, allow other signature schemes) etc. But they are only interested in their own proprietary formats.

Comment by Doug Schepers

@Henri: Let’s assume all 3 of your points are right (Marcos’ disagreement aside). Is the relative severity of all of them enough to justify dividing the market?

1) XML vs. JSON: the manifest file is, what, 20, 30 lines long? The authoring experience of the manifest file is not much different, and no more difficult, than authoring HTML. This seems a minor quibble, at worst.

2) XML Signature versus jar signing: Or really, any other kind of signing, like OpenPGP or S/MIME. I’m no expert, but from what I recall, the profile of XML Dig Sig is relatively small and, though admittedly more complex because of the canonicalization issue, still not a huge implementation burden. Still, it could be better to go with some other signing mechanism that is scoped well to the problem and has stable off-the-shelf libraries that could be used. But even if we stuck with XML DIG Sig, it would be a relatively minor (albeit irritating) problem.

3) AppCache vs. Zip (also Zip vs. 7zip): There are use cases for both dynamically cached apps and static, persistent, locally installed apps (it’s more of a continuum, really). But even if both types of apps exist in one implementation (such as in Chrome), it’s still not much extra code, or much more for Web developers to learn (or authoring tools to manage).

By contrast, with the amount of market fragmentation around installable apps (I can think of at least 12 different “installable app” runtimes from 6 major companies, most of them with significant overlap), Web developers have to jump through flaming hoops just to reach half that many platforms. That doesn’t seem very “Webby”.

In other words, though the Widgets specs as they stand may have flaws (and I agree they could stand to have some changes, especially in v2), those flaws together are vastly less harmful, even in the long term, than the effect on developers and users from fragmentation.

The perfect is the enemy of the good. I think that there is now enough collective experience around installable apps among the browser vendors that y’all could come to the table and come up with a best-of-breed solution that’s good enough to satisfy the market need.

Comment by c69

Obviously its a political and economical issue, not technical.

Widgets (or html apps) are quite weak in their capabilities, thus – they cannot compete with native apps.

Purely offline widget does not provide a way to monetize for developer / publisher – as all apps are open source and can be pirated freely. Thus, you are either making a generic crap for fun, branded clone of popular app for vanity / promotion, or a mere client for online service, which is often also accessible from a regular browser.

XML vs JSON, 7z vs gz, signing method, name discussions – are just a rights bragging among vendors. Though they will not matter until the economical issues are solved. And when they will be – developers will be just using some kind of compilers to create proper platform specific versions.

Comment by Marcos Caceres

c69, you are missing the point of W3C widgets. It’s just a generic patent-unencumbered/open packaging format for various classes of application. A good example of usage is Opera: who use it to contain apps written with web technologies, for their extensions platform, as well as Opera Unite (iirc). So, it’s not about selling anything or making just one class of application. Lots of software is built because its is useful and not sold for money (e.g., PHP, Python, your web browser)… and a lot of it is open source… and a lot of that software is packaged in a Zip file. Sure, the web and widgets are missing a lot of APIs, but if widgets start getting traction, then we have a compelling reason to add those. Please try to be more incremental and forward thinking instead of dismissing the technology because it does not do what you want right now… HTML sucks compared to native apps, but it’s incremental improvements are constantly making things better (to the point where it becomes a serious threat to platforms like Flash). W3C Widgets make use of the Web platform and evolve with it.

Comment by David Storey

(leaves my well worn employers hat at the table)

I think there are a few things here we need to think about carefully.

First of all the Widgets spec is mostly a packaging and configuration format. If I represented a mythical major software company that wanted to build a platform with installable apps based on HTML/CSS/JS, if packaging format was so important I had to make it vendor specific to create lock-in then, well I have major problems. Supporting Widgets is the easiest way to do a hat tip to the standards world, even if I’m only half serious. The value I can add to make my platform viable and of interest is always built on top of that layer, such as a beautiful easy to use UI, advanced APIs to make developers lives easier, monitisation, developer tools, cost, whatever. I can see no reason not to standardise on this layer, unless there are technical or marketing reasons not to.

On the marketing side there is the name. Widgets is confusing cause developers think of UI widgets (components, whatever) or that widgets are just little toy apps. If you are building a serious app platform, you probably don’t want the apps you want to take over the world being associated with little toy apps. You want them to be taken seriously as full blooded complex powerful applications.

The name come from Opera I guess, which in turn probably came from Dashboard Widgets and Konfabulator. It has out grown this. The easy solution is to rename. The hard part is to select a name, and the fact it breaks compatibility if you change the root element to a different element name. I would suggest for version 2 that there is enough political backlash to bite the bullet to do it. The implementation cost wont be too much. I’d suggest calling a spade a spade and calling the root element either manifest or app. As long as it isn’t widget.

The next issue is the technological one. I’m not sure what issues people have with the spec, but the likes of Mozilla and Google use JSON. I guess it is the big new trendy thing.I personally like XML but I guess we bite the bullet and support a JSON serialisation too. It at least makes it easy to manipulate with JavaScript if that is what you need. Things like RDF have multiple serialisations so it isn’t without precedence. It also reduces the reasons why vendors don’t use it.

There is talk in this thread about issues with XML Signatures. I don’t know enough about those, but we can take things one step at a time. Even if vendors adopt the manifest file but not the other parts it is still one step closer. I’m sure there is a solution for this and the gzip vs 7zip questions too. At least if we fix the major identified issues it becomes harder to justify why a supposedly open platform doesn’t support the W3C packaging format.

Comment by Mozilla Web API: Friend or foe? | Scott's Workblog

[...] However it would have been better to have made this completely clear at the outset rather than announce the WebAPI as if it were a completely green field idea and then handle the “WTF?” reaction. (A similar argument emerged in relation to Mozilla announcing what appeared to be a competing installable web application/widget format) [...]

Comment by pozycjonowanie stron trójmiasto

Excellent post. Keep writing such kind of information on your
page. Im really impressed by it.
Hello there, You have done an excellent job. I will definitely digg it and in my opinion recommend to my friends.

I am confident they’ll be benefited from this web site.