Ironick

Epiphany: Replace HATEOAS With "Hypermedia Describes Protocols"

As a few of my friends know, I live for epiphanies. I love to connect concepts. So I'm really happy to be having one now (it's been a while as regular readers of my blog -- if any remain -- can tell).

For a LONG time, I've been talking about how all interfaces can be defined in terms of IFaPs (Identifiers, Formats, and Protocols). My canonical example of an interface composed of IFaPs is of course the Web: URL (I), HTML (F), and HTTP (P). All three intersect in a particular instance of HTML, say my blog's home page. The HTML for my blog's home page is filled with URLs, HTML tags, and even HTTP "verbs" (though these are quite rare, mostly in an HTML form or embedded JavaScript).

Then along came REST and with it the concept of HATEOAS: Hypermedia As The Engine of Application State. And everyone, myself included, spent a lot of time trying to grok it and explain it to others. We're still trying. One way I try to explain it is by highlighting that HATEOAS requires that each server response must contain not only the requested data -- but also control information (in the forms of specially tagged URLs) describing the next set of permitted interactions with the server. It is this additional control information (at a bare minimum just some links to more data) that turns mere media into hypermedia.

Now along comes Jim Webber with a much better (dare I say brilliant) way of explaining HATEOAS and hypermedia: "Hypermedia Describes Protocols!" (See slide 26.) At first this might seem counterintuitive, since I said that HTTP is the Protocol and HTML is the Format in the WWW. But URLs, HTML, and HTTP are just generic description languages for describing domain-specific identifiers, formats, and protocols. Thus, think of a web of specific HTML pages as a domain-specific protocol. Jim Webber uses the example of ordering a Starbuck's coffee. (What's important is that each hypermedia DSL is composed using the generic languages of URL, HTML, and HTTP.)

This notion of bringing together identifiers, formats and verbs to describe a protocol is not new. One of the best descriptions of this was in the WS-BPEL 1.1 spec:

In thinking about the data handling aspects of business protocols it is instructive to consider the analogy with network communication protocols. Network protocols define the shape and content of the protocol envelopes that flow on the wire, and the protocol behavior they describe is driven solely by the data in these envelopes. In other words, there is a clear physical separation between protocol-relevant data and "payload" data. The separation is far less clear cut in business protocols because the protocol-relevant data tends to be embedded in other application data.

So if WS-BPEL was already thinking about mixing protocol data with "payload" data, what's so new about HATEOAS? The fundamental difference is that WS-BPEL is based on the concept of providing an entire static protocol description up front once and for all -- and providing it out of band. But HATEOAS is based on the notion of progressive description (don't bother Googling the term, I coined it; and not to be confused with progressive disclosure). More and more of the description of the protocol is provided to the client (in band in the protocol itself) as the client executes its part of the protocol. I guess another good term might be JIT Protocol Description (Just In Time). Another good term might be "self-describing protocol". So now when explaining HATEOAS, instead of saying "each server response must contain control information" (huh?), I can say "each server response progressively self-describes the current protocol."

Now there are pros and cons to static/complete vs dynamic/progressive protocol descriptions. How can I program a client to execute its part of a protocol if I don't have a full description of it up front? But if I encode the complete static description of the protocol into my client up front, how can I change the protocol dynamically?

Love to hear others' thoughts. I'm going to think about this some more. That's why I love epiphanies -- they make you think about things in new ways.

Long Live the Web

Anne Thomas Manes has started the new year of with a bang by declaring: SOA is Dead; Long Live Services.

While I agree with many of the sentiments behind Anne's declarations that "SOA is dead", I disagree with her way forward: "long live services".

It is services thinking, as conventionally understood, that led to the mess in which we find ourselves: fragmentation caused by entity-specific (service) interfaces. I'd say instead, "long live the web." I'm shocked that Anne's blog post does not even mention the web!

I agree when Anne says, "it requires redesign of the application portfolio. And it requires a massive shift in the way IT operates." But the disruptive redesign required is to make IT more Web-like -- both in the architecture of software and in the way the ITO operates.

The most "spectacular gains" we have are those of Google, Amazon, and even Salesforce. What they have in common is an embrace of the Web, including web architecture, web community, and web business models. To paraphrase Anne's blog post:

"Web-orientation is a prerequisite for rapid integration of data and business processes; it enables situational development models, such as mashups; and it's the foundational architecture for SaaS and cloud computing."

WOA is Phony? Ouch!

Judith Hurwitz apparently doesn't like WOA. That's OK. To each their own. [BTW, I choose to use the ungrammatical their/they/them as a universal 3rd party pronoun to avoid having to use gender specific his/he/him or her/she/her.] I can't be too upset because her blog post turned me on to PollDaddy.com. Here is my first poll:

Good Walk-Through of a RESTful Transactional Application

How to GET a Cup of Coffee is a very detailed discussion of a RESTful application of ordering and paying for coffee at a hypothetical Starbucks. It shows a lot of the details needed to understand the power and flexibility of the approach. It even manages to explain HATEOAS (Hypermedia As The Engine Of Application State. Nicely done.

The Customer State Machine

The Barista's State Machine

A revisionist history of 'Web Service'

Talk about revisionist history! According to this article, the "web" in "Web Services" has NOTHING to do with the World Wide Web(!):

The term web service is unfortunately a bit misleading, because many people think that it means a service specialized for use across the World Wide Web. This leads to opinions like "web services will fail because there won't be enough demand for subscriptions to Internet software services." In fact, the term is an abbrevation for Web of Services, meaning that distributed applications will be assembled from a web of software services in the same way that web sites are assembled from a web of HTML pages. If there's one thing that I hope to convince you of, it's that a majority of distributed applications will be built out of web services, regardless of whether they are deployed to a single machine, a corporate intranet, or the Internet.

Where in the world did Graham Glass come up with this "definition" of "web services"? It certainly did NOT mean "web OF services" originally. Nor did it mean the strawman he knocks down: "a service specialized for use across the WWW".

Instead, it meant a service that was based on an interface technology based on web technologies.

The problem is that web technologies were misapplied in the case of WS-* style interfaces, which is why they are generally unsuitable of use on the WWW, which is why we need the revisionist history to justify why they are still a good thing.

What were we thinking?

It just struck me, after 5+ years of analyzing the ins and outs of SOAP, how little need there has turned out to be for the SOAP binding model (i.e., binding SOAP onto various "transport" protocols). If some endpoint is going to go to all the trouble of enabling processing of URIs and XML (a prerequisite for processing the SOAP envelope), what are the chances that said endpoint would not go ahead and enable HTTP processing? The scenario of a mainframe endpoint that is able to process a SOAP envelope, but is unable to process HTTP to receive the envelope strikes me as ludicrous.

If the Internet vision of "IP over everything and everything over IP" has any validity, and I think it does, then it's a pretty safe bet that we will also see the Web vision of "HTTP over everything" (though I'm not nearly as sure of the reverse "everything over HTTP"). So its only a matter of time when we can count on HTTP support on virtually ever endpoint, from mainframe to handset. (Note that just about every appliance that need to be configured now supports HTTP, e.g., your Linksys Wireless Access Point.) Then the only real argument is about how to make HTTP more robust: XML headers inside the HTTP envelope (the SOAP approach) or evolving the HTTP envelope itself (the waka approach). With the ever slowing pace of SOAP header ratification, interop certification, and implementation, I'm getting a lot more excited about waka-like dark horse.

So who really cares that SOAP is able to be bound to MQ or IIOP or SMTP, today? Apparently, very few--since there has been virtually no progress towards standardizing any SOAP binding other than to HTTP for years. (I thought there had once been standardized SOAP binding for at least SMTP as well, but in looking back at the SOAP 1.1 and 1.2 specs, I see I was mistaken. SMTP is used as an example in the SOAP 1.2 primer, but no standard binding was ever defined.) Yes, there's lots of talk about how SOAP can be bound to different transports, and there's even a fair number of vendor implementations that support alternative bindings, but in my experience there are woefully few organizations actually doing it.

Accordingly, it seems to me that the WS-* stack could be made a lot less complex for the average developer if the SOAP and WSDL binding models were simply deprecated and replaced with simpler "native HTTP" binding (or in the case of WSDL native SOAP). (WSDL currently only standardizes binding to SOAP and HTTP.) Doing so would also do a lot towards narrowing the rift between the RESTafarians who see the Web (especially HTTP) as a good enough application architecture and the WS-* proponents who sometimes appear to believe that WS-* is a hostile overlay of the Web.

It's not just Light vs. Heavy; It's Dynamic vs. Static.

This entry is my response to an interesting thread spun out of a bet about the future of markup languages. It can be read standalone, but is probably easier to follow if you read the bet and the thread first.

I believe that we will all author in something much closer to XHTML+microformats than we author in DocBooks or WordML or TeX, because the former better enables Tim Berners-Lee's vision of the two-way web: a web whose "interactive content" can be immediately edited using the same basic tools for browsing as for editing. I think the traditional "batch style" of authoring and publishing using "compiled markup languages" is giving way to the "interactive style" of authoring and publishing using "dynamic markup languages". [While "compiled markup language" seems to be in use, there does not yet appear to be any discussion of compiled vs. dynamic markup languages. So this may be the first.]

The batch style is exemplified by the phrase "WYSIWYG": What You See Is What You Get. What has been compiled for rendering is what you get--take it or leave it. To put it more pejoratively "What You See Is What You Are Stuck With." The interactive style is exemplified by WYSIWYE ("What You See Is What You Edit"). [For about five minutes, I was pleased with myself for having of this succinctly insightful term; until I Googled it and got 200+ hits. Oh well, its still a great alternative term for Tim Berners-Lee's read write web.]

So...because I believe that eventually we will be able to edit and annotate and cut and paste everything we SEE on the web, this leads me to believe that the language used to present such information will be the language we use to author it as well. Note that we may still use fancy WYS editors for authoring. But the author's mental model of the underlying representation will be based on a dynamic markup language.

Blogs and wikis are heading in this direction. Few blog or wiki writers author in Word and then publish in XHTML. Instead, they author while thinking in much more HTML-native ways. For example, they use pidgen HTML markup languages (==header==) and viewers that can toggle between cooked (rendered) and raw views. Or go back even further when we used to author in long hand and a WP specialist would reproduce it in electronic format. Now almost all writers are expected to author in electronic format.

So for me, the interesting question is: What will tomorrow's "dynamic markup language" look like? I'm betting it's going to look a lot more like today's XHTML+microformats than today's compiled markup languages", e.g., DocBooks, WordML, Postscript, TeX. (Note: I think a similar dialectic is happening between dynamic programming languages and compiled programming languages for similar reasons, i.e., the transition to WYRIWYE--What You Run Is What You Edit.)

The other interesting question is how quickly will dynamic markup languages and the WYSIWYE interactive document lifecycle displace compiled markup languages and the WYSIWYG batch document lifecycle? Unfortunately, technical writing, which is often expected to be published and not changed much thereafter, e.g., academic journals, is probably the last form of writing to follow this trend. The biggest determinant of the pace of such change will be the change in thinking about academic collaboration. It's already changing with preprint archives like arXiv. The lines between preprint, print, and postprint are blurring because the relationships between the peer review process and the publishing process are in flux. However, given the pace of such cultural changes, I think five years is very optimistic--but still possible.

Great WS-* Case Study.

Tim Bray has done a great case study write up about WS-*. Here's the intro:

Last week at Java One, Ashesh Badani, a Sun SOA marketing person, wanted to have lunch with me to talk about WS-*. He brought along T.N. Subramaniam, Director of Technology for RouteOne, a car-loan aggregator. (Sun loves RouteOne, they're a reference customer not only for us but for SeeBeyond, which we're in the process of acquiring). Anyhow, neither Ashesh nor Ashok Mollin, a Sun guy who's been engaged at RouteOne, got a chance to say much, because T.N. and I hit it off and had a good time talking about Web Services. Which RouteOne are doing, big time and for big bucks and successfully. They are exactly the kind of people that those of us struggling in the WS-* morass ought to be looking to for lessons. This, I think, will be the first ever ongoing piece structured as an interview; with T.N.'s help, I've tried to reconstruct our conversation at lunch. I think some conclusions are obvious, but I'll leave them for you to draw.

I'd bottom line the advice from TN as "So far so good, but as you fill out WS-* KISS (instead of addressing marginal cases)."

Some say, WS-* is too complex; I say, Compared to what?

While the controversy over WS-Complexity (which seems to have been catalyzed by Tim Bray's Loyal WS-Opposition post; see these references to Tim's post; see also my collection of WS-Complexity references) seems to be dying down, I did want to add my two cents to the debate.

In all the blog postings I've seen on the subject, not one has compared the current WS-* processes for developing, debating, and adopting Web Services and their output to the IETF standards processes and their output.

Clearly, in terms of number of pages, the IETF specs far outweigh WS-* specs, yet few would argue that the IETF process of "rough consensus and working code" is a failure. Nobody complains that anyone can submit an RFC draft at the drop of a hat. How is that different from MSFT, IBM, et al announcing a proposed WS-* spec every week?

I believe that the "federated" standards processes emerging around WS-* is a worthy successor to the IETF approach. I characterize it as "federated" because WS-* related standards are designed in different standards bodies and WS-I is emerging as the "meta-standards body" that integrates and interop-certifies the standards coming out of the other standards bodies.

I characterize the WS-* federated standards processes as a "worthy successor" to the IETF, because the traditional approach to standards is a dead as the dodo. Ever since the US federal legislation enabling industry consortia in the 1990s, they have become the standards center of gravity, for better or worse. Standards setting bodies and their interactions has become the central domain of the emerging economic model of co-opetition. It just so happens that the Web and WS-* are the first major Guinea pigs of this new paradigm.

The point I'm trying to make is to suggest that the current WS-* proliferation of WS-* specs is a sign of a vibrant, decentralized innovative community. The same is true of the proliferation of XML vocabularies across the board. This is a sign that this stuff is easy to design with. Let's declare victory and move on. The wheat will be separated from the chaff when such specs are put to use in the marketplace.

Pervasive integration requires pervasive intermediation.

Jon Udell makes a great point about pervasive intermediation. His basic point is the architecture of SOAP processing model is necessary to generalize and extend the ad hoc intermediation architecture of HTTP, which has proved so essential to the web's scalability and flexibility. This is a fundamental point about SOAP intermediaries that I have been trying to drive home as well.

This year's METAmorphosis conference has the theme pervasive integration, which leads to the "obvious" conclusion that pervasive integration requires pervasive intermediation. The web did not disintermediate, it is reintermediating (refactoring intermediation).