Ironick

Epiphany: Replace HATEOAS With "Hypermedia Describes Protocols"

As a few of my friends know, I live for epiphanies. I love to connect concepts. So I'm really happy to be having one now (it's been a while as regular readers of my blog -- if any remain -- can tell).

For a LONG time, I've been talking about how all interfaces can be defined in terms of IFaPs (Identifiers, Formats, and Protocols). My canonical example of an interface composed of IFaPs is of course the Web: URL (I), HTML (F), and HTTP (P). All three intersect in a particular instance of HTML, say my blog's home page. The HTML for my blog's home page is filled with URLs, HTML tags, and even HTTP "verbs" (though these are quite rare, mostly in an HTML form or embedded JavaScript).

Then along came REST and with it the concept of HATEOAS: Hypermedia As The Engine of Application State. And everyone, myself included, spent a lot of time trying to grok it and explain it to others. We're still trying. One way I try to explain it is by highlighting that HATEOAS requires that each server response must contain not only the requested data -- but also control information (in the forms of specially tagged URLs) describing the next set of permitted interactions with the server. It is this additional control information (at a bare minimum just some links to more data) that turns mere media into hypermedia.

Now along comes Jim Webber with a much better (dare I say brilliant) way of explaining HATEOAS and hypermedia: "Hypermedia Describes Protocols!" (See slide 26.) At first this might seem counterintuitive, since I said that HTTP is the Protocol and HTML is the Format in the WWW. But URLs, HTML, and HTTP are just generic description languages for describing domain-specific identifiers, formats, and protocols. Thus, think of a web of specific HTML pages as a domain-specific protocol. Jim Webber uses the example of ordering a Starbuck's coffee. (What's important is that each hypermedia DSL is composed using the generic languages of URL, HTML, and HTTP.)

This notion of bringing together identifiers, formats and verbs to describe a protocol is not new. One of the best descriptions of this was in the WS-BPEL 1.1 spec:

In thinking about the data handling aspects of business protocols it is instructive to consider the analogy with network communication protocols. Network protocols define the shape and content of the protocol envelopes that flow on the wire, and the protocol behavior they describe is driven solely by the data in these envelopes. In other words, there is a clear physical separation between protocol-relevant data and "payload" data. The separation is far less clear cut in business protocols because the protocol-relevant data tends to be embedded in other application data.

So if WS-BPEL was already thinking about mixing protocol data with "payload" data, what's so new about HATEOAS? The fundamental difference is that WS-BPEL is based on the concept of providing an entire static protocol description up front once and for all -- and providing it out of band. But HATEOAS is based on the notion of progressive description (don't bother Googling the term, I coined it; and not to be confused with progressive disclosure). More and more of the description of the protocol is provided to the client (in band in the protocol itself) as the client executes its part of the protocol. I guess another good term might be JIT Protocol Description (Just In Time). Another good term might be "self-describing protocol". So now when explaining HATEOAS, instead of saying "each server response must contain control information" (huh?), I can say "each server response progressively self-describes the current protocol."

Now there are pros and cons to static/complete vs dynamic/progressive protocol descriptions. How can I program a client to execute its part of a protocol if I don't have a full description of it up front? But if I encode the complete static description of the protocol into my client up front, how can I change the protocol dynamically?

Love to hear others' thoughts. I'm going to think about this some more. That's why I love epiphanies -- they make you think about things in new ways.

Great set of resources on scalability and other -ility issues

Recommended: http://blog.maxindelicato.com/2009/01/17-distributed-systems-and-web-scalability-resources.html

My history of the (Internet) Robustness Principle.

In making what I thought was a small update to my copy of a quotation of the Robustness Principle, I quickly fell down the rat hole of documenting as many versions of it as I could find. Here are the fruits of my labor. Full text of the RFCs can be viewed at the RFC Sourcebook. See also my earlier post regarding the Robustness Principle.

The Internet Robustness Principle: "Be liberal in what you accept, and conservative in what you send." Attributed to Jon Postel. See the Postel Center In Memoriam web page.

According to the Postel Center, the principle originated in RFC 760 "DOD STANDARD INTERNET PROTOCOL" (1980) using different wording:

3.2. Discussion

The implementation of a protocol must be robust. Each implementation must expect to interoperate with others created by different individuals. While the goal of this specification is to be explicit about the protocol there is the possibility of differing interpretations. In general, an implementation should be conservative in its sending behavior, and liberal in its receiving behavior. That is, it should be careful to send well-formed datagrams, but should accept any datagram that it can interpret (e.g., not object to technical errors where the meaning is still clear).

This may be the first RFC containing the principle, but two prior "Internet" specs contain the principle as well: IEN 111 "Internet Protocol" (August 1979) and IEN 123 " DOD STANDARD INTERNET PROTOCOL" (December 1979). The context and wording are identical, so the text is not quoted again.

It next appears in RFC 761 "DOD STANDARD TRANSMISSION CONTROL PROTOCOL" (1980), this time with the designation of "Robustness Principle":

2.10. Robustness Principle

TCP implementations should follow a general principle of robustness: be conservative in what you do, be liberal in what you accept from others.

Notice the switch from "sending behavior" to simply "do" and the switch from "receiving behavior" to "accept".

It also [next?] appears in RFC 791 "Internet Protocol" (1981), which obsoleted RFC 760:

3.2 Discussion

The implementation of a protocol must be robust. Each implementation must expect to interoperate with others created by different individuals. While the goal of this specification is to be explicit about the protocol there is the possibility of differing interpretations. In general, an implementation must be conservative in its sending behavior, and liberal in its receiving behavior. That is, it must accept any datagram that it can interpret (e.g., not object to technical errors where the meaning is still clear).

Notice the change from "should" to "must".

It also [next?] appears in RFC 793 "Transmission Control Protocol" (1981). The context and wording are identical, so the text is not quoted again.

It also [next?] appears in RFC 1122 "Requirements for Internet Hosts -- Communication Layers" (1989):

1.2.2 Robustness Principle

At every layer of the protocols, there is a general rule whose application can lead to enormous benefits in robustness and interoperability [IP:1]:

"Be liberal in what you accept, and conservative in what you send"

Software should be written to deal with every conceivable error, no matter how unlikely; sooner or later a packet will come in with that particular combination of errors and attributes, and unless the software is prepared, chaos can ensue. In general, it is best to assume that the network is filled with malevolent entities that will send in packets designed to have the worst possible effect. This assumption will lead to suitable protective design, although the most serious problems in the Internet have been caused by unenvisaged mechanisms triggered by low-probability events; mere human malice would never have taken so devious a course!

Adaptability to change must be designed into all levels of Internet host software. As a simple example, consider a protocol specification that contains an enumeration of values for a particular header field -- e.g., a type field, a port number, or an error code; this enumeration must be assumed to be incomplete. Thus, if a protocol specification defines fourpossible error codes, the software must not break when a fifth code shows up. An undefined code might be logged (see below), but it must not cause a failure.

The second part of the principle is almost as important: software on other hosts may contain deficiencies that make it unwise to exploit legal but obscure protocol features. It is unwise to stray far from the obvious and simple, lest untoward effects result elsewhere. A corollary of this is "watch out for misbehaving hosts"; host software should be prepared, not just to survive other misbehaving hosts, but also to cooperate to limit the amount of disruption such hosts can cause to the shared communication facility.

While it is clear that the quoted text is quoting Jon Postel, it is not clear whose interpretation of that text follows in the remainder of section 1.2.2. The author of the RFC is Robert (Bob) Braden. Who worked at ISI with Jon Postel and authored other RFCs with hime (e.g., RFC 1009). Note that RFC 1122 is the RFC referenced by the Postel Center In Memoriam page.

What I like about the RFC 1122 version of the Robustness Principle is that it clarifies a common misconception about the Robustness Principle, which is that it makes receiving systems vulnerable to attack. On the contrary, the principle as interpreted in RFC 1122 advises: "In gerneral, it is best to assume that the network is filled with malevolent entities that will send in packets designed to have the worst possible effect. This assumption will lead to suitable protective design..."

Notice also that this version says that it applies at "every layer of the protocols"--including the application layer.

The last [latest?] formulation of the Robustness Principle is in RFC 1958 "Architectural Principles of the Internet" (1996):

3.9 Be strict when sending and tolerant when receiving.

Implementations must follow specifications precisely when sending to the network, and tolerate faulty input from the network. When in doubt, discard faulty input silently, without returning an error message unless this is required by the specification.

I personally think this is a poor version of the principle and leads to much of the misunderstanding about it. It seems to advise a passive approach to "tolerance" and "liberal acceptance".

The Robustness Principle obviously also applies to life. Tim Berners-Lee has made this part of his religious beliefs, see: http://www.w3.org/People/Berners-Lee/UU.html and http://www.w3.org/DesignIssues/Principles.html.

Deconstructing Internet Protocols.

I thought *I* was pushing the interdisciplinary envelope pretty far, but someone else seems to be *way* out there. Alexander R. Galloway, Assistant Professor of Media Ecology at New York University (my law school alma mater), has recently written a book called Protocol, which appears to apply critical theory (a descendent of deconstruction a la Derrida) to Internet Protocols! Here is the book's blurb:

Is the Internet a vast arena of unrestricted communication and freely exchanged information or a regulated, highly structured virtual bureaucracy? In Protocol, Alexander Galloway argues that the founding principle of the Net is control, not freedom, and that the controlling power lies in the technical protocols that make network connections (and disconnections) possible. He does this by treating the computer as a textual medium that is based on a technological language, code. Code, he argues, can be subject to the same kind of cultural and literary analysis as any natural language; computer languages have their own syntax, grammar, communities, and cultures. Instead of relying on established theoretical approaches, Galloway finds a new way to write about digital media, drawing on his backgrounds in computer programming and critical theory. "Discipline-hopping is a necessity when it comes to complicated socio-technical topics like protocol," he writes in the preface.

Galloway begins by examining the types of protocols that exist, including TCP/IP, DNS, and HTML. He then looks at examples of resistance and subversion -- hackers, viruses, cyberfeminism, Internet art -- which he views as emblematic of the larger transformations now taking place within digital culture. Written for a nontechnical audience, Protocol serves as a necessary counterpoint to the wildly utopian visions of the Net that were so widespread in earlier days.

It seems somewhat related to Code and Other Laws of Cyberspace by Lawrence Lessig, which according to reviews I've read, has a thesis that plays on the dual meaning of code as computer software and code as a body of law. We'll see.

I found Alexander's home page, which has links to several essays, including an essay with the same title as the book. I liked it so much, I emailed him at NYU to see if we might collaborate on something.

Yet more network vs. edge debate.

Cross posting my comment to Radovan here:

Radovan: I am talking about *THE SAME ISSUES* being dealt with on different levels. For the SAME REASONS that transparent intermediaries exist at BOTH the Internet IP level, AND exist at the HTTP level (think WebSSO, Akamai, and web load balancers), they WILL exist at the SOAP and BUSINESS LEVEL.

Let me give you an example roughly based on a composite of several client discussions I've recently had. Global bank needs to comply with the illegal money laundering (ILM) provisions of the Patriot Act. So it needs to match every "open account" message going over its global network against a DB of "bad guys", because the ILM forbids banks from doing business (offering bank accounts) with bad guys.

So, its plan is to transparently route all "account open" messages past this new "bad guy matching service" without having to touch any of the dozens of account management services this bank has built around the globe. (To be more accurate, this bank didn't build these account management apps, they acquired them via M&A.)

So here's a perfect example of a service provider having its messages rerouted without its knowledge in order to add new functionality to the overall system without having to make changes to the provider. This is exactly how a Web Services Network should be designed.

Does this reduce the autonomy of the account management "service providers"? You bet. They no longer unilaterally decide who can open an account, because this matching intermediary now imposes its decisions on them. Big deal.

As for your FedEx/DHL example, the way this should work at a business document level is that I, as the provider, should indicate that a business report must be delivered to the recipient the next day. (Note that I have the authority, ie autonomy, to make this timing decision.) I put the document in my outbox on my desk with a routing slip attached and my corporation's mail room decides what overnight carrier we use for which destinations, given the discounting deals we've arranged.

Does this reduce my absolute autonomy? You bet. I've lost the precious right to decide which overnight delivery service I prefer. Big deal.

As for your blog "service" example, check out Feedburner . This intermediate service is exactly the kind of ["intelligence" in the Web Services Network] I am talking about. Rather than having my blog rss feed provider have to know all the different feed formats or having the ability to track feed pings, etc., I let this rss feed service intermediary, feedburner, do it for me. I tell feedburner my simple rss feed address for my blog, I put feedburner's rss feed icons on my blog home page and suddenly, every new rss subscriber is getting MY rss feed VIA feedburner. What I get (and what I really want that Radio Userland does not supply) is the ability to see who's reading my blog via rss. Pretty slick. This is exactly the kind of management in the middle you seem to be arguing against. Why?

Does this reduce the autonomy of my blog service? You bet. Now its dependent on feedburner being up and running. Big deal.

The autonomy of web service providers with regard to the web services network will be more like the "relative autonomy" of state governments with regard to the US federal government, not the "absolute autonomy" of a sovereign nation (which these days isn't really so sovereign itself, what with UN, WTO, etc.). Some decision making rights will be in the hands of the service providers (the states) and some decision rights will in the hands of the "network" (the federation). The network is the federation. Autonomy will be federated autonomy. That's why I've been writing about SOA as a federated architecture for the past several years. (For publicly available material see http://radio.weblogs.com/0126951/2004/02/18.html#a98 and http://techupdate.zdnet.com/techupdate/stories/main/Service_Oriented_Architecture_Enables_Global_ITO_print.html and http://www3.ca.com/Files/IndustryAnalystReports/SOA.pdf)

Autonomy and interdependency (or conformity) are very relative and very intertwined concepts. This subtle interplay has been endlessly discussed in various fields from philosophy to economics to sociology to physics (see the book "Six Degrees"). In sociology, this discussion goes under the title "Structure and Agency" (See http://en.wikipedia.org/wiki/Structure_and_agency and http://human.ntu.ac.uk/pgcert/structureandagency.html).

In the IT realm, the structure/agency discussion often goes under the title "network vs. endpoint" or "network vs. edge". Given how subtle the thinking and passionate the debate has been in other disciplines over the duality of structure and agency, I expect no less in SOA discussions. But lets at least learn something from these prior debates in these other disciplines and quickly move beyond the shallow debate of absolute autonomy vs. its absolute interdependency, into a deeper discussion of what aspects of autonomy should a service provider have and not have under what circumstances. Or even better, how I can dynamically shift responsibility from the edge into the network and back again based on various collaborative policy decisions.

Why the "End to End Arguments in System Design" is wrong.

(This thought just hit me so I wanted to capture it immediately.) Briefly, with more to come later, a federated architecture, e.g., an internetwork, requires not only the federation of heterogeneous and autonomous endpoints (i.e., hosts), it requires the federation of heterogeneous and autonomous "subnets." Now the Internet enabled heterogeneous link architectures, which it unified with IP transport. But according to the end to end arguments all other "unification" should done at the end points, e.g., TCP for reliable sessions, SSL for secure sessions.

But the end to end arguments directly contradict the Internet's design goal of being a "network of networks"! The Internet is not fully a "network of networks" because it does not enable network services such as security, reliability, transactionality, etc. to be federated across heterogeneous instances of such services into a unified end to end service. The end to end arguments claimed that such services were not "network services" but "application services". This merely begs the question of how to build a federated "application level" network. Web services is the answer to this question.

I'm surprised I haven't run across this argument against the end to end arguments given how compelling it is. Has anyone run across it?

I suppose this counts as the follow up I promised in an earlier entry on the End to End.

End-to-End Internet

Interesting post by Lawrence Lessig, a Stanford Law professor who focuses on IP issues. He cites Saltzer, Clark and Reed's "End-To-End Arguments In System Design" as his definition of the "Internet." Given my beliefs about how Web services breaks down the end-to-end vs in-the-net dichotomy, I'll respond when I have more time.