Overview of Message Draft 4

Just finished draft 4 of REST-* Messaging.  Please check our our discussion group if you want to talk more about it.  Here’s a list of resources and their corresponding relationships for a high level overview.  See the spec for more details.  It relies heavily on Link headers.  The current draft shows a lot of example HTTP request/responses to give you a good idea of what is going on.

Destination

A queue or a topic resource.

Relationships:

• post-message – link to POST/create a message to a queue or topic.  This can be
• post-message-once – link to POST/create a message to a queue or topic using the POST-Once-Exactlly (POE) pattern
• post-batch – POST/create a batch of messages using Atom or Multipart
• post-batch-once – POST/create a batch using the POE pattern.

Message

Every message posted creates a message resource that can be viewed for adminstration, auditing, monitoring, or usage.

Links Relationships:

• reply-to.  Destination that you can reply to
• reply-to-once.  Destination that you can reply to once using the POST-Only-Once pattern
• reply-to-batch. Destination that you can reply with a batch of messages
• reply-to-batch-once.  Batch using POE.
• next.  If pulling from a topic, messages will have this link.  This is the next message after this one.
• acknowledgement.  If pulling from a queue, this link allows you to change the state of the message to acknowleged or not.
• generator, Destination that the message came from

Topic

Has the same links as Destination with these added:

Link Relationships:

• next.  GET the next incoming message.  Clients should use the Accept-Wait header to set a blocking timeout (see later).
• last.  Last posted message to the topic
• first.  First message posted to topic, or, the first message that is still around.
• subscribers. URL that allows you to view (GET) or create/register (POST) a list of URLs to forward messages to.

Queue

Same as Destination, but these additional link relationships:

Link Relationships:

• poller.  Alias to the next available message in the queue.  Returns a message representation along with a Content-Location header pointing to the real one.  a POST is required to access this because the state of the queue is changed.
• subscribers.  Same as topic push model.

Reliable Acknowledgement

I wanted to add acknowledgement to the queue consumer pull model in REST-* Messaging.  The way it would work is that consumers do a POST on the queue’s URL.  They receive the message as well as a Link header pointing to an acknowledgement resource.  When the client consumer successfully processes the message, it posts a form parameter, acknowledge=true to the acknowledgement link.

There is a problem with this though.  The design is connectionless to honor the stateless REST principle.  So there is no specific session resource that the client consumer is interacting with.  The consumer may never acknowledge the message, so I need the server to re-enqueue the message and deliver it to a new consumer.  The problem is, what if the old consumer tries to acknowledge after the message is re-enqueued or even after it is redelivered to a different consumer?

I first thought of letting the first consumer to acknowledge win and do something like POST-Once-Exactly (POE).  The problem with this is, what if there’s a network failure and the consumer doesn’t know if the acknowledgement happened or not?  It would redeliver the message and get back a Method Not Allowed response error code.  With this code, the consumer doesn’t know if somebody else acknowledged the message or if the older request just went through. At first, I thought about in my personal blog of doing a conditional POST using ETag and If-Match.  The acknowledgement link, when performing a GET on it, would return an ETag header that the consumer must transmit with the acknowledgement POST.  If the message was re-enqueued, then the underlying ETag would change, and the conditional post would fail for the older consumer.

On my blog comments, Mike Amundsen pointed out that you can embed the etag information directly within the URL.  Then, the client does not have to be aware of the of the conditional POST requirement and can just POE to the ack link.  You also save on the extra GET request to obtain the ETag header of the resource.  The protocol would look something like this:

1. Consume a messageRequest:

POST /myqueue/consumer

Response:

HTTP/1.1 200 OK
Link: </myqueue/messages/111/acknowledgement;etag=1>; rel=acknowledgement
Content-Type: ...

...  body ...

2. Acknowledge the message.  Its just a POST back tot he URL.  Remember this is a POE link so if the consumer has to redeliver the POST, the request may return a 405, method not allowed.  Also notice that there is a matrix parameter named etag within the ack URL.  The server will use this information to determine if the correct consumer is acknowledging.
Request:

POST /myqueue/messages/111/acknowledgement;etag=1
Content-Type: application/x-www-form-urlencoded

acknowledge=true

Success Response:

HTTP/1.1 204 No Content

Response when it was updated by somebody else.

HTTP/1.1 412 Precondition Failed

POE Redelivery Response.  It was already successfully updated by the consumer.

HTTP/1.1 405 Method Not Allowed

I wonder if I even need the POE semantics with this. I guess it doesn’t matter if the request is redelivered.

No Wait in HTTP

One thing the HTTP specification does not have is a “Server Timeout” response code.  The 408 and 504 response codes are the only thing that comes close.  The idea of a “Server Timeout” code is that the server received the request, but timed out internally trying to process the request.  Another thing I think that is missing from HTTP is a way for the client to tell the server how long it is willing to wait for a request to be processed.

I’ve run into both of these scenarios with the REST-* Messaging specification when I have pulling client consumers.  For the “Server Timeout” I decided upon 202, Accepted.  It seems to fit as I can tell the client to try again at a specific URL.  As for the client requesting a wait time?  I invented a new request header:  “X-RS-Wait”.  Its value is the time in seconds that the client wants to wait for a request to be processed.  Maybe there is a standard or drafted protocol I missed in my Google search?

REST-* Messaging Draft 3

After prototyping, I’m back to writing another draft. This is a little bit more formal draft. I created a OSS project at:

http://sf.net/projects/rest-star

A draft of the PDF is at:

http://reststar-messaging.googlegroups.com/web/REST-Star-Messaging-draft-3.pdf

(You’ll have to click through an additional link that is *very* long). This draft only talks about the Message Publishing Protocol. I still have to write up the topic and queue push and pull model.  To discuss the draft please post at the REST-* Messaging Group.  I’m also looking for people to help prototype the specification.  Post to the RESTEasy developers list if you’re interested.

New Workflow/BPM Effort

I’ve put together some preliminary thoughts that have been swimming around in my head for the past few months on workflow and BPM.  Its hosted on our new REST-* work-group on the subject.  While we still have some work to do on defining some of the initial media types and going beyond simple use cases, the design is very link driven.  I’ve asked Tom Baeyerns and the jBPM team to help flush out the most common use cases users deal with.  I also need a REST heavyweight lurking around to make sure we’re not introducing unRESTful designs.  Anyways, Enjoy!

We screwed up with first submissions

We really screwed up with the initial drafts we put up for the initial REST-* specifications on a number of levels.  First the specifications were written 1+ years ago by me for the messaging submission and 8+ years ago by  Mark Little.  Both of us, at the time were still trying to come to terms with what REST actually was (we still are) and how you design applications/systems/services around it.  As a result, those initial drafts, while they did follow the uniform interface principle of REST, weren’t very RESTful beyond that.  The thing is, I knew this!  I knew these initial drafts were crap.  I had planned to completely rework them and have draft 2 ready for when we announced.  So, when the press picked up Mark Little’s announcement of REST-* at JBoss World (It was actually only one bullet point on a “future directions” slide), the cat was out of the bag too early and we ended up seriously botching the announcement because we werent’ ready.

While the REST-* name itself was bound to cause friction within the hard-core REST community, when the lurkers on the rest-discuss mail list actually looked at these initial drafts, to them, their greatest fears were realized:  That what we were doing was just a re-hash of the same old architecture vendors have been rehashing over and over again for almost 20 years now (DCE, rehashed to CORBA, rehashed to WS-*) and that we were just repackaging our existing products and calling it REST to be buzzword compliant.  Even a few feared that we were trying to coopt REST itself which was totally incorrect.  We just want to figure out where middleware fits into this new model, specifically the middleware problems that Red Hat is used to solving.

So, we definitely are marketing morons, well specifically me.  I botched the announcement by not being ready.  All I can say is mae culpa.  If you’ve read my blog over the past few days you’ve seen me scurrying to catch up on writing the technical material I had always intended to write.  Hopefully you can see that we’re headed in the right direction in that we’re describing things that are much more RESTful than before.  All we can do is plug on…

Credit Cards, Transactions, and REST

One of the interesting thing about credit cards is that actions performed on your account are not operations, but instead modeled as things that are first related to your account, then that actually change the state of your account.  When you buy something at a store online or in person, you are not actually manipulating your account directly.  A credit card purchase is converted into a thing:  specifically a Debit.  A refund of a purchase is converted into a thing called a Credit.  This credit or debit is logged at Visa or Mastercard.  Once a week (i forget the frequency), a company like Capital One (I used to work for them in the mid-90s) receives a huge magnetic tap roll with all the transactions that are applicable to Capital One’s accounts.  Capital One then pulls these individual credits and debits and matches them with a specific credit card account.  Finally it processes the debits and credits for a account to calculate a final balance.

The key thing to take from this is that credit card transactions are not operations, but actual things.  Another similar construct is an Airline Reservation.  A reservation is very similar to a credit or debit.  It is a representation of the final thing you want to do.  Create a ticket on an airplane.

Where REST and Transactions meet

I think if we constrain ourselves to modeling transactions as things instead that are themselves resources REST and Transactions can co-exist.  So, what transaction processing becomes in RESTful terms is relating one resource to another.  For Credit Cards, transaction processing is the result of relating Debit and Credit resources with Credt Card Account resources.

I think the wrong way to merge REST and Transactions is the way the O’Reilly RESTful Web Services book suggests.  In the book, the authors talk about manipulating an account balance by creating a parallel resource of an account as a subresource of a transaction object:

/transactions/111/account/222

The transaction sub resource is manipulated with representations of the account.  At “commit” time, the “real” account is updated with a new representation of the account.  The funny thing is, as I talked about earlier, the real world doesn’t work in this manner.  This idea that a RESTful Transaction must work in the same way you interact (with JDBC or something) with your database and that uncommitted state modifications to a resource must be viewable just isn’t true and not really feasible, and it is just not RESTful.

Where do Transaction Managers fit in?

A Transaction Manager is a generic service whose sole purpose is to ensure that a distributed system is in a consistent state.  It does this by coordinating actions between various things on the network.  I think if we model actions as things and model transaction processing as the action of relating one or more resource to one or more other resources, Transactions, Transaction Managers, and REST can all fit together.  To illustrate this let’s model a Travel Agent transaction where the Travel Agent is reserving a flight and a room within a hotel.   While there are many different transactional protocols, we’ll model the protocol that RESTafarians deem the most-unRESTful, 2-Phase-Commit.

The resources in our example will be: Reservation, Ticket, Room, Transaction-State, Transaction, and Transaction Manager.  Let’s walk through on how these resources are created.  The end product of these interactions will be a Ticket and  Room.

1. Travel Agent creates a transaction

The Travel Agent interacts withte Transaction Manager resource to create a transaction resource.  For example:

POST /transaction-manager
Content-Type: application/tx+xml

<transaction timeout="1 minute"/>

The Transaction Manager responds with:

HTTP/1.1 200, OK
Content-Type application/tx+xml
Location: /transaction-manager/transactions/11

<transaction timeout="11" state="uncommitted">
   <link rel="participants" href="/transaction-manager/transactions/11/participants"/>
</transaction>

2. The Travel Agent makes an Airline Reservation

POST /airline/reservations
Content-Type: application/airline-rsv+xml

<reservation>
   <date>...</date>
   <flight>...</flight>
   <seat>...</seat>
   <link rel="tranasaction" href="/transaction-manager/transactions/11" type="application/tx+xml"/>
</reservation>

Here the client is creating the reservation.  Notice that the reservation created is related to a transaction.  The server responds with:

HTTP/1.1 200, OK
Location: /airline/reservations/222
Content-Type: application/airline-res+xml

<reservation>
   <date>...</date>
   <flight>...</flight>
   <seat>...</seat>
   <link rel="transaction" href="/transaction-manager/transactions/11" type="application/tx+xml"/>
   <link rel="transaction-state" href="/airline/reservations/22/transaction-state"/ type="application/tx-state+xml"/>
</reservation>

Notice that a new relationship is defined on the Reservation: “transaction-state”.  This is the state of the reservation.  Another thing to note is that this representation returned with the response of the POST would be the same representation returned if you did a GET /airline/reservations/22.

3. Register Transactional State With Transaction

IMO, it doesn’t matter who relates the Reservation’s Transactional State resource with the Transaction resource.  Both the Travel Agent and the Reservation have the links needed to perform the operation.  Whoever does this, the operation might look like this:

POST /transaction-manager/transactions/11/participants
Content-Type: uri-list

http:.../airline/reservations/22/transaction-state

Another way to do this might be to use Link headers and the HTTP LINK operation (don’t quote me on how exactly to do this!):

LINK /transaction-manager/transactions/11/participants
Link: <http:.../airline/reservations/22/transaction-state>; rel="airline-reservation"; type="application/tx-state+xml"

4. Travel Agent Creates Hotel Room Reservation

The Hotel reservation works in the same way as the Airline Reservation, so I don’t want to burden you with reading duplicate interactions

5. Client Commits the Transaction

To Commit the transaction, the client simply POSTs to the Transaction resource

POST /transaction-manager/transactions/11
Content-Type:  application/tx+xml

<transaction state="committed"/>

This would block until the Transaction Manager has coordinated all services.  Or, it could return a 202, Accepted and the client could check back later to see if the state changed or not.

6. Transaction Manager Prepares

The particpants of the transaction have been registered.  The TM initiates the “Prepare” phase of the 2pc protocol by trying to change the state of each partcipants Transaction-State resource to “prepared”, i.e.:

PUT /airline/reservations/22/transaction-state
Content-Type: application/tx-state+xml

<transactional-state state="prepared"/>

What’s happening behind the scenes?  IMO, it doesn’t really matter.  Maybe we’re locking up database resources, maybe we’re just checking to see if it is still possible to purchase a ticket for that flight.

7. Commit each participant

Now that all the participants are in prepare mode, the TM will guarantee that each participant commits by logging the vote to its transaction log.  If any body fails, ( a crash or something), including the Tranactaion Manager, the TM can recover and put things in a consistent state, or if, it can’t, notify a human that it must do something.  This is what a transaction maanger was designed for.  Reliability.  So, the TM changes the state of each particpant to COMMIT. i.e.

PUT /airline/reservations/22/transaction-state
Content-Type: application/tx-state+xml

<transactional-state state="committed"/>

8. Retrieve the Ticket and Room

In this example system, the act of committing the reservation will create a Ticket or Room behind the scenes.  So, now, when the Travel Agent does a GET on the Airline or Hotel Reservation, a new link will pop up that points to a Ticket or Hotel Resource:

GET /airline/reservations/22
Content-Type: application/airline-res+xml

<reservation>
...
   <link rel="ticket" href="/tickets/333" type="application/ticket+xml"/>
</reservation>

Conclusion

An interesting thing to note about this example is that when the interaction completes, there is a complete log of who participated within the transaction through the use of link relationships.

Finally, the point of this blog was not to promote the use of 2PC or Transaction Managers, but to show that it may be possible to model transactions RESTfully by turning actions into things and modeling transaction processing by relating one resource to another.  This is how I envision REST-*.org defining the relationship with REST and Transactions.  If you want to discuss this more, ping us on our working group.

We’re listening: Changes To Org

Message Change

• It is now an open source project.
• We will be publishing the final content on IETF as a set of RFCs.
• We’re still focusing on middleware and middleware services.

“REST-* is an open source project dedicated to bringing the architecture
of the web to traditional middleware services.”

“REST has a the potential to re-define how application developers
interact with traditional middleware services. The REST-* community
aims to re-examine which of these traditional services fits within the
REST model by defining new standards, guidelines, and specifications.
Where appropriate, any end product will be published at the IETF.”

Governance Changes

• *No more trying to be a better JCP. We’ll let the IETF RFC process govern us when we’re ready to submit something.
• An open source contributor agreement similar to what Apache, Eclipse, or JBoss has to protect users and contributors.

(FYI we already required ASL, open source processes, NO-field-of-use
restrictions, etc…)

If you have any other suggestions, let me know:

http://www.jboss.org/reststar/community/gov2.html

RESTful Interfaces for Un-RESTful Services

Many traditional middleware services do not fit into the RESTful style
of development. An example is 2PC transaction management. Still, these
services can benefit from having their distributed interface defined
RESTfully. The nomenclature will be RESTful Service vs. RESTful Interface.

• 2PC transactions would be considered a RESTful interface under REST-*.org. Meaning using it makes your stuff less-RESTful, but at least the service has a restful interface.
• Messaging, compensations, and workflow services would be considered “RESTful Services” that fit in the model.

Guidelines Section

This is where I want to talk about how existing patterns, RFC’s and such
fit in with the rest of what we’re doing. An example here could be
Security. What authentication models are good when? When should you
use OAuth and OpenID? How could something like OAuth interact with
middleware services?

Some of this stuff is already up on the website. (You may have to reload
it to see it due to cache-control policies.)

Finally, apologies for the jboss.org redirection. We don’t want is as a JBoss.org project.  It was meant to be a separate entity.  It is a problem with
our infrastructure.

New Messaging Submissions

After some feedback from numerous sources on how too raw the initial messaging submission was on REST-*.org, I finally put together a bunch of ideas that have been brewing in my head for weeks now that I hadn’t had time write up.  It still needs some work, but hopefully it gives a better idea of where we want to go.  It hopefully is a bit more restful than the previous submission was.

• RESTful Queue Submission 2
• RESTful Pub/Sub Submission 2

I’ll try to put together some new thoughts around the transaction specification next.  This API needs to become more RESTful by defining link relationships.  It currently is to RPCish and relies too heavily on URL patterns.

REST-* Launched

I’m pleased to announce the launch of REST-*.org (http://rest-star.org).  REST-*.org is dedicated to bringing the architecture of the Web to traditional middleware services.  While REST has gained huge momentum in the SOA community, there hasn’t been a lot of standardization of traditional middleware services. The REST-* community aims to introduce new REST-based standards for these traditional services where none exist and provide well-defined guidelines where protocols do exist.

Architecturally we aim to be lightweight and pragmatic with a heavy emphasis on avoiding envelope formats and leveraging the full capabilities of the HTTP protocol.  From a community point of view, we have a totally open process and open source IP.  Anybody can contribute and join the fun.

We’re starting off with two specifications: Transactions and Messaging.  We’ll be expanding it to other areas as the community permits.

Mark Little defined an initial design of RESTful transactions 8 years ago at HP of both 2pc, ACID model as well as a forward-compensation engine.  Michael Musgrove recently used RESTEasy to implement this for the JBoss Transaction Manager.  Some consider REST + Transactions an oxymoron but there are some systems that may require a two-phase-commit protocol.  Compensating Transactions though fit quite nicely in the RESTful paradigm as in the end a TM is really just a coordination service.

For messaging we’re building off of the ideas of both JBoss and others in the open source community.  While Atom has become a popular mechanism for pub/sub in the REST community, it is dependent on an envelope format and was really meant for publishing blogs.  We feel that something simpler is needed to satisify the variety of data formats exchanged between intra and inter business machine-based applications.

Both of these specifications are very raw right now.  We feel that launching with something that is incomplete will allow us to build a set of contributors that have a vested interest in the specifications.

Why are we doing this?

The thing is SOAP has failed as an interoperability protocol. It is too complex to implement both from a vendor point of view and from an application point of view.  Yet, the REST world still lacks many of the core services that enterprise developers have grown accustomed to (rightly or wrongly).  REST-*.org will attempt to fill in the gaps.

Is this vendor driven?

Yes, Red Hat is a vendor.  Yes, we want big vendors to get involved as this will help out with adoption.  But all REST-* specifications are and will be run using open source processes. This means anybody can participate.  Yes, the organization will have to have some structure.

How will we avoid the mistakes of SOAP and WS-*?

Blind idealism.  Red Hat has a good track record for developing software that is community and user driven and then bringing these efforts to standards bodies (Hibernate and Seam are great examples of this).  In this case though, we will be the ones jumpstarting and founding the standards body itself.  We are battle tested in specifications efforts at the JCP and other bodies.  We’ve often been frustrated by the closed and inflexible attributes of these organizations.  We feel our open source roots and ideals make us an excellent candidate to drive and host standardization efforts.