I/O

The Dojo project is working to build a modern, capable, "webish", and easy to use DHTML toolkit. Part of that effort includes smoothing out many of the sharp edges of the DHTML programming and user experience. On the back of such high-profile success stories such as Oddpost, Google Maps, and Google Suggest, the XMLHTTP object has been getting a lot of attention of late. Sadly, in spite of all the coverage, developers have been on their own when it comes down to solving the usability problems that come along for the ride.

Cross Domain XMLHttpRequest using an IFrame Proxy

Note: The code for this feature is available in Dojo 0.4 and later. IE 7 Support in Dojo 0.4.1 and later.

Background

The browser security model does not allow using XMLHttpRequest (XHR) from one web page domain to contact an URL on another domain. However, there are cases when it would be nice to do cross domain XHR requests. There is a proposal in the W3C's Web API group to address this need (see this Mozilla tracking bug, and the bug comments for a link to the proposal).



As with most standards, it will take a while for this proper solution to saturate the marketplace. In the meantime, to get something like cross domain XHR requests today, there are the following options:

  • Set up a proxy server on the web page domain and have it forward the requests to the real XHR endpoint (requires server infrastructure).
  • Use Flash (user has to have Flash installed).
  • Use script tags (can do cross domain requests but return type must be JavaScript/JSON, and a callback mechanism needs to be established).
Another way to allow cross domain requests is to use the technique that is now available via dojo.io.XhrIframeProxy: use iframes that communicate with each other by changing URL fragment identifiers. This has the benefit of being just plain HTML and JavaScript (no additional server infrastructure or Flash), and it should be able to accommodate any asynchronous XHR request. It has been tested and works in IE 6.0, Firefox 1.5, Safari 2.0.3, and Opera 9.



It also contains a security mechanism that API providers can use to restrict the allowed cross-domain requests.

IFrames, Fragment Identifiers and XHR Proxying

Fragment Identifiers are the part of an URL that comes after the # sign:



http://www.a.com/path/to/file.html#fragmentIdentifier



A document in an IFrame can change the fragment identifier on its parent document (the document containing the IFrame). Changing the fragment identifier does not cause the page to reload. Similarly, the parent document can change an IFrame's fragment identifier without causing page reloads. Since the pages don't reload, state can be maintained inside the page.



To communicate between two cross domain documents :

  • A document (the Client document) defines an IFrame that loads the other document (the Server document).
  • Define a protocol to pass information through fragment identifiers.
  • Tell each document about the URL for the other document (so they can set the fragment identifiers correctly -- the browser needs a complete URL when setting a cross domain location).
  • Use a JavaScript timer to check for changes in the fragment identifiers.
To send an XHR request to another domain:

  • Define a JavaScript object that implements the XHR interface (a Facade).
  • Use that object instead of an actual XHR object.
  • For the Facade's send() method, serialize the request headers, method, URL and data.
  • The browser places a limit on the size of a document's URL, so the Client document breaks this serialized data into a set of fragment identifiers that will fit under the URL limit.
  • The Client document sends each fragment identifier to the Server document. The Server document sends an acknowledgement back to the Client, and the Client sends the next fragment identifier, until all are sent.
  • The Server document assembles the fragment identifier parts into the original serialized data, unpacks it into an object, then uses a real XHR object (now on the Server's domain) to do the final API service call.
  • The Server document then serializes the XHR response, and sends it back to the Client using fragment identifier segments.
  • The Client unpacks the serialized response, and sets the appropriate values on the XHR Facade.

Trade-Offs

Pros

  • 100% pure browser. No Flash or additional server infrastructure.
  • It can be dropped in fairly transparently to code that is already using XHR.
Cons
  • The technique uses IFrames and loads documents into the IFrames, so it takes more browser memory than native XHR. It would be interesting to compare the resource requirements with the amount needed to run Flash.
  • More network traffic to download xip_client.html and xip_server.html (the contents of the IFrames). However, you can configure your web server to tell the browser to cache these files for a very long time.
  • Timers are involved, with message serialization and deserialization.
  • Setting all of those URLs in the IFrames causes MSIE to make lots of those "clicking" sounds (the sound normally to indicate to the user they clicked on a link).

Security Considerations

This approach does not allow cross domain access to any XHR-enabled API service. For it to work, the API service must place the Server document (web page) on its server. That web page is given the Client URL and the XHR request in serialized form, so it can restrict who can contact the service and what types of requests are allowed. Note that all request validation happens inside the Server document's JavaScript.



You should not experiment with this technique unless you are very restrictive on the clients and API URLs that are allowed. Placing the Server document on your web server means opening up the allowed URLs to the world.

Dojo Implementation/Examples

As of 7/31/2006, the Dojo tree has support for XHR IFrame Proxying. The relevant files are:

  • src/io/XhrIframeProxy.js: the Dojo package, dojo.io.XhrIframeProxy, that provides the XHR Facade and manages the use of xip_client.html.
  • src/io/xip_client.html: the Client document. Used internally by dojo.io.XhrIframeProxy.
  • src/io/xip_server.html: the Server document. Used by API service providers to enable cross domain XHR requests.
  • tests/io/iframeproxy: test files.
The test files are running here if you want to try it out (note that the API server for these tests is not a powerful box, so it may seem slower than usual to get the responses).

For web page developers

In addition to doing the normal things for dojo.io.bind(), do the following:

  • To enable src/io/xip_client.html, find the commented out script tag under the <!-- Security protection: uncomment the script tag to enable. --> comment and remove the comments from that opening script tag.
  • dojo.require("dojo.io.XhrIframeProxy");
  • Define an iframeProxyUrl parameter to dojo.io.bind(). This will be an URL to the xip_server.html file on the API service server.
  • Only asynchronous XHR requests are supported.
Example code snippet:



dojo.require("dojo.io.*");

dojo.require("dojo.io.XhrIframeProxy");



dojo.io.bind({

iframeProxyUrl: "http://some.domain.com/path/to/xip_server.html",

url: "http:/some.domain.com/path/to/api",

load: function(type, data, evt, kwArgs){

/* do stuff with the result here */

}

});


For API service providers

API service providers will not care about src/io/XhrIframeProxy.js or xip_client.hml. They will be most interested in xip_server.html. For security reasons, xip_server.html will not run "out of the box". The following function needs to be defined:



function isAllowedRequest(request){

/* Decide if you want to allow the request. Return true or false */

}



By default, it is expecting this to be declared in an isAllowed.js file in the same directory as xip_server.html. See the comments in xip_server.html for more information.

In addition to defining the isAllowedRequest() function, the script in xip_server.html needs to be enabled. To enable xip_server.html, find the commented out script tag under the <!-- Security protection: uncomment the script tag to enable. --> comment and remove the comments from that opening script tag.

Reusable Parts for Non-Dojo Implementations

  • src/io/XhrIframeProxy.js: Provides the XHR Facade and manages the use of xip_client.html. It does not have all XHR methods defined, only the ones needed by Dojo's usage of XHR. You can look at the package code to see how it manages the Facade objects and the interaction with xip_client.html.
  • src/io/xip_client.html: Does not depend on any Dojo files, but it makes a call to a Dojo function when it receives a response from the Server document. Just replace the function call to your own function. Used internally by XhrIframeProxy.js.
  • src/io/xip_server.html: Does not depend on any Dojo files. Used for the final XHR request to the API service.

Introduction to I/O bind

The dojo.io package provides portable code for XMLHTTP and other, more complicated, transport mechanisms. Additionally, the "transports" that plug into it each provide their own logic to make each of them easier to use. The rest of this article will cover how the XMLHTTP transport from Dojo provides ways around the book-marking and back button problems.

Most of the magic of the dojo.io package is exposed through the bind() method. dojo.io.bind() is a generic asynchronous request API that wraps multiple transport layers (queues of iframes, XMLHTTP, mod_pubsub, LivePage, etc.). Dojo attempts to pick the best available transport for the request at hand, and in the provided package file, only XMLHTTP will ever be chosen since no other transports are rolled in. The API accepts a single anonymous object with known attributes of that object acting as function arguments. To make a request that returns raw text from a URL, you would call bind() like this:

dojo.io.bind({

url: "http://foo.bar.com/sampleData.txt",

load: function(type, data, evt){ /*do something w/ the data */ },

mimetype: "text/plain"

});

That's all there is to it. You provide the location of the data you want to get and a callback function that you'd like to have called when you actually DO get the data. But what about if something goes wrong with the request? Just register an error handler too:

dojo.io.bind({

url: "http://foo.bar.com/sampleData.txt",

load: function(type, data, evt){ /*do something w/ the data */ },

error: function(type, error){ /*do something w/ the error*/ },

mimetype: "text/plain"

});

It's possible to also register just a single handler that will figure out what kind of event got passed and react accordingly instead of registering separate load and error handlers:

dojo.io.bind({

url: "http://foo.bar.com/sampleData.txt",

handle: function(type, data, evt){

if(type == "load"){

// do something with the data object

}else if(type == "error"){

// here, "data" is our error object

// respond to the error here

}else{

// other types of events might get passed, handle them here

}

},

mimetype: "text/plain"

});

One common idiom for dynamic content loading is (for performance reasons) to request a JavaScript literal string and then evaluate it. That's also baked into bind, just provide a different expected response type with the mimetype argument:

dojo.io.bind({

url: "http://foo.bar.com/sampleData.js",

load: function(type, evaldObj){ /* do something */ },

mimetype: "text/javascript"

});

And if you want to be DARN SURE you're using the XMLHTTP transport, you can specify that too:

dojo.io.bind({

url: "http://foo.bar.com/sampleData.js",

load: function(type, evaldObj){ /* do something */ },

mimetype: "text/plain", // get plain text, don't eval()

transport: "XMLHTTPTransport"

});

Being a jack-of-all-trades, bind() also supports the submission of forms via a request (with the single caveat that it won't do file upload over XMLHTTP):

dojo.io.bind({

url: "http://foo.bar.com/processForm.cgi",

load: function(type, evaldObj){ /* do something */ },

formNode: document.getElementById("formToSubmit")

});

Phew. Think that about covers the basics. Good thing you weren't planning on implementing all that stuff yourself, right?

Remote Procedure Calls (RPC)

As you have seen, Dojo provides powerful, yet simple, ways of performing a variety of I/O functions through the use of dojo.io.bind. However, during the development of a typical application, a developer will have many I/O calls to make and will typically gravitate towards a common way of making those I/O calls on both the server and the client. This will often include defining functions that take some input and perform the appropriate request, as well as hooking that request to a callback function to process the results. In effect, the developer is required to implement a way of marshaling the request to the server in a way that it expects and then to have the client receive the contents in a way it expects. Dojo's RPC service aims to make this less error prone, easy to do, and require less code.

Remote Procedure Calls (RPC), also know as Remote Method Invocations, are a mainstay of the client/server development world. Essentially, RPC allows a developer to invoke a method on a remote host. Dojo provides a basic RPC client class that has been extended to provide access to JSON-RPC services and Yahoo services. It was designed so that it is also fairly trivial to implement custom RPC services.

Let's pretend that we have a little application that we want to make some server calls with. For simplicity's sake, we'll say the methods we want the server to do are add(x,y) and subtract(x,y). Without using anything special, like an RPC client, we might do something like this:

add = function(x,y) {



request = {x: x, y: y};



dojo.io.bind({

url: "add.php",

load: onAddResults,

mimetype: "text/plain",

content: request

});

}



subtract = function(x,y) {



request = {x: x, y: y};



dojo.io.bind({

url: "subtract",

load: onSubtractResults,

mimetype: "text/plain"

content: request

});

}


As you can see, this isn't particularly difficult. However, this is quite the simple application, despite our every attempt to make it complicated by having the server add or subtract two numbers instead of performing these operations in the client in the first place. What happens if our application is not so simple and has 30 different requests to make? I guess we would have to just write this same code over and over for each different request; each time making a request object, specifying URLs, potentially validating parameter types, and so on. This is simply error prone and boring to write.

Dojo's RPC clients simplify this whole process by taking a simple definition of the remote methods and application needs and generating client side functions to call these methods. A developer need only write this definition, and initialize a RPC client object and then all of these remote methods are available for the developer to use as normal.

The definition file, called a Simple Method Description (SMD) file, is a simple JSON string that defines a URL that will process the RPC requests, any methods available at that URL, and the parameters those methods take. The definition for our example above might look like this:

{

"serviceType": "JSON-RPC",

"serviceURL": "rpcProcessor.php",

"methods":[

{

"name": "add",

"parameters":[

{"name": "x"},

{"name": "y"}

]

},

{

"name": "subtract",

"parameters":[

{"name": "x"},

{"name": "y"}

]

}

]

}

Once the definition has been created, the code its pretty simple. The definition can be supplied either as a URL to retrieve it, a JSON string, or a JavaScript object.

var myObject = new dojo.rpc.JsonService("http://localhost/definition.smd");

var myObject = new dojo.rpc.JsonService({smdStr: definitionJSON});

var myObject = new dojo.rpc.JsonService({smdObj: definition});

Thats it! Now all thats left is to call the method.

myObject.add(3,5);

I'll bet you are saying to yourself, "Nice try, but I want to get the results of the add method, not just call it." You are correct, but that is also simple to achieve. Recall that we are making asynchronous calls to the server. While we could make the request synchronous, it would likely provide for a bad user experience because it would block the user interface during the call. Instead, the return value of the myObject.add() call, is a deferred object. The deferred object, something that might be familiar to users of Twisted Python or MochiKit, allows a developer to attach one or more callbacks and errbacks to the resultant data event. Our simple example can be expanded as such:

var myDeferred = myObject.add(3,5);

myDeferred.addCallback(myCallbackMethod);

or more succinctly:

var myDeferred = myObject.add(3,5).addCallback(myCallbackMethod);

As you can see, we've added myCallbackMethod as a callback for the deferred object returned from myObject.add(). In this case myCallbackMethod will be called with parameter with a value of 8. Likewise, an errback method can be attached to the deferred object to process an errors returned from the server. We can add as many callbacks and errbacks to our deferred object as we want and they will be called in the order that they were connected to the deferred object.

This discussion has revolved around using dojo.rpc.JsonService, which is Dojo's JSON-RPC client. In addition to JsonService, Dojo offers an RPC client for connecting to Yahoo services, dojo.rpc.YahooService. The syntax and call structure is identical. While Dojo is currently limited to these two RPC clients, the design of the dojo.rpc.RpcService base class, which is inherited by dojo.rpc.JsonClient and dojo.rpc.YahooService allows a developer to easily customize and extend dojo.rpc.RpcService, to create services that meets their specific needs. These customizations will be discussed later in Part II when we discuss how to get the most out of Dojo.

Transports

dojo.io.bind and related functions can communicate with the server using various methods, called transports. Each has certain limitations, so you should pick the transport that works correctly for your situation.

The default transport is XMLHttp.

IFrame I/O

The IFrame I/O transport is useful because it can upload files to the server. Example usage:

<script type="text/javascript">

dojo.require("dojo.io.*");

dojo.require("dojo.io.IframeIO");



function mySubmit() {

dojo.io.bind ({

url: 'server.cfm',

handler: callBack,

mimetype: "text/plain",

formNode: dojo.byId('myForm')

});

}
   function callBack(type, data, evt) {

//The data object will be different

//depending on the mimetype used in the dojo.io.bind()

//call. See below for more info.

dojo.byId('result').innerHTML = data;

}

</script>


The response type from the above URL can be text, html, or JS/JSON.

IframeIO responses need to be a little different from the ones that are sent back from XMLHttpRequest responses. Because an iframe is used, the only reliable, cross-browser way of knowing when the response is loaded is to use an HTML document as the return type.

If the return type (specified by the mimetype) is text/plain, text/javascript or text/json, then the server response should be an HTML page that has a <textarea> element. The data that you want returned to the dojo.io.bind() load callback should be the text inside the textarea element. For the text/javascript or text/json return types, the text inside the textarea element will be converted to JavaScript or JSON, repectively, and that will be the data sent to the load callback.

If the return type is text/html as the return type, then the data parameter will be the complete HTML document that is in the iframe.

For IframeIO, XML responses are not supported because we can't get a nice cross-browser solution. If you want text/html as the mimetype, what you get back is the document object for the document in the iframe.

See these tests for more info:

text/plain: http://archive.dojotoolkit.org/nightly/tests/io/test_IframeIO.text.html

text/html: http://archive.dojotoolkit.org/nightly/tests/io/test_IframeIO.html.html

text/javascript: http://archive.dojotoolkit.org/nightly/tests/io/test_IframeIO.html

ScriptSrcIO

Due to security restrictions, XMLHttp cannot load data from another domain. The ScriptSrcIO transport is useful for doing this. Yahoo's RPC service is implemented using ScriptSrcIO.

To use ScriptSrcIO, use the following require statements

  • dojo.require("dojo.io.*");
  • dojo.require("dojo.io.ScriptSrcIO");

and use the normal dojo.io.bind() method.

To force a ScriptSrcTransport request, use transport: "ScriptSrcTransport" in the keyword arguments to dojo.io.bind(). The mimetype argument is also required.

Example:
dojo.require("dojo.io.*");
dojo.require("dojo.io.ScriptSrcIO");

dojo.io.bind({
    url: "http://example.com/json.php",
    transport: "ScriptSrcTransport",
    mimetype: “application/json",
    jsonParamName: "callback",
    content: { ... }
});

ScriptSrcIO (which provides ScriptSrcTransport) allows for four basic types of requests:

Each type uses [script src="url"][/script] to accomplish the request.

Here is a list of bind() keyword arguments that are supported for all types of requests. The four types of transport requests are:

Simple

Simply adds a script element with a src. Does not do any polling and does not expect a callback. Also does not support any timeouts. Example:

dojo.io.bind({
	url: "http://the.script.url/goes/here",
	transport: "ScriptSrcTransport",
	mimetype: “text/javascript"
});

Polling

Adds a script element with a src. It will poll to see if a typeof expression does not equal undefined. When the typeof check succeeds, a load callback is called. Timeout and error callbacks are supported with this type of request.

Example:

dojo.io.bind({
	url: "http://the.script.url/goes/here",
	transport: "ScriptSrcTransport",
	mimetype: "text/javascript",
	checkString: "foo", //This means (typeof(foo) != undefined) indicates that the script loaded.
	load: function(type, data, event, kwArgs) { /* type will be "load", data and event null, , and kwArgs are the keyword arguments used in the dojo.io.bind call. */ },
	error: function(type, data, event, kwArgs) { /* type will be "error", data and event will have the error, , and kwArgs are the keyword arguments used in the dojo.io.bind call. */ },
	timeout: function() { /* Called if there is a timeout */},
	timeoutSeconds: 10 //The number of seconds to wait until firing timeout callback in case of timeout.
});

JSONP and JSON Callbacks

Adds a script element with a src. This sort of usage allows using services that use the JSONP convention to specify the callback that the server will use.  Specify the name of the JSONP callback parameter using jsonParamName. Yahoo! Web Services use a jsonParamName of "callback". Some other services use jsonParamName of "jsonp". Timeouts are supported with this type of request. Example for a data service that uses "callback" as the URL parameter:

dojo.io.bind({
	url: "http://the.script.url/goes/here",
	transport: "ScriptSrcTransport",
	mimetype: "application/json",
	jsonParamName: "callback",
	load: function(type, data, event, kwArgs) { /* type will be "load", data will be response data,  event will null, and kwArgs are the keyword arguments used in the dojo.io.bind call. */ },
	error: function(type, data, event, kwArgs) { /* type will be "error", data will be response data,  event will null, and kwArgs are the keyword arguments used in the dojo.io.bind call. */ },
	timeout: function() { /* Called if there is a timeout */},
	timeoutSeconds: 10 //The number of seconds to wait until firing timeout callback in case of timeout.
});

Here is a real example of using JSONP to look up the del.icio.us bookmarks.

<style type="text/css">
.bookmarks {
  width: 300;
  background: lightGray;
  border-style: solid;
  border-width: 2px;
  border-color: black
}
</style>
<script type="text/javascript" src="dojo.js"></script>
<script type="text/javascript">
dojo.require("dojo.io.*"); 
dojo.require("dojo.io.ScriptSrcIO");

dojo.addOnLoad(getBookmarks);

function getBookmarks() {
    dojo.io.bind({ 
        url: "http://del.icio.us/feeds/json/dojomaster", 
        transport: "ScriptSrcTransport", 
        jsonParamName: "callback",
        load: function(type, data, event, kwArgs){showBookmarks(data);},
        mimetype: "application/json",
        timeout: function() {alert('timeout');},
        timeoutSeconds: 10
    });
}

// The code for showing the bookmarks is courtesy of del.icio.us
// http://del.icio.us/help/json
function showBookmarks(posts) {
     var ul = document.createElement('ul');
     for (var i=0, post; post = posts[i]; i++) {
         var li = document.createElement('li');
         var a = document.createElement('a');
         a.style.marginLeft = '20px';
         var img = document.createElement('img');
         img.style.position = 'absolute';
         img.style.display = 'none';
         img.height = img.width = 16;
         img.src = post.u.split('/').splice(0,3).join('/')+'/favicon.ico'
         img.onload = showImage(img);
         a.setAttribute('href', post.u);
         a.appendChild(document.createTextNode(post.d));
         li.appendChild(img);
         li.appendChild(a);
         ul.appendChild(li);
     }
     document.getElementById("container").appendChild(ul);
}
function showImage(img){ return (function(){ img.style.display='inline' }) }
</script>
<div id="container" class="bookmarks"></div>

To customize this scirpt simply change the URL http://del.icio.us/feeds/json/dojomaster to include your del.icio.us user name. This example shows the bookmarks for the user "dojomaster".

DSR/Multipart

Adds a script element with a src. Uses the Dynamic Script Request convention to specify the callback that the server will use. Multipart requests (splitting a long request across multiple GET requests) is supported. Timeout and error callbacks are supported with this type of request. Example:

dojo.io.bind({
	url: "http://the.script.url/goes/here",
	transport: "ScriptSrcTransport",
	mimetype: "application/json",
	useRequestId: true, //adds the _dsrId to request with a generated ID. If a specific request ID is wanted, use apiId: "myId" instead
	//optional: forceSingleRequest: true, //Will not segment the request to multipart requests even if it is a long URL.
	constantParams: "name1=value1&name2=value2" //params to be sent with each request that is part of a multipart request. See spec.
	load: function(type, data, event, kwArgs) { /* type will be "load", data will be response data, event will be onscriptload event, and kwArgs are the keyword arguments used in the dojo.io.bind call. */ },
	error: function(type, data, event, kwArgs) { /* type will be "error", data will be response data, event will be onscriptload event, and kwArgs are the keyword arguments used in the dojo.io.bind call. */ },
	timeout: function() { /* Called if there is a timeout */},
	timeoutSeconds: 10 //The number of seconds to wait until firing timeout callback in case of timeout.
});

Common bind() arguments

ScriptSrcTransport supports the following arguments across all types of requests. In general, all of these arguments have the same meaning and use in XMLHTTPTransport.

  • mimetype: REQUIRED. Tells ScriptSrcTransport how to deal with the response. The only allowed values are "text/javascript", "text/json" or "application/json".
  • transport: RECOMMENDED. Tells dojo.io.bind() which transport you want to use. If you do not specify this transport, there is a chance that the XMLHTTPTransport might try to handle the request.
  • formNode: Uses a form to generate query parameters.
  • backButton, back, forward, forwardButton, changeUrl: used to register a back/forward handler. See dojo.undo.browser for more info.
  • content: a JS object that gets turned into query parameters.
  • postContent: Adds raw name=value parameters to query parameters.
  • sendTransport: Adds dojo.transport=scriptsrc to query parameters.
  • preventCache: Adds dojo.preventCache=[unique ID] to bypass browser cache and force a fresh GET.
  • handle: A function that accepts the following arguments: function(type, data, event) {}. This can be used instead of specifying a separate load, error and timeout handler. The type parameter will be a string that specifies the callback type ("load", "error", "timeout").

XMLHttp

The XMLHttp transport is the default transport.

It works well in most cases, but it cannot transfer files, cannot work across domains (ie, cannot connect to another site than the current page), and doesn't work with the file:// protocol.

Example usage:

<script type="text/javascript">
   dojo.require("dojo.io.*");

   function mySubmit() {
     dojo.io.bind ({
       url: 'server.cfm',
       handler: callBack,
      formNode: dojo.byId('myForm')
     });
   }
   function callBack(type, data, evt) {
      dojo.byId('result').innerHTML = data;
   }
</script>