A Simple Data Source

The easiest data store is a static one, so let's begin with that. The file in the following example has the /pantry_spices.json URL:

{ identifier: 'name',
   items: [
	{ name: 'Adobo', aisle: 'Mexican' },
	{ name: 'Balsamic vinegar', aisle: 'Condiments' },
	{ name: 'Basil', aisle: 'Spices' },
	{ name: 'Bay leaf', aisle: 'Spices' },
        { name: 'Beef Bouillon Granules', aisle: 'Soup' },
...
	{ name: 'Vinegar', aisle: 'Condiments' },
	{ name: 'White cooking wine', aisle: 'Condiments' },
        { name: 'Worcestershire Sauce', aisle: 'Condiments' }]}

In this example:

• The data source is /pantry_spices.json.
• Each item is an ingredient, and each item has two attributes, name and aisle.
• The name attribute is an identifier. Each ingredient has a different name to prevent confusion.

This is a simple but useful technique. Because this data source is a separate file, you can share the data among many pages. But what can you do with it? The following simple sample displays a select list of pantry items:

The class for the PantryStore, dojo.data.ItemFileReadStore, tells dojo.data to expect the data in a specific format that uses JSON structure to store the data. Our static file is in pantry_items.json so this is the URL. The target could also be a dynamic, server-run script that returns the specific data in the defined JSON format. Other widgets, like Tree and Grid, also use the dojo.data objects and URL to load data into them.

Fetching Single Items and Values

For this example, we'll assume the following simple data source:

{ identifier: 'name', 
  items: [
	{ name: 'Adobo', aisle: 'Mexican' },
	{ name: 'Balsamic vinegar', aisle: 'Condiments' },
	{ name: 'Basil', aisle: 'Spices' },
	{ name: 'Bay leaf', aisle: 'Spices' },
        { name: 'Beef Bouillon Granules', aisle: 'Soup' },
// ...
	{ name: 'Vinegar', aisle: 'Condiments' },
	{ name: 'White cooking wine', aisle: 'Condiments' },
        { name: 'Worcestershire Sauce', aisle: 'Condiments' },
        { name: 'pepper', aisle: 'Spices' }
]}

Working with One Item

You might want to access items directly and work with one item at a time. Stores that implement the identity interface allow you to do this quite easily. In this example of accessing an item by its unique identifier, the following APIs are used:

Identity

fetchItemByIdentity() Fetches an item by its key value. Because the identity value of each item is unique, you are guaranteed at most one answer back.

Read

getValue() Takes an item and an attribute and returns the associated value

The following code fragment returns the aisle pepper is in:

var pantryStore = new dojo.data.ItemFileReadStore({
  url: "pantry_items.json" 
});


pantryStore.fetchItemByIdentity({
  identity: "pepper", 
  onItem: function(item){
    console.debug("Pepper is in aisle ", pantryStore.getValue(item,"aisle"));
  }
});

Note: In the previous example, the fetch is asynchronous. This is because many Datastores will need to go back to a server to actually look up the data and some I/O methods do not readily allow for a synchronous call.

Fetching Multiple Items and Values

For this example, we'll assume the simple data source detailed on the previous page

Working with Multiple Items

You will likely want to access multiple items from such a data source as in the preceding example. No problem! Dojo.data Read API provides a mechanism for loading a set of items. All you have to do is provide the following information to the fetch function of the Read API:

• This is what I want (if I don't tell you something, get everything)
• Do this if there is an error
• Do that when everything is loaded

If this sounds like it might be event-driven, that's because it is. The prime method to call, dojo.data.api.Read.fetch(), is asynchronous. This is because, in a wide variety of data access cases, the datastore will have to make a request to a server service to get the data. Many I/O methods for doing server requests only behave in an asynchronous manner. Therefore, the primary search function of dojo.data is asynchronous by definition.

In this example, the Read API is used with the following values:

fetch()

Asynchronous API that fetches a set of items which match a list of attributes.

getValue()

Takes an item and an attribute and returns the associated value.

The following code fragment returns all items:

Working with Lots of Items

Now that we've looked at dealing with getting a list of items in one batch, what if the list is huge? It could take a long time to get all the items, push them into an array, and then call the callback with the array of items. Wouldn't it be nice if you could stream the items in, one at a time, and do something each time a new item is available? Well, with dojo.data, you can do that! There is an alternate callback you can pass to fetch() that is called on an item by item basis. It is the onItem callback.

In the following example, the code will request that all items be returned (an empty query). As each item gets returned, it will add a textnode to the document. In this example, the Read API is used with the following values:

fetch()

Asynchronous API that fetches a set of items which match a list of attributes.

getValue()

Takes an item and an attribute and returns the associated value.

The following code fragment loads all items and streams them back into the page:

Note: If the onItem callback is present in the parameters to fetch, then the first parameter to the onComplete callback, the items array, will always be null. Therefore, onItem is streaming only mode and does not rely on onComplete for anything other than a signal that the streaming has ended.

Selecting Items

There are many times when you might not want an entire item list. Though you could fetch the entire list, and loop through to select elements, dojo.data's API definition has facilities to do the tough work for you.

Selecting subsets of items requires a query. A query is a JavaScript object that has attributes which look a lot like the attributes of the data store. It's a kind of query-by-example. So for our pantry, selecting the items from the Spice aisle involves this query:

{ aisle: "Spice" }

Each type of data store can have its own query syntax. With JsomItemStore, you can use wildcards: * to mean any characters. and ? to mean one character. This notation will be familiar to you if you've worked with Perl, Java, UNIX shell regular expressions, or even old BATCH scripts. And in general, the dojo.data community would highly recommend that all stores try to follow this method of specifying the query for consistency. Even datastores that are backed by an SQL database should be able to handle such character matching, because * maps to %, and ? maps to _ in SQL syntax.

The following query will pick up items in the Condiments aisle:

{ aisle: "Condiment*" }

Multiple attributes assume an "and" between the terms. So a query like the following one will match spices with the word pepper inside them, but not "green peppers" in the vegetable aisle:

{ name: "*pepper*", aisle: "Spices" }

Once we have constructed the query, we pass it to fetch() along with the other parameters as shown in the following example:

Finally, it's important to note that searches are case-sensitive by default. If you want to make a case-insensitive query, just add the ignoreCase option to the queryOptions object as shown in the following example:

itemStore .fetch({ 
        queryOptions: {ignoreCase: true},
        query: { name: "*pepper*", aisle: "Spices" },
        onComplete: 
                ...    
   });

This example will match both "Black Pepper" and "white pepper."

In general, any option that would affect the behavior of a query, such as making it case insensitive or doing a deep scan where it scans a hierarchy of items instead of just the top level items (the deep:true option), in a store belongs in the queryOptions argument.

Why isn't it just SQL for a query?

The simple and short answer to this question is that not all datastores are backed directly by a database that handles SQL. An immediate example is ItemFileReadStore, which just uses a structured JSON list for its data. Other examples would be datastores that wrap on top of services like Flickr and Delicious, because neither of those take SQL as the syntax for their services. Therefore, the dojo.data API defines basic guidelines and syntax stores that can be easily mapped to a service (for example, attribute names can map directly to parameters in a query string). The same is true for an SQL backed datastore. The attributes become substitutions in a prepared statement that the stores use (when they pass back the query to the server) and a simple common pattern matching syntax, the * and ?, which also map easily across a wide variety of datasource query syntax.

Nested Items and Lazy Loading

Items can be handled in chunks, as well as streamed in. The other articles about datastores show items as a flat list with no hierarchy. So, what if you want a datastore to represent hierarchical data? And how do you walk across the hierarchy? Walking the hierarchy is, in fact, quite easy to do. The following example shows how to do this using JsonItemStore.

Assume a datasource of:
  { identifier: 'name',
     label: 'name',
     items: [
       { name:'Africa', type:'continent',
           children:[{_reference:'Egypt'}, {_reference:'Kenya'}, {_reference:'Sudan'}] },
       { name:'Egypt', type:'country' },
       { name:'Kenya', type:'country',
           children:[{_reference:'Nairobi'}, {_reference:'Mombasa'}] },
       { name:'Nairobi', type:'city' },
       { name:'Mombasa', type:'city' },
       { name:'Sudan', type:'country',
           children:{_reference:'Khartoum'} },
       { name:'Khartoum', type:'city' },
       { name:'Asia', type:'continent',
           children:[{_reference:'China'}, {_reference:'India'}, {_reference:'Russia'},          {_reference:'Mongolia'}] },
       { name:'China', type:'country' },
       { name:'India', type:'country' },
       { name:'Russia', type:'country' },
       { name:'Mongolia', type:'country' },
       { name:'Australia', type:'continent', population:'21 million',
           children:{_reference:'Commonwealth of Australia'}},
       { name:'Commonwealth of Australia', type:'country', population:'21 million'},
       { name:'Europe', type:'continent',
           children:[{_reference:'Germany'}, {_reference:'France'}, {_reference:'Spain'}, {_reference:'Italy'}] },
       { name:'Germany', type:'country' },
       { name:'France', type:'country' },
       { name:'Spain', type:'country' },
       { name:'Italy', type:'country' },
       { name:'North America', type:'continent',
           children:[{_reference:'Mexico'}, {_reference:'Canada'}, {_reference:'United States of America'}] },
       { name:'Mexico', type:'country',  population:'108 million', area:'1,972,550 sq km',
           children:[{_reference:'Mexico City'}, {_reference:'Guadalajara'}] },
       { name:'Mexico City', type:'city', population:'19 million', timezone:'-6 UTC'},
       { name:'Guadalajara', type:'city', population:'4 million', timezone:'-6 UTC' },
       { name:'Canada', type:'country',  population:'33 million', area:'9,984,670 sq km',
           children:[{_reference:'Ottawa'}, {_reference:'Toronto'}] },
       { name:'Ottawa', type:'city', population:'0.9 million', timezone:'-5 UTC'},
       { name:'Toronto', type:'city', population:'2.5 million', timezone:'-5 UTC' },
       { name:'United States of America', type:'country' },
       { name:'South America', type:'continent',
           children:[{_reference:'Brazil'}, {_reference:'Argentina'}] },
       { name:'Brazil', type:'country', population:'186 million' },
       { name:'Argentina', type:'country', population:'40 million' }
  ]}

The above datasource for JsonItemStore uses references to other items to build the hierarchy. Other datasources and datastores might use different internal representations for hierarchy. But, in this example, notice that the continent type items have children that are countries, which in turn have children that are cities.

The following code snippet walks across this hierarchy and displays all country items contained by the continent items:

var store = new dojo.data.ItemFileReadStore({url: "countries.json"});

//Load completed function for walking across the attributes and child items of the
//located items.
var gotContinents = function(items, request){
    //Cycle over all the matches.
    for(var i = 0; i < items.length; i++){
        var item = items[i];

        //Cycle over all the attributes.
        var attributes = store.getAttributes(item);
        for (var j = 0; j < attributes.length; j++){
            //Assume all attributes are multi-valued and loop over the values ...
            var values = store.getValues(item, attributes[j]);
            for(var k = 0; k < values.length; k++){
                var value = values[k];
                
                if(store.isItem(value)){
                    //Test to see if the items data is fully loaded or needs to be demand-loaded in (the item in question is just a stub).
                    if(store.isItemLoaded(value)){
                        console.log("Located a child item with label: [" + store.getLabel(value) + "]");
                    }else{
                        //Asynchronously load in the child item using the stub data to get the real data.
                        var lazyLoadComplete = function(item){
                            console.log("Lazy-Load of item complete.  Located child item with label: [" + store.getLabel(item) + "]");
                        }
                        store.loadItem({item: value, onItem: lazyLoadComplete});
                    }
                }else{
                    console.log("Attribute: [" + attributes[j] + "] has value: [" + value + "]");
                }
            }           
        }
    }
}

//Call the fetch of the toplevel continent items.
store.fetch({query: {type: "continent"}, onComplete: gotContinents});

Note: The previous sample also demonstrates how lazy-loading (on-demand) of items can be done through the combination of the isItemLoaded() and loadItem() functions. For a demonstration of a lazy-loading approach that uses an extended version of ItemFileReadStore, see the demo in dojox/data/demos/demo_LazyLoad.html.

Pagination

As shown in the other datastore sections, the fetch method of the Read API can query across and return sets of items in a variety of ways. However, there is generally only so much space on a display to list all the data returned. Certainly, an application could implement its own custom display logic for just displaying subsets of the data, but that would be inefficient because the application would have had to load all the data in the first place. And, if the data set is huge, it could severely increase the memory usage of the browser. Therefore, dojo.data provides a mechanism by which the store itself can do the paging for you. When you use the paging options of fetch, all that is returned in the callbacks for fetch is the page of data you wanted, no more. This allows the application to deal with data in small chunks, the parts currently visible to you.

The paging mechanism is used by specifying a start parameter in the fetch arguments. The start parameter says where, in the full list of items, to start returning items. The index 0 is the first item in the collection. The second argument you specify is the count argument. This option tells dojo.data how many items, starting at start, to return in a request. If start isn't specified, it is assumed to be 0. If count isn't specified, it is assumed to return all the items starting at start until it reaches the end of the collection. With this mechanism, you can implement simple paging easily.

To demonstrate the paging function, we'll assume an ItemFileReadStore with the following datasource:

{   
    identifier: 'name', 
    items: [
        { name: 'Adobo', aisle: 'Mexican' },
        { name: 'Balsamic vinegar', aisle: 'Condiments' },
        { name: 'Basil', aisle: 'Spices' },
        { name: 'Bay leaf', aisle: 'Spices' },
        { name: 'Beef Bouillon Granules', aisle: 'Soup' },
        ...
        { name: 'Vinegar', aisle: 'Condiments' },
        { name: 'White cooking wine', aisle: 'Condiments' },
        { name: 'Worcestershire Sauce', aisle: 'Condiments' }
        { name: 'pepper', aisle: 'Spices' }
    ]
}

The following code snippet would allow for paging over the items, 10 at a time:

var store = new dojo.data.ItemFileReadStore({url: "pantryStore.json" });

    var pageSize = 10;
    var request = null;
    var outOfItems = false;

    //Define a function that will be connected to a 'next' button
    var onNext = function(){
       if(!outOfItems){
           request.start += pageSize;
           store.fetch(request);
       }
    };
    //Connect this function to the onClick event of the 'next' button
    //Done through dojo.connect() generally.

    //Define a function will be connected to a 'previous' button.
    var onPrevious = function(){
       if (request.start > 0){
          request.start -= pageSize;
          store.fetch(request);
       }
    }
    //Connect this function to the onClick event of the 'previous' button
    //Done through dojo.connect() generally.

    //Define how we handle the items when we get it
    var itemsLoaded = function(items, request){
       if (items.length < pageSize){
          //We have found all the items and are at the end of our set.  
          outOfItems = true;
       }else{
          outOfItems = false;
       }
       //Display the items in a table through the use of store.getValue() on the items and attributes desired.
       ...
    }

    //Do the initial request.  Without a query, it should just select all items.  The start and count limit the number returned.
    request = store.fetch({onComplete: itemsLoaded, start: 0, count: pageSize});

Note: The previous sample shows how the fetch() function returns a request object. According to the dojo.data specification, the request object contains the query parameters passed in plus an abort() function appended to it. In general, the abort function is intended for cases in which a request might take too much time to process or, in using the streaming callback of fetch(), a way to stop the streaming.

The request object also serves another purpose for datastores. It is a location where the store can cache hidden information about the request in process, such as a cache entry key for boosting performance through specifying exactly what internal cache might be in use for this particular query. Therefore, datastores can avoid calls to the server if possible. And, in the paging case, it becomes important to reuse the request object returned from fetch(). Also note that not all stores will append cache information to the request, but some might. Therefore, when in doubt, reuse the request object when paging.

Sorting

Items are, in general, returned in an indeterminate order. This isn't always what you want to happen; there are definite cases where sorting items based on specific attributes is important. Fortunately, you do not have to do the sorting yourself because dojo.data provides a mechanism to do it for you. The mechanism is just another option passed to fetch, the sort array.

The sort array will look something like the following example:

var sortKeys = [{attribute: "aisle", descending: true}];

Each sort key has an attribute, which should be an attribute in the data store item, and a descending boolean flag. If an attribute passed isn't an actual attribute of the item, it will generally be ignored by the sorting; it is treated as an undefined comparison.

For compound sorts, you can define as many sort keys as you want. The order in the array defines which keys take priority over other keys when sorting. The following example shows this:

var sortKeys = [
    {attribute: "aisle", descending: true},
    {attribute: "name", descending: false}
];

A Complete Example

For this example, we'll use the ItemFileReadStore data source:

{   identifier: 'name',
    items: [
        { name: 'Adobo', aisle: 'Mexican' },
        { name: 'Balsamic vinegar', aisle: 'Condiments' },
        { name: 'Basil', aisle: 'Spices' },
        { name: 'Bay leaf', aisle: 'Spices' },
        { name: 'Beef Bouillon Granules', aisle: 'Soup' },
        ...
        { name: 'Vinegar', aisle: 'Condiments' },
        { name: 'White cooking wine', aisle: 'Condiments' },
        { name: 'Worcestershire Sauce', aisle: 'Condiments' }
        { name: 'pepper', aisle: 'Spices' }
    ]
}

Now, assume you want to sort the items in the store by aisle first, then by name. The following code snippet would do this:

var store = new dojo.data.ItemFileReadStore({url: "pantryStore.json"});

var sortKeys = [
    {attribute: "aisle", descending: true},
    {attribute: "name", descending: false}
];

store.fetch({ 
       sort: sortKeys;
       onComplete: 
           ...
});
//When onComplete is called, the array of items passed into it 
//should be sorted according to the denoted sort array.