Tuesday, November 22, 2011

New localstorage based database API

We were initially working on Firefox 4.0 and Indexed DB, and I thought of writing some tips and tricks on getting it working. It turns out that Indexed DB is not yet supported on Android browsers and iPhone browser, which is a shame, because we want to run our web-app throughout the whole ecosystem of devices that are now available in the market.

Most browsers support a basic subset of HTML5 standards. But Indexed DB, surprisingly, is not yet heavily adopted. So we were faced with the dilemma of how should we get our app running on Linux, Windows, Mac, iOS, Android etc.

We looked at Web database and Indexed DB, both of which have serious limitations, as of now. So I decided that something had to be done. And I clicked upon a great idea of writing my own API for storage, that works on localstorage.

Now local storage is being supported by all browsers. But it is a very simplistic approach of storing client side data in key-value pairs. This means that you can probably store a To-do list using it, but you may not find it convenient for having something like an electronic data-collection app storing electronic forms, data filled in for the forms, users, settings all in local storage. All this complexity needs a database like storage and organization in form of object stores or tables.

So the local storage API came into being. What it does is - it is just a layer above the localstorage of the browser, so that a developer can write statements to create a table, add columns, add records, retrieve records, search on the records, drop tables, drop full database and more. The API actually stores all data in certain specific key formats so that organization is possible and the API can easily do the operations required by the developer.

The advantages of this approach are manifold, some of which were already mentioned, but here they are again:
1. Requires local storage to be working on the browser, something that's supported on all major browsers right now, including mobile phone browsers.
2. Local storage is synchronous in nature. I am not sure since this could be a disadvantage for some, since they may be looking at asynchronous operations. But the thing is, our app needed synchronous operation, and it much easier to write code with synchronous requests.
3. Is somewhat like Indexed DB in implementation, no sql knowledge is needed.
4. Fairly easy to use the API

For more details and for downloading it, please go to:
https://code.google.com/p/gwt-localstorage-db/

Thursday, May 19, 2011

Indexed DB tips and tricks (For Firefox 4)

In these days, webapps are becoming more and more popular and we have a strong case for webapps to replace even desktop apps. For one, webapps run the same way on different operating systems. They just need a browser for running.


After coming up of HTML5 spec, we are looking at really powerful webapps which can have the ability to work offline. Different browsers try to implement HTML5 in their own manner, which is a problem for developers. Especially, since HTML5 is new, there are lot of uncertainties in this area.


One important thing I came across recently is the Indexed DB spec that Firefix 4 and other browsers are implementing. They have made a big change from the Gears implementation (Sqlite) as well as the WebSQL implementation (Sqlite again). Sometimes it is difficult to find proper documentation in this regard.


Since we are currently working on getting IndexedDB working on Firefox 4, we went through a lot of Mozilla documentation and found out that it is quite confusing for a person to understand IndexedDB API.


So I thought that I should share my experiences in this regard, and give some tips and tricks, and some sample code on how to get started.


1. First thing first, let me give you the correct links for the documentation:
https://developer.mozilla.org/en/IndexedDB/IndexedDB_primer

This is an excellent IndexedDB primer from the Mozilla team. It also gives links to other important documentation like the Mozilla IndexedDB API and W3 spec.

Other documentation found on sites like HTML5 rocks etc. sadly don't work on Mozilla Firefox4.
http://hacks.mozilla.org/2010/06/comparing-indexeddb-and-webdatabase/
This is also grossly outdated, and I wasted a lot of my precious time in trying to run the code given here.


2. Note that creating or opening a db on the Firefox 4 has a little different way of doing it. You write a call through mozIndexedDB, not window.indexedDB


var request = mozIndexedDB.open("MyTestDatabase");


To do error handling at one place, so that you don't have to do it repeatedly, you can include this code in the onsuccess() method for opened db.



db.onerror = function(event) {
  // Generic error handler for all errors targeted at this database's
  // requests!
  alert("Database error: " + event.target.errorCode);
};


3. When you try to run IndexedDB through java script embedded in a local HTML file, you will get nasty errors : errors of type unknown and it won't work whatever you do. But if you put in on a web server, then the same html file works and your IndexedDB will be happily created.


One way to create local IndexedDB so that you can test out your applications is: we created a GWT project, and put all our JavaScript code inside a script called from the HTML file inside the GWT project. Then we ran the GWT project and sure enough, it worked well. (I have to say here that I have been working on Java and GWT for quite some time, and am relatively new to JS and JSNI)


4. On Firefox4, if you want to debug your JavaScript code, you need to goto
Tools->Web Console
and this should bring up the web console where you get all the error messages.


5. We tried writing all IndexedDB code as a JSNI function inside our Java classes (since we are coding in GWT or Java), and we had lot of problems in certain JS commands like delete or const. So there is a very simple way to include your IndexedDB code as JS script and still access it from your Java code. Let me give you short steps on how to do that:


First, write all your JS script files inside the main HTML file or access it there using something like:

<script type="text/javascript" src="createDB.js">
</script>
where createDB.js contains the code for creating database and objectStores (say there is a function called createDB() inside the .js)


Now you can easily access this file from your HTML file, by just including the following code:

<script type="text/javascript">
createDB();
</script>

Ah, but this is not very impressive, since we want to access this function inside your Java class, not through your HTML file. So, in order to do that, you do something like this in your Java file:

public void onModuleLoad() {
createDb();
}


public native void createDb() /*-{
$wnd.createDB();
}-*/;

And voila, you have all functions defined inside .js file accessible through JSNI in your GWT java file.


You can pass objects from Java to JS, JS to Java etc. without any problems from now on.


6. An important thing to note is that you will have to open the database every time you want to do a transaction on the Indexed DB. It seems like that the db closes itself once you are done with the transaction and you have no more code for it in that particular function. This is a bit annoying since it increases the size of your code, but my guess is that this was deliberately done to prevent memory leakages or having unsaved changes on the db.


7. In case you were wondering how to specify an object store with autoIncrement as true, here is some sample code:
var user_form = db.createObjectStore("user_form", {keyPath: "user_form_id", autoIncrement: true} );

Then you can create indexes inside the createIndex command like this:
user_form.createIndex("name", "name", { unique: false });
unique:true will make the column as a unique column type, so that it is always unique.


8. Some people may tell you that the 'put' function is something like the 'add' function for adding the data to the Indexed DB. That is a misconception, since 'put' has some very important differences that should be carefully noted. 


What they tell you is that you can use 'put' function to either add or replace a record in the db. But what they don't tell you is that 'put' function needs the Primary key as an input for it to work.
Let me explain by example. If suppose, your object store(table) had an auto increment key as Sno and a data column called Name.
In order to use the add function, you can just specify Name to the function and it will auto-generate the key by itself. But if you use the 'put' function here, it won't work and it will show an error. And that goes for even if the Name you specified is already existing. For doing a put, you will have to specify the key. So, if you really want to auto generate the key, you have no option but to use the 'add' function, and if you want to edit certain location, then you have to do a search and find what is the key for that particular data, and then pass it on the 'put' function.
Yes, I know this is going to make your code longer, but I don't think there is any other method to doing this. If you have a situation when you don't know whether you have to do an 'add' or a 'put', then you have to first search for a particular index, if found, do a 'put' at the found key, else do an 'add'.

That's it for now. I will keep on updating with more tips and tricks as I learn more while getting the Indexed DB working on FF.