Migrating data from SQL to NoSQL

Introduction :

NoSQL - probably the hottest term in database technology today. Database volumes have grown continuously since the earliest days of computing, but that growth has intensified dramatically over the past decade as databases have been tasked with accepting data feeds from customers, the public, point of sale devices, GPS, mobile devices, RFID readers and so on.    

Cloud computing also has placed new challenges on the database.  The economic vision for cloud computing is to provide computing resources on demand with a "pay-as-you-go" model.  A pool of computing resources can exploit economies of scale and a leveling of variable demand by adding or subtracting computing resources as workload demand changes.  The traditional RDBMS has been unable to provide these types of elastic services.

First step of trying out any NoSQL database is, migrating the data from SQL to NoSQL system. But it isn’t straight forward task to do so. Often we have to de-normalize our schema before importing, because most of the NoSQL solutions don’t have support for table joins.

SQL to NoSQL IMPORTER:

I have worked with Solr’s data import handler to import data from mysql systems in my previous work place. It’s a pretty flexible one as you can write your own queries to fetch the data, create flat documents with your desired structure. I was searching for something like this to import data from mysql to mongodb. The main important requirement was, the tool has to de-normalize the data on the fly while importing. I wasn’t able to find anything useful. As it’s a simple utility I sat down to write it on my own. The result is http://code.google.com/p/sql-to-nosql-importer/ . It has support for MongoDB, CouchDB and ElasticSearch.

 

HOW to RUN:

1)      Head over to project site and download the latest version http://code.google.com/p/sql-to-nosql-importer/downloads/list

2)     Go to conf folder and open the import.properties file. This is the file where you have to supply properties of the NoSQL system. Default configuration file is shown below.

3)     In the same folder there is one more file called “db-data-config.xml” where you have to supply all the source db (SQL) related properties. This file is more like solr’s data import handler input file. Default is shown below.

4)     After changing these values just run the run.sh or run.bat depending on your operating system.

Conclusion:

Migrating data from SQL to NoSQL system is the initial task as well as important task in trying out NoSQL systems. Give more attention to your schema design before trying out importing the data. Please let me know in the comments section if you face any difficulties.