The ease of Python, SQLite, and Storm
I began learning Python this spring, and I must say, the more I program in it the more I like it. I chose the language because of the libraries that are available for it. There is a library for everything. :) Also, there are tools for Natural Language Processing that are a great help, but that’s for another time and another post.
I was originally thinking about using Postgres, and it would probably give me better speed and scalability. But then, I began to think if I really needed a full RDBMS for my application. After all, I’m not expecting the project to get too large, and being able to easily move it from one computer to another by just moving a single files sounds very convenient. SQLite has a great page to see if it’s right for you. I ended up settling on SQLite, and so far am happy with the decision.
Installing SQLite was a breeze. I just opened the package manager in openSuSE and installed the packages. I also installed the python Storm package. There is no daemon process, as there is with Postgres, because you’re just accessing one file on your filesystem. There is a great tool for setting up a SQLite database called SQLite Manager. It will let you create tables, view your data, and run queries. The fact that it’s available as a firefox extention makes it easy to install on many platforms.
Now is when the real fun begins. Enter Storm.
Storm is an object relation mapping (ORM) tool for Python. It allows you to manipulate the database through the manipulation of Python objects. After you map your python objects to database tables, you manipulate them, and your changes will show up in the database for you. I’ve used other ORM tools in the past (Hibernate for Java), but I was amazed at the simplicity of the setup/configuration step when using Storm.
It takes two lines to connect to your sqlite database:
DATABASE = create_database('sqlite:db_name') # or simply create_database('sqlite:') for in-memory STORE = Store(DATABASE) |
Mapping a class to a table can be done with ease. Here is an example of one of my classes:
class Sentence(object): __storm_table__ = "TABLE_SENT" sent_id = Int(primary=True) loc_id = Int() location = Reference(loc_id, Location.loc_id) sentence = Unicode() def __init__(self, sent, loc = None): if loc: self.location = loc # sent cannot be None if not isinstance(sent, unicode): self.sentence = unicode(sent, "utf-8") else: self.sentence = sent |
If you access loc_id, it will give you the database id. If you access the variable without location through a Reference, it will hand you the corresponding database object.
Now, I set this up in about 45 minutes from start to finish, so it might need some more fiddling, but overall it seems to work pretty well. I needed to set something up in one night to keep moving on other parts of the project, and this allowed me to.
It can’t all be sunshine and rainbows, there was one thing that tripped me up a bit. Being new to Python, I wasn’t aware of the u”String” for unicode. It was used in their examples, and after I got an error assumed that’s what it was for, but it tripped me up. As you can see in my constructor, I added some code to handle the case when a string that isn’t unicode is passed in.
As I get into the more advanced aspects of SQLite/Storm, I hope I continue to be impressed.
on March 11, 2009 at 10:28 am
Permalink
I’am using storm in a postgres project. After one evening I fall in love with this nice and easy ORM.. Storm is great.. The documention is old and not enough. That is the one and only bad thing..
on March 11, 2009 at 12:41 pm
Permalink
Hello,
please add your site at http://www.sweebs.com where other people can find you among the best sites on the internet!
Regards
Kris
on May 18, 2009 at 6:41 pm
Permalink
[…] I did the post about Storm, someone suggested that I look into Elixir. Since I didn’t have time to at the […]
on December 1, 2009 at 2:34 pm
Permalink
Thanks for the intro to Storm. Another approach is to deal directly with Python objects (dictionaries, for example) which represent your data, and then serialize/compress/warehouse. To get started in less than 10 minutes, check out the y_serial module:
http://yserial.sourceforge.net/
Development is fast and flexible since the focus is schema-less data. For linguistic work, yserial has also been helpful in storing millions of text snippets in SQLite.