Assumptions

• I’m assuming you have a MySQL database running. More info here
• you have the MySQLdb package installed for Python.
• The database is running on localhost, the user is root, and the password is p@55w0rd.
• You know some SQL (at least enough to appreciate some of these examples)

Caveats

While the code below is specific to MySQLdb, no matter what database API you use you should be able to use the same syntax (as outlined in PEP 249).

For more details, see the official documentation for MySQLdb. Here I’m just trying to explain things slightly differently.

Example usage

Import MySQLdb, and connect to the database

import MySQLdb
my_connection = MySQLdb.connect(host='localhost', user='root', passwd='p@55w0rd')
cursor = my_connection.cursor()

That’s it! You’re ready to start sending SQL statements to your MySQL database!

The cursor is everything!

The cursor now contains all the information it needs to send information to and get information from the running MySQL server. It’s your key to the database.

The two most often-used methods of the MySQLdb cursor are

1. cursor.execute(), which executes a query (but doesn’t return the data)
2. cursor.fetchall(), which fetches the data from the most recently executed query.

You send commands to MySQL by passing strings of SQL statements to cursor.execute(). When doing so, you can take advantage of Python’s multi-line string (delimited by triple quotes (“”")) and the fact that SQL syntax doesn’t care that there are newlines in the query. Furthermore, MySQLdb automatically adds semicolons to the end of SQL statements if you forget them.

Interacting with the database

Create the database and a table

Make a new database by sending the standard SQL query, ‘CREATE DATABASE testdb’, to the database you connected to. Note that MySQLdb automatically adds semicolons to the end of statments if you don’t add them yourself.

cursor.execute('CREATE DATABASE testdb')

If you do this in an interactive session, you will notice that this method returned a long format integer (1L). This is the number of lines returned by the cursor. Don’t worry about it quite yet.

Now make that new database the active one:

cursor.execute('USE testdb')

Now create a table in the testdb database to hold some addresses:

cursor.execute('''CREATE TABLE addresses (
                    name VARCHAR(20),
                    street VARCHAR(20),
                    zipcode INT,
                    city VARCHAR(20),
                    state CHAR(2)
                    )
                    ''')

Note the use of triple quotes so that you can visually organize the SQL query for clarity.

Import data from Python into MySQL

The general syntax for passing Python data to an SQL query through the cursor is:

cursor.execute(SQL,tuple)

where SQL is a valid SQL statement. If SQL has N placeholders of the form %s, then tuple must have length N. Hopefully an example will make more sense.

Let’s create some Python lists that we’ll import into this table. The beauty of it is that these data could have been parsed from a text file with hundreds or thousands of names, and we can import them into the database automatically. For now we’ll just enter three records though.

Here’s the data that will go into the database:

names = ['Bob', 'Alfred', 'Jen']
streets = ['123 Elm Street', '55 Ninth Ave', '1 Paved Rd']
zips = [00123, 34565, 30094]
cities = ['Newark', 'Salinas', 'Los Angeles']
states = ['NJ', 'CA', 'CA']

And here’s how to get that data into the addresses table:

cursor.executemany('''INSERT INTO addresses
                     (name, street, zipcode, city, state)
                     VALUES
                     (%s, %s, %s, %s, %s)''',
                     zip(names, streets, zips, cities, states))

A couple of things to note here:

• This time we used cursor.executemany(), which will accept a list of lists as input, instead of cursor.execute().
• There were 5 fields into which we inserted data (name, street, zipcode, cities, and state)
• There were 5 %s placeholders in the SQL query.
• Even though zipcode is an INT field and not a string, we used %s. This will always be the case: use %s as a placeholder no matter what the datatype.
• There were 5 lists that were zipped together. They need to be zipped so that the result is a list of lists, and the length of each item in the list = 5.
• The order in which these lists were zipped corresponded to the fields into which they were to be inserted.

Retrieving data from the database

There are two steps to retrieving data: executing the query, then fetching the results.

To select all addresses in California, first execute this query (it’s a one-liner so triple quoting isn’t really needed)

cursor.execute("SELECT * FROM addresses WHERE state = 'CA' ")

Alternatively . . . often you will want to feed Python variables into the query. Say the state abbreviation ‘CA’ is saved in a Python variable called my_state. Then this query will do the same thing as the one above:

cursor.execute('''SELECT * FROM addresses WHERE state = %s''', my_state)

By the way, my_state is not a tuple, but that’s OK since there is only one %s placeholder in the query. MySQLdb knows where it should go.

Now to retrieve the results:

results = cursor.fetchall()

Note that a cursor object is similar to a file object or an iterator: once you fetch everything, there is nothing left in the cursor to retrieve. So executing the command above a second time would result in an empty list until the query is executed again.

results is a tuple of tuples and looks like this:

(('Alfred', '55 Ninth Ave', 34565L, 'Salinas', 'CA'),
 ('Jen', '1 Paved Rd', 30094L, 'Los Angeles', 'CA'))

That’s all there is to it! Armed with this knowledge, now you can execute queries from Python to import, retrieve, and plot data from your database. This was a simple demo of what MySQL and Python can do, but you can construct ever-larger databases and ever-more-sophisticated queries to manipulate data in ways that would be impossible without these tools.

site  sp1 sp2 sp3 elev
a      3   4   9   100
b      1   8   10  210
c      4   8   15  165

and reorganize into something like this (perhaps so we can do an ANOVA using species as a factor):

site  elev  sp  abundance
a     100  sp1  3
a     100  sp2  4
a     100  sp3  9
b     210  sp1  1
b     210  sp2  8
b     210  sp3  10
c     165  sp1  4
c     165  sp2  8
c     165  sp3  15

Assuming the first dataframe above is called d, the second dataframe can be obtained using the following code:

> library(ggplot2)
> m = melt(d, id=c('site','elev'))

melt works like this: You specify the ID variables, which are those variables that will REMAIN as dataframe variables. Any others will be considered measured variables. If it’s easier for your data, you can do it the other way: specify the measured variables and the others will be considered ID variables.

Melting results in two new variables, variable and value. variable contains the names of the original columns of the dataframe as factors, and value contains the corresponding values.

Another example

Here’s another example using the built-in dataset, airquality. First, unmelted:

# make all the variable names lowercase
names(airquality) <- tolower(names(airquality))
head(airquality)
  ozone solar.r wind temp month day
1    41     190  7.4   67     5   1
2    36     118  8.0   72     5   2
3    12     149 12.6   74     5   3
4    18     313 11.5   62     5   4
5    NA      NA 14.3   56     5   5
6    28      NA 14.9   66     5   6

and melted:

> head(melt(airquality,id=c('month','day')))
  month day variable value
1     5   1    ozone    41
2     5   2    ozone    36
3     5   3    ozone    12
4     5   4    ozone    18
5     5   5    ozone    NA
6     5   6    ozone    28

Long answer:

To use a database, you’ll have to learn another language, so you’ll have to decide for yourself if that “intellectual investment” is worth it. You also have to decide if you have enough data to make it worth the investment.

As far as learning a new language, SQL is one of the easier languages to learn (you can probably figure out what the following SLQ query does: SELECT title FROM books WHERE author = ‘John Tukey’).

As far as how much data is enough to justify a database, well that’s pretty subjective. But let me give you a specific example that might help guide your decision.

An example from personal experience

I was working on a project where we put out temperature loggers at many different sites. Every two weeks or so the temperature loggers would be moved to a different site, and we would download the data and keep them as text files.

Now, if there were just one or two temperature loggers, I probably wouldn’t make a database. If I wanted to compare sites, it would be easy enough to load the one or two text files from each site and compare them.

If I had, say, 10 temperature loggers but they stayed in the same place all the time, then I probably would write a Python script that read the data in from the text files each time I wanted to analyze the data. Still not enough to justify a database in my opinion.

But I had 48 temperature loggers all at different locations. Keeping track of text files would quickly get out of hand. Plus, there was other information that went along with the different sites.

Having extra data that went along with the sites is important. That is information that was NOT contained within the temperature logs, it was in my field notebook (and ultimately in a spreadsheet). Without a relational database, it would be difficult to pull out the temperatures where the sites were mud as opposed to sand. In fact, it would be awkward to do that if I only had 10 temperature loggers, let alone 48.

When you have this sort of data — where some parts can be related to other parts — it’s a good sign you should be using a database.

This is only an example of one case where you might want to use a database. You could use a database for anything from organizing recipes to clinical studies to meta-analysis of literature.

Next steps

Once you’ve decided to use a database, you’ll have to decide which one to use. The two most popular open source ones are PostgreSQL (here’s how to pronounce it) and MySQL.

To get started, check out XAMP for Windows, MAMP for Mac, or LAMP for Linux. Note you only need Apache for using phpMyAdmin, the user friendly, browser-based database management program. You might try MySQL Query Browser
instead of running Apache and phpMyAdmin.

Next up is designing your database. Databases are just a bunch of separate tables, and they are not truly connected until you perform a query.

Once everything is ready, you’ll want to import your data into it. My personal preference is to use Python to parse data files, and then insert the data into the database using something like MySQLdb (a way for Python to talk to a MySQL database). If you have simple files and don’t need the flexibility of Python, you can use a command like MySQL’s LOAD DATA command to do it for you.

Final notes

Keep in mind that taking the plunge and porting your data into a database can be a large project. But the flexibility you gain by using a database allows you to easily call up combinations of data that would be laborious without it goes a long way.