I had many, many figures in my working directory, but I didn’t use all of them in the Latex document. I was trying to figure out a way to send the source files — *.tex, *.cls, *.bst, *.bib, etc, plus only the images files that were actually in the document — to someone else so they could edit on their own and compile on their own. I didn’t want to set up a version control (SVN, etc), I just wanted a tar file.

After some poking around I couldn’t find anything already made that would do this (Kile has an Archive menu item, but this doesn’t include figures). It was easy enough to get a Python script going.

This script parses an input file, looks at the various documents and figures that are included, and archives them in a tar.gz file which can then be sent to someone. Note that as it stands, it only looks two levels deep for \include tags. If I use this more I’ll have to make it recursive (it’s not obvious to me how to do that, I haven’t used recursion much before).

Consider this script a rough draft. It worked perfectly for me, but your mileage may vary.

"""
This script gathers the necessary images and files (from
an arbitrarily large number of unneeded figures) and
puts it all in a tarball for distribution.

Usage: latexpackager.py main.tex dissertation.tar.gz
"""

import sys
import re
import os
import tarfile

def find_references(f):
    '''Returns a list of Latex files that f refers to,
    by parsing \include, \bibliography, \bibliographystyle,
    \input, etc.

    If nothing was found, returns an empty list.'''

    s = open(f).read()

    # Find the .tex files.
    texs = []
    for i in re.finditer(r"""[^%]\\include\{(.*)\}""", s):
        texs.append(i.groups()[0]+'.tex')

    # Find the .bib files.
    bibs = []
    for i in re.finditer(r"""[^%]\\bibliography\{(.*)\}""", s):
        bibs.append(i.groups()[0]+'.bib')

    # Find the styles.
    styles = []
    for i in re.finditer(r"""[^%]\\bibliographystyle\{(.*)\}""", s):
        styles.append(i.groups()[0]+'.bst')

    # Find the document class description file
    docclass = []
    for i in re.finditer(r"""[^%]\\documentclass\{(.*)\}""", s):
        docclass.append(i.groups()[0]+'.cls')

    # Look for any inputs.
    inputs = []
    for i in re.finditer(r"""[^%]\\input\{(.*)\}""", s):
        texs.append(i.groups()[0]+'.tex')

    # Here is everything that was referenced in f:
    return texs + bibs + styles + docclass + inputs

def find_figures(f):
    '''Returns a list of figures found in the file.  Only
    looks in .tex files.  If not a .tex file or no figures found,
    returns an empty list.'''

    # Short circuit if not a .tex file.
    if f[-4:] != '.tex':
        return []

    includegraphics = r"""[^%].*\\includegraphics\[.*\]\{([^\}]*)\}"""
    figures = []
    s = open(f).read()
    matches = re.finditer(includegraphics, s)

    for match in matches:
        basename = match.groups()[0]
        if basename[-4] == '.':
            # that is, it has an extension already.
            # This is for things like .png images.
            figures.append(basename)
        else:
            figures.append(basename + '.pdf')
            figures.append(basename + '.eps')

    return figures

main = sys.argv[1]
tarfn = sys.argv[2]

projectdir, main = os.path.split(main)
if projectdir == '':
    projectdir = os.getcwd()

keepers = find_references(main)

# Don't forget to add the main .tex file.
keepers.append(main)

# For each of those that main.tex referenced, look for more.
# These are files referenced two levels deep.

for f in keepers:
    if f[-4:] != '.tex':
        continue
    keepers.extend(find_references(f))

# Now look for graphics.

figures = []
for f in keepers:
    figures.extend(find_figures(f))

#paths = [os.path.join(projectdir, i) for i in keepers + figures]
paths = keepers + figures

tarball = tarfile.open(tarfn, 'w:gz')
for path in paths:
    print path
    tarball.add(path)
tarball.close()

Note that in order to use R from Python, you need to know a little of both . . . so the learning curve can be steep. You also need to have a feel for what would be easy in R and what would be easy in Python.

There are some detailed examples below if you want to skip right to ‘em.

I use Python for most tasks, but when I need high-powered stats, I embed R code in my Python scripts to perform the analysis.

Disclaimer: I figured all of this stuff out by trial and error. The RPy documentation, while complete, was difficult for me to make sense of when I was learning. If there’s a better way to do things, please let me know! For the details that I don’t cover here, check the online documentation

Why use R?

You’ll need R if you want to do any sort of sophisticated (or even not-so-sophisiticated) statistical analysis. There are no solid statistics libraries that I’ve come across for Python . . . but maybe that’s because R is the best possible statistics library there could be.

Be warned however that accessing R from Python can get tricky at times. I’ve tried to outline some of what I’ve learned here to make it easier for others.

Why use RPy instead of writing files out to R, then using R scripts to deal with it? I did this for a little while and found that it was too much work to maintain two separate code bases . . . one for Python, then one for R. If I changed anything in the output of a Python script, I’d have to fire up R and open my R scripts to modify and debug them. I’ve found that using RPy lets me put all my code in one spot, resulting in fewer bugs and less maintenance.

R and Python are separate . . .

I found that the easiest way to think about this is to think about doing things “inside R” or “inside Python”. Things that are to be done inside R are typically wrapped in a string (a Python string). For example, this creates a variable inside R called x with a value of 5.

from rpy import *
r('x=5')

Assuming this was typed into a fresh Python session, Python has no idea about the existence of the variable x! It works in reverse, too: R has no idea about what’s in the Python namespace. So you can do this in Python:

x = 'I'm a Python string'

and the variable x inside R is still the same:

r('print(x)')  # still 5

. . . but they can talk to each other

RPy does some automatic conversions:

x_from_R = r('x')  # 5

What happened here is that RPy looked at what x was inside R, saw that it was an integer, and returned that integer to Python, which assigned it to the Python variable x_from_R. So that’s how you get data from R to Python: by sending a string (the variable name you want to retrieve in R) to the r object.

At first you might think this is how you send data from Python to R:

r('x_from_python') = x
#SyntaxError: can't assign to function call

Nope. Turns out you have to use the r.assign() function to do that:

r.assign('x_from_python', x)
r('print(x_from_python)')  # 'I'm a Python string'

So that’s how you get data from Python to R: by using the r.assign() function, first giving the name of the variable you want to be assigned in R followed by the Python object to be sent to R.

Other data types

OK, so you can get integers back from R. And as you can imagine, strings work the same way. But what about more complex data types? This list of conversions tells you which R objects will be converted into which Python objects. It’s pretty intuitive, a string becomes a string, a list becomes a list, etc.

But then there are things like data frames in R, which have row names and column names.

It’s not on that list linked above, but an R data frame is converted to a Python dictionary. For example, the Motor Trend car data set, which comes standard in R, is a data frame.

from rpy import *
r('print(head(mtcars))') # print just the first 6 lines.  Note the variable names.

# Returns:
#                   mpg cyl disp  hp drat    wt  qsec vs am gear carb
# Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
# Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
# Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
# Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
# Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
# Valiant           18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

Now send the whole thing to Python and check the keys of the dictionary that is created:

mt = r('mtcars')
mt.keys()

Note that the keys are the same as the variable names in the dataframe.

Just like you get a Python dictionary from a dataframe, you can send a dictionary to R:

r.assign('df', dict(a=1, b=2, c=3))
r('print(df)')
r('names(df)')

May have to convert it into a dataframe once inside R though:

r('df = data.frame(df)')

R functions

So far, with the exception of r.assign(), we’ve just been sending strings to the r object. But the r object also has methods. Unfortunately, you can’t see them all using IPython’s introspection. Personally I find that I don’t use this functionality that much, (I use r.assign() to get the data into R and then operate on it in there) but here it is for completeness.

There is a trick here. Remember, before we were sending a string to the r object and it was executing the code inside R:

r('x=5')

But when you use a method of the r object, you pass it raw Python objects. For example, you can plot a Python list in R using the plot() method of the r object:

x = [1,2,3]
r.plot(x)

There are some slight name changes though. R tends to use a “.” as a spacer in function names, like “_” tends to be used in Python. The “.” however is special in Python, so in method names of the r object, “.” is converted to “_”. For example, R’s t.test() function becomes r.t_test().

These methods of the r object are what Python sees, so that’s why their names have to be changed. On the other hand, you call R function with its true name when you send the r object a string, like we were doing before. So both of these refer to the same underlying t-test function in R:

r.t_test
r('t.test')

This next one is tricky. First, since print is a Python function, it needs to have a slightly different name when you want to use the version in R. So an underscore is added to the end. Second, what’s in the parentheses is a Python string. So all that will get printed is the string, ‘x’ . . . not 5, or “I’m a Python string” or anything else.

r.print_('x') # 'x'

In practice though, if I want to print something I’ll either use Python’s print or if I want to print something from R, I’ll do this:

r('print(x)')  # prints 5

Plotting examples

Here’s are a couple of examples of creating a plot. In each case a plot is created of the list 1,2,3. These are trivial examples, but they illustrate different ways of getting data to and from R.

Option 1: Do everything in R

You can execute arbitrary R commands by sending them as a string to the r object. Here, everything is done in R: a list is created and plotted. In this example, the variable x is never seen by Python.

from rpy import *
r("""
y = c(1,2,3)
plot(y)
""")

Note that you can send many R commands in a multi-line string.

Option 2: Use a method of the r object

Here, we start with a Python list, and then send it as the argument to the r.plot() method.

from  rpy import *
y = [1,2,3]
r.plot(y)

Option 3: Get a list from R and plot it with matplotlib in Python

This trivial because you don’t gain anything from making a list in R instead of Python, but it shows that you can send data both ways.

from r import *
import pylab as p
y = r('c(1,2,3)')
p.plot(y)
p.show()

Option 4: Use r.assign() to get data to R, then call it inside R

I tend to use this method a lot with large data sets. The idea is to pass the data into R once, then you can use it from inside R. The trick is to use the r.assign() method.

from rpy import *
y = [1,2,3]
r.assign('Y', y)
r('plot(Y)')

Getting help on R functions

Use the r.help() function. For example, to view the help on anova:

r.help(anova)

This displays the help on screen; it doesn’t return a string.

Non-trivial examples

Plotting and printing things are not what you’d want to use R and RPy for. Instead, you’d want to use them for things that you can’t do in available packages for Python.

Here are some examples where R can really fill in the gaps in Python’s statistical functionality. Anything you can do in R, you can do from Python. Given the wide variety of packages available for R, this is some stupendous power at your fingertips. Now to learn how to wield it!

Linear models in R

Say I have a Python script already up and running, and it returns some data . . . and I want to know if the slope of two variables is significant. I haven’t found any statistics libraries for Python, but in R this kind of functionality comes standard, in the function lm().

Viewing the help for lm(), you can see that it takes a model specification, like “y~x” which means “y on x”. Now, the components of this model specification, y and x, can either refer to variables in the R workspace (which is separate from Python, remember) or they can be variables in a dataframe which is supplied in an optional argument to lm().

So first we need to figure out how to send the data to R; performing the linear regression should be trivial, then we need to get the data back out.

First, let’s set up some test data in Python:

import numpy as npy
x = npy.arange(10)
y = npy.arange(10) + npy.random.standard_normal(x.shape)</pre>

Now send it to R:
<pre>r.assign('x',x)
r.assign('y',y)

(exercise for the reader: instead of assigning x and y individually, how would you get them into R as a dataframe?)

In R, run the linear model and save it as a variable in R. Here, I’m simultaneously saving it as a Python dictionary (sneaky!)

LM = r('linear_model = lm(y~x)')

OK, here’s where it take a little exploring. The dictionary you get back may take some navigating. Looking at it for a little bit, you might notice the ‘coefficients’ key of the dictionary LM, which in turn has two more keys: ‘(Intercept)’ and ‘x’.

{'assign': [0, 1],
 'call': <Robj object at 0xb7d3e790>,
 'coefficients': {'(Intercept)': 0.28490682478866736,
                  'x': 0.86209804871669171},
 'df.residual': 8,
 'effects': array([-13.16882479,   7.83039439,   1.22245056,   0.18398967,
         0.51108108,   0.8141431 ,  -0.45120018,  -1.1985602 ,
         1.54636612,   0.51341949]),
 'fitted.values': array([ 0.28490682,  1.14700487,  2.00910292,  2.87120097,  3.73329902,
        4.59539707,  5.45749512,  6.31959317,  7.18169121,  8.04378926]),
 'model': {'x': array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.]),
           'y': array([-0.64212347,  1.39389811,  3.06676323,  2.84957073,  3.99793052,
        5.12226093,  4.67818603,  4.7520944 ,  8.3182891 ,  8.10661086])},
 'qr': {'pivot': [1, 2],
        'qr': array([[ -3.16227766, -14.23024947],
       [  0.31622777,   9.08295106],
       [  0.31622777,   0.15621147],
       [  0.31622777,   0.0461151 ],
       [  0.31622777,  -0.06398128],
       [  0.31622777,  -0.17407766],
       [  0.31622777,  -0.28417403],
       [  0.31622777,  -0.39427041],
       [  0.31622777,  -0.50436679],
       [  0.31622777,  -0.61446316]]),
        'qraux': [1.316227766016838, 1.2663078500948464],
        'rank': 2,
        'tol': 9.9999999999999995e-08},
 'rank': 2,
 'residuals': array([-0.92703029,  0.24689324,  1.05766031, -0.02163025,  0.2646315 ,
        0.52686386, -0.77930909, -1.56749877,  1.13659789,  0.0628216 ]),
 'terms': <Robj object at 0xb7d3e780>,
 'xlevels': {}}

So if all we were after were the slope and intercept, then

slope = LM['coefficients']['x']
intercept = LM['coefficients']['(Intercept)']

But what about a P-value for the slope? It’s nowhere to be seen in that dictionary. Turns out, you need the summary() function in R, and it takes as its input a linear model (among other possible inputs, but here we’re just using a linear model). So save it in R (just in case) and simultaneously save it in Python:

summary = r('LM_summary = summary(linear_model)')

Hmm.

{'adj.r.squared': 0.88847497651170382,
 'aliased': {'(Intercept)': False, 'x': False},
 'call': <Robj object at 0xb7d3e770>,
 'coefficients': array([[  2.84906825e-01,   5.39776217e-01,   5.27823968e-01,
          6.11943659e-01],
       [  8.62098049e-01,   1.01109349e-01,   8.52639301e+00,
          2.75251311e-05]]),
 'cov.unscaled': array([[ 0.34545455, -0.05454545],
       [-0.05454545,  0.01212121]]),
 'df': [2, 8, 2],
 'fstatistic': {'dendf': 8.0, 'numdf': 1.0, 'value': 72.699377758431851},
 'r.squared': 0.90086664578818121,
 'residuals': array([-0.92703029,  0.24689324,  1.05766031, -0.02163025,  0.2646315 ,
        0.52686386, -0.77930909, -1.56749877,  1.13659789,  0.0628216 ]),
 'sigma': 0.9183712712215929,
 'terms': <Robj object at 0xb7d3e7c0>}

There’s the r-squared and adjusted r-squared,

R_squared = summary['adj.r.squared']

but no P value. What gives? Turns out Python can’t convert everything perfectly, and a little more exploration is in order. Try printing the summary from R:

r('print(LM_summary)')

Well, that makes more sense, and you can see the P value for the slope is 2.75E-5. But how to extract it from Python?

Call:
lm(formula = y ~ x)

Residuals:
    Min      1Q  Median      3Q     Max
-1.5675 -0.5899  0.1549  0.4613  1.1366 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   0.2849     0.5398   0.528    0.612
x             0.8621     0.1011   8.526 2.75e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.9184 on 8 degrees of freedom
Multiple R-squared: 0.9009,	Adjusted R-squared: 0.8885
F-statistic:  72.7 on 1 and 8 DF,  p-value: 2.753e-05

The trick is to match output from the summary printout in R with the dictionary returned to Python. Here, it looks like the key ‘coefficients’ in the summary dictionary in Python gives the numbers in the 2nd row, 3rd column:

P = summary['coefficients'][1,2]

Whew, and there you have it. See, it takes some digging around to get what you need, but now since I’ve done the work for you, you can now do linear regressions from Python. All together it looks like this (can be wrapped in a function or class for your own reuse):

r.assign('x', x)
r.assign('y', y)
LM = r('linear_model = lm(y~x)')
summary = r('summary_LM = summary(linear_model)')
slope = LM['coefficients']['x']
intercept = LM['coefficients']['(Intercept)']
P = summary['coefficients'][1,2]

Redundancy analysis

OK, say you have this data set to perform redundancy analysis (RDA) on. First, you need the package vegan installed, which is fantastic for multivariate stats. It’s probably best to fire up R proper (from a command line, or the GUI if you have it in Windows or OSX) and run

install.packages("vegan", dep=T)

Here’s a heavily commented script, rpy-demo.py, that will:

• load and format the data included in the script
• send the data to R
• perform an RDA in R
• plot the ordination
• save the ordination as a PNG
• print the variance explained by constrained and unconstrained axes as well as each RDA axis.

If you have RPy installed and the vegan package installed, you should be able to just run this Python script.

Often-run analyses that you need R for can be wrapped in a class or module to encapsulate your data analysis needs, so you don’t need to clutter your code with it. Once things are set up that way, it would be as easy as

from myRstuff import lm, rda
results = lm(x,y)
ordination = rda(data)

For much, much more see the online documentation for RPy, but hopefully I gave you enough to at least get started.

By default, the key to sort by is the first letter of each string. Or the first item in a sequence if it’s list of sequences. But Python allows you to specify any key that you want, using the key parameter for the sort() function. The key is the name of a function.

The way it works is this: sort runs every item in the list through the function. Whatever the function returns is used to sort, overriding the default of using the first thing.

Example 1

So to sort a list of strings by the second letter instead of the first letter then we simply need a function that returns the second letter of a string. Here’s such a function, and how to use it as a key to sort. Note that key=secondletter, NOT key=secondletter(). We’re specifying the reference to secondletter, not trying to call it.

def secondletter(x):
    return x[1]

mylist = ['orange', 'banana', 'apple']

mylist.sort(key=secondletter)

# ['banana', 'apple', 'orange']

By the way, for simple one-liner functions like this, we could have used the lambda syntax instead of defining the secondletter function:

mylist = ['orange', 'banana', 'apple']
mylist.sort(key=lambda x: x[1])

Example 2

OK, while it’s a good first example, sorting on the second letter isn’t terribly useful. How about sorting a list of people’s names by their last name? We simply need a function to return the last name, and use the name of that function as the sort key.

def lastname(x):
    firstname, lastname = x.split()
    return lastname

presidents = ['Abraham Lincoln', 'George Washington',
              'Benjamin Harrison', 'Millard Fillmore']

presidents.sort(key=lastname)

#['Millard Fillmore',
# 'Benjamin Harrison',
# 'Abraham Lincoln',
# 'George Washington']

Of course, in practice you would have to be careful with this . . . if there’s a middle name in there, then it would break the lastname function. This one works better:

def lastname2(x):
        return x.split()[-1]

not_all_presidents = ['Abraham Lincoln', 'George Washington', 'Benjamin Harrison',
                     'Millard Fillmore', 'Prince', 'Madonna', 'Arthur C. Clarke']

not_all_presidents.sort(key=lastname2)

#['Arthur C. Clarke',
# 'Millard Fillmore',
# 'Benjamin Harrison',
# 'Abraham Lincoln',
# 'Madonna',
# 'Prince',
# 'George Washington']

…but if you have names like ‘King George III’, you’ll have to fix the function to deal with them.

Example 3

How about sorting a list of stocks by their maximum closing price for this week? (Use reverse=True so that highest are listed first)

stocks = [ [56, 94, 13, 90, 91], [33, 76, 22, 34, 105], [25, 28, 29, 30, 35] ]
stocks.sort(key=max, reverse=True)

# [[33, 76, 22, 34, 105], [56, 94, 13, 90, 91], [25, 28, 29, 30, 35]]

Example 4

You can get tricky…knowing that sort() changes the list in-place, sort individual items by stock price, then sort each stock by its max.

def mymax(x):
    x.sort(reverse=True)
    return x[0]

stocks = [ [56, 94, 13, 90, 91], [33, 76, 22, 34, 105], [25, 28, 29, 30, 35] ]
stocks2 = stocks[:] # make a copy, cause we're about to change it

stocks2.sort(key=mymax, reverse=True)

#[[105, 76, 34, 33, 22], [94, 91, 90, 56, 13], [35, 30, 29, 28, 25]]

Example 5

Or the sort a list by absolute value instead of paying attention to negative signs:

deviations = [10, -34, -5, 90, -87]
deviations.sort(key=abs)
# [-5, 10, -34, -87, 90]

As you can see, specifying the sort key can be pretty useful if you know it’s there. Coming up with these examples really helped me see where this technique would be useful. You can find more info on sorting on the Python wiki, and comparisons between Python and Perl sorting.

In order to use foreign keys, you need to convert your tables to the InnoDB engine:

ALTER TABLE mytable ENGINE = INNODB;

To add a foreign key to mytable where the unique keys are coming from othertable, use this:

ALTER TABLE mytable ADD FOREIGN KEY (otherID) REFERENCES othertable (otherID);

If you run that same line several times, several identical foreign keys will be created, which will confuse SQLAlchemy. In that case you need to delete the keys. To do so, you need their name. To see the name, use

SHOW CREATE TABLE mytable;

The foreign key’s label will be something that ends in something similar to fk_1 or fk_2. Using that label, you can then delete the foreign key:

ALTER TABLE mytable DROP FOREIGN KEY mytable_dbfk_2;

Assumptions

• I’m assuming you have a MySQL database running. More info here
• you have the MySQLdb package installed for Python.
• The database is running on localhost, the user is root, and the password is p@55w0rd.
• You know some SQL (at least enough to appreciate some of these examples)

Caveats

While the code below is specific to MySQLdb, no matter what database API you use you should be able to use the same syntax (as outlined in PEP 249).

For more details, see the official documentation for MySQLdb. Here I’m just trying to explain things slightly differently.

Example usage

Import MySQLdb, and connect to the database

import MySQLdb
my_connection = MySQLdb.connect(host='localhost', user='root', passwd='p@55w0rd')
cursor = my_connection.cursor()

That’s it! You’re ready to start sending SQL statements to your MySQL database!

The cursor is everything!

The cursor now contains all the information it needs to send information to and get information from the running MySQL server. It’s your key to the database.

The two most often-used methods of the MySQLdb cursor are

1. cursor.execute(), which executes a query (but doesn’t return the data)
2. cursor.fetchall(), which fetches the data from the most recently executed query.

You send commands to MySQL by passing strings of SQL statements to cursor.execute(). When doing so, you can take advantage of Python’s multi-line string (delimited by triple quotes (“”")) and the fact that SQL syntax doesn’t care that there are newlines in the query. Furthermore, MySQLdb automatically adds semicolons to the end of SQL statements if you forget them.

Interacting with the database

Create the database and a table

Make a new database by sending the standard SQL query, ‘CREATE DATABASE testdb’, to the database you connected to. Note that MySQLdb automatically adds semicolons to the end of statments if you don’t add them yourself.

cursor.execute('CREATE DATABASE testdb')

If you do this in an interactive session, you will notice that this method returned a long format integer (1L). This is the number of lines returned by the cursor. Don’t worry about it quite yet.

Now make that new database the active one:

cursor.execute('USE testdb')

Now create a table in the testdb database to hold some addresses:

cursor.execute('''CREATE TABLE addresses (
                    name VARCHAR(20),
                    street VARCHAR(20),
                    zipcode INT,
                    city VARCHAR(20),
                    state CHAR(2)
                    )
                    ''')

Note the use of triple quotes so that you can visually organize the SQL query for clarity.

Import data from Python into MySQL

The general syntax for passing Python data to an SQL query through the cursor is:

cursor.execute(SQL,tuple)

where SQL is a valid SQL statement. If SQL has N placeholders of the form %s, then tuple must have length N. Hopefully an example will make more sense.

Let’s create some Python lists that we’ll import into this table. The beauty of it is that these data could have been parsed from a text file with hundreds or thousands of names, and we can import them into the database automatically. For now we’ll just enter three records though.

Here’s the data that will go into the database:

names = ['Bob', 'Alfred', 'Jen']
streets = ['123 Elm Street', '55 Ninth Ave', '1 Paved Rd']
zips = [00123, 34565, 30094]
cities = ['Newark', 'Salinas', 'Los Angeles']
states = ['NJ', 'CA', 'CA']

And here’s how to get that data into the addresses table:

cursor.executemany('''INSERT INTO addresses
                     (name, street, zipcode, city, state)
                     VALUES
                     (%s, %s, %s, %s, %s)''',
                     zip(names, streets, zips, cities, states))

A couple of things to note here:

• This time we used cursor.executemany(), which will accept a list of lists as input, instead of cursor.execute().
• There were 5 fields into which we inserted data (name, street, zipcode, cities, and state)
• There were 5 %s placeholders in the SQL query.
• Even though zipcode is an INT field and not a string, we used %s. This will always be the case: use %s as a placeholder no matter what the datatype.
• There were 5 lists that were zipped together. They need to be zipped so that the result is a list of lists, and the length of each item in the list = 5.
• The order in which these lists were zipped corresponded to the fields into which they were to be inserted.

Retrieving data from the database

There are two steps to retrieving data: executing the query, then fetching the results.

To select all addresses in California, first execute this query (it’s a one-liner so triple quoting isn’t really needed)

cursor.execute("SELECT * FROM addresses WHERE state = 'CA' ")

Alternatively . . . often you will want to feed Python variables into the query. Say the state abbreviation ‘CA’ is saved in a Python variable called my_state. Then this query will do the same thing as the one above:

cursor.execute('''SELECT * FROM addresses WHERE state = %s''', my_state)

By the way, my_state is not a tuple, but that’s OK since there is only one %s placeholder in the query. MySQLdb knows where it should go.

Now to retrieve the results:

results = cursor.fetchall()

Note that a cursor object is similar to a file object or an iterator: once you fetch everything, there is nothing left in the cursor to retrieve. So executing the command above a second time would result in an empty list until the query is executed again.

results is a tuple of tuples and looks like this:

(('Alfred', '55 Ninth Ave', 34565L, 'Salinas', 'CA'),
 ('Jen', '1 Paved Rd', 30094L, 'Los Angeles', 'CA'))

That’s all there is to it! Armed with this knowledge, now you can execute queries from Python to import, retrieve, and plot data from your database. This was a simple demo of what MySQL and Python can do, but you can construct ever-larger databases and ever-more-sophisticated queries to manipulate data in ways that would be impossible without these tools.

You can read about the details of the optparser module here, but here are the basics:

Create an OptionParser object.

from optparse import OptionParser
parser = OptionParser()

Use parser.add_option() to add options that will be passed to the script from the command line. For example,

parser.add_option('--i', action='store', dest='input_file')
(options,args) = parser.parse_args

Each time you add an option, you have to specify:

• the option flag (say, -i or –infile for an input file). One dash for single characters, two dashes for anything longer.
• the action. This is typically store to store the variable, or store_true to use the option as an on-off switch.
• the destination. The option will be stored in an options object (described shortly . .)

There are other things you can add to, like defaults and help strings. See the documentation for more.

To get everything that was sent to the command line into your script, you get tell the OptionParser to get the parse the arguments from the command line like this:

(options,args) = parser.parse_args()

Now options has attributes named after the destination. In the example above, I added the command line option, -i, which will store the argument given right after it in the command line in options.input_file.

So if I called my script from the commandline like this:

python script.py -i mydata.txt

Then in my script, I could access the ‘mydata.txt’ that was provided on the commandline using

options.input_file

Or, to open the file specified on the command line within my script,

f = open(options.input_file)

Here’s a longer example:

import optparser

parser = OptionParser()

parser.add_option("-i",
                         action="store",
                         dest="infile",
                         help="specify the input file")

parser.add_option("-o",
                         action="store",
                         dest="outfile",
                         help="specify the output file")

# This is a boolean (True/False) option.
# I set the default to False; using the --tabs
# option at the command line will make it true.
parser.add_option("--tabs",
                         action="store_true",
                         dest="use_tabs",
                         default=False,
                         help="tell the script to export as tabs")

parser.add_option("--useless",
                         action="store",
                         dest="dummy_variable",
                         help="not used for anything")

(options, args) = parser.parse_args()

# This is how you access the options:

print options.infile
print options.outfile
print options.use_tabs
print options.dummy_variable

Now if you run this at the command prompt:

python test.py -i input.txt -o output.txt

then you’ll see that the options were passed to the script. If you don’t specify an option, its default value is None. You can also set a default value, like I’ve done for the –tabs option above.

Then I ran across doctest. And it is so astoundingly easy to use that I might start writing tests even for one-line scripts.

Here are the three files you need, along with what I’m going to be calling them:

1. some code to test, “code.py”
2. a text file containing documentation for that code, “tutorial.txt”
3. a two line test-runner script, “test.py”

1. code.py

code.py, the stuff I want to test, looks like this:

def addxy(x,y):
    return x+y
def sayHi():
    print "Hi!"

2. tutorial.txt

tutorial.txt contains what to expect when you type certain things at the prompt, along with text. This is plain text.

This tests the module "code.py".

First, make sure that addxy() works:

>>> from code import *
>>> addxy(4,5)
9

Now make sure sayHi() works:

>>> sayHi()
Hi!

3. test.py

test.py contains these two lines. Note that tutorial.txt is referred to in this file, so make sure it points to whatever you called your testing text.

import doctest
doctest.testfile('tutorial.txt')

Now test!

Run test.py to test everything in the tutorial.txt file:

python test.py

In this case, it tests the stuff in code.py. If all goes well, nothing is returned. If you would like some feedback, then use the -v option at the commandline when running test.py.

python test.py -v

Results

Here’s what you get when all tests pass and you use the -v option:

Trying:
    import code
Expecting nothing
ok
Trying:
    code.addxy(4,5)
Expecting:
    9
ok
Trying:
    code.sayHi()
Expecting:
    Hi!
ok
1 items passed all tests:
   3 tests in tutorial.txt
3 tests in 1 items.
3 passed and 0 failed.
Test passed.

To see what happens when a test fails, try changing the “+” in the definition of addxy() to a “-”. When you run the test again using:

python test.py

then you get something like this:

**********************************************************************
File "tutorial.txt", line 3, in tutorial.txt
Failed example:
    code.addxy(4,5)
Expected:
    9
Got:
    -1
**********************************************************************
1 items had failures:
   1 of   3 in tutorial.txt
***Test Failed*** 1 failures.

So it tells you where the testing failed: on line 3 when it tried running code.addxy(4,5). It said it was expecting 9 but instead got -1. After changing that “-” back to a “+”, everything should work again.

First, the ugly way

To show how useful the shelve module is, think for a moment what it would take to save variables as tab-delimited text files:

1. write each list out to a tab-delimited text file
2. copy each list’s text file to the other computer
3. for each list, create an empty list
4. open its file (a way of automatically knowing the filenames would be nice — sequential numbering perhaps?
5. append each line to the new list

Ugh! Now what if I wanted to use two lists and a dictionary on another computer? That would get pretty annoying, pretty quickly.

A better way

Instead, I used the shelve module, which comes standard with Python. The way it works is almost self-explanatory (though maybe I’d call it a cupboard rather than a shelf since you open and close it).

Add stuff to be transported

import shelve

x = shelve.open('my_shelf.dat')

# Add stuff to the shelve object
x['first list'] = list1
x['second list'] = list2
x['a dictionary'] = dictionary1

# close it when you're done.
x.close()

As you can see, the shelve object acts much like a dictionary. It can store arbitrary objects, too. But not that it’s in there, how do you use it?

Use the stuff you added

To use the shelve object you just created, say, on another computer, just copy that single file over (in this case, my_shelf.dat). Then on that other computer, open it up and ask for what was in it. Like so:

import shelve

y = shelve.open('my_shelf.dat')

list_a = y['first list']
list_b = y['second list']
dict_1 = y['a dictionary']

y.close()

# insert code here that uses list_a, list_b, or dict_1 . . .

So the shelve object acts almost like a flat database.

IPython from any script

IPython can be embedded in a script.  Wherever the command ipshell() occurs in your code, an IPython shell will pop up, giving you full access to everything in the namespace of the script.  This is extremely useful if you want to add interactive functionality to a program, or need to check on some variables without using the debugger.

from IPython.Shell import IPShellEmbed 
ipshell = IPShellEmbed()
ipshell()  # add this wherever you would like to open an IPython window

Each ipshell() will open up a new shell.  You have to close the newly opened shell using exit() or Exit before the script will continue.

Checking performance of your code