Note that in order to use R from Python, you need to know a little of both . . . so the learning curve can be steep. You also need to have a feel for what would be easy in R and what would be easy in Python.

There are some detailed examples below if you want to skip right to ‘em.

I use Python for most tasks, but when I need high-powered stats, I embed R code in my Python scripts to perform the analysis.

Disclaimer: I figured all of this stuff out by trial and error. The RPy documentation, while complete, was difficult for me to make sense of when I was learning. If there’s a better way to do things, please let me know! For the details that I don’t cover here, check the online documentation

Why use R?

You’ll need R if you want to do any sort of sophisticated (or even not-so-sophisiticated) statistical analysis. There are no solid statistics libraries that I’ve come across for Python . . . but maybe that’s because R is the best possible statistics library there could be.

Be warned however that accessing R from Python can get tricky at times. I’ve tried to outline some of what I’ve learned here to make it easier for others.

Why use RPy instead of writing files out to R, then using R scripts to deal with it? I did this for a little while and found that it was too much work to maintain two separate code bases . . . one for Python, then one for R. If I changed anything in the output of a Python script, I’d have to fire up R and open my R scripts to modify and debug them. I’ve found that using RPy lets me put all my code in one spot, resulting in fewer bugs and less maintenance.

R and Python are separate . . .

I found that the easiest way to think about this is to think about doing things “inside R” or “inside Python”. Things that are to be done inside R are typically wrapped in a string (a Python string). For example, this creates a variable inside R called x with a value of 5.

from rpy import *
r('x=5')

Assuming this was typed into a fresh Python session, Python has no idea about the existence of the variable x! It works in reverse, too: R has no idea about what’s in the Python namespace. So you can do this in Python:

x = "I'm a Python string"

and the variable x inside R is still the same:

r('print(x)')  # still 5

. . . but they can talk to each other

RPy does some automatic conversions:

x_from_R = r('x')  # 5

What happened here is that RPy looked at what x was inside R, saw that it was an integer, and returned that integer to Python, which assigned it to the Python variable x_from_R. So that’s how you get data from R to Python: by sending a string (the variable name you want to retrieve in R) to the r object.

At first you might think this is how you send data from Python to R:

r('x_from_python') = x
#SyntaxError: can't assign to function call

Nope. Turns out you have to use the r.assign() function to do that:

r.assign('x_from_python', x)
r('print(x_from_python)')  # "I'm a Python string"

So that’s how you get data from Python to R: by using the r.assign() function, first giving the name of the variable you want to be assigned in R followed by the Python object to be sent to R.

Other data types

OK, so you can get integers back from R. And as you can imagine, strings work the same way. But what about more complex data types? This list of conversions tells you which R objects will be converted into which Python objects. It’s pretty intuitive, a string becomes a string, a list becomes a list, etc.

But then there are things like data frames in R, which have row names and column names.

It’s not on that list linked above, but an R data frame is converted to a Python dictionary. For example, the Motor Trend car data set, which comes standard in R, is a data frame.

from rpy import *
r('print(head(mtcars))') # print just the first 6 lines.  Note the variable names.

# Returns:
#                   mpg cyl disp  hp drat    wt  qsec vs am gear carb
# Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
# Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
# Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
# Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1
# Hornet Sportabout 18.7   8  360 175 3.15 3.440 17.02  0  0    3    2
# Valiant           18.1   6  225 105 2.76 3.460 20.22  1  0    3    1

Now send the whole thing to Python and check the keys of the dictionary that is created:

mt = r('mtcars')
mt.keys()

Note that the keys are the same as the variable names in the dataframe.

Just like you get a Python dictionary from a dataframe, you can send a dictionary to R:

r.assign('df', dict(a=1, b=2, c=3))
r('print(df)')
r('names(df)')

May have to convert it into a dataframe once inside R though:

r('df = data.frame(df)')

R functions

So far, with the exception of r.assign(), we’ve just been sending strings to the r object. But the r object also has methods. Unfortunately, you can’t see them all using IPython’s introspection. Personally I find that I don’t use this functionality that much, (I use r.assign() to get the data into R and then operate on it in there) but here it is for completeness.

There is a trick here. Remember, before we were sending a string to the r object and it was executing the code inside R:

r('x=5')

But when you use a method of the r object, you pass it raw Python objects. For example, you can plot a Python list in R using the plot() method of the r object:

x = [1,2,3]
r.plot(x)

There are some slight name changes though. R tends to use a “.” as a spacer in function names, like “_” tends to be used in Python. The “.” however is special in Python, so in method names of the r object, “.” is converted to “_”. For example, R’s t.test() function becomes r.t_test().

These methods of the r object are what Python sees, so that’s why their names have to be changed. On the other hand, you call R function with its true name when you send the r object a string, like we were doing before. So both of these refer to the same underlying t-test function in R:

r.t_test
r('t.test')

This next one is tricky. First, since print is a Python function, it needs to have a slightly different name when you want to use the version in R. So an underscore is added to the end. Second, what’s in the parentheses is a Python string. So all that will get printed is the string, ‘x’ . . . not 5, or “I’m a Python string” or anything else.

r.print_('x') # 'x'

In practice though, if I want to print something I’ll either use Python’s print or if I want to print something from R, I’ll do this:

r('print(x)')  # prints 5

Plotting examples

Here’s are a couple of examples of creating a plot. In each case a plot is created of the list 1,2,3. These are trivial examples, but they illustrate different ways of getting data to and from R.

Option 1: Do everything in R

You can execute arbitrary R commands by sending them as a string to the r object. Here, everything is done in R: a list is created and plotted. In this example, the variable x is never seen by Python.

from rpy import *
r("""
y = c(1,2,3)
plot(y)
""")

Note that you can send many R commands in a multi-line string.

Option 2: Use a method of the r object

Here, we start with a Python list, and then send it as the argument to the r.plot() method.

from  rpy import *
y = [1,2,3]
r.plot(y)

Option 3: Get a list from R and plot it with matplotlib in Python

This trivial because you don’t gain anything from making a list in R instead of Python, but it shows that you can send data both ways.

from r import *
import pylab as p
y = r('c(1,2,3)')
p.plot(y)
p.show()

Option 4: Use r.assign() to get data to R, then call it inside R

I tend to use this method a lot with large data sets. The idea is to pass the data into R once, then you can use it from inside R. The trick is to use the r.assign() method.

from rpy import *
y = [1,2,3]
r.assign('Y', y)
r('plot(Y)')

Getting help on R functions

Use the r.help() function. For example, to view the help on anova:

r.help(anova)

This displays the help on screen; it doesn’t return a string.

Non-trivial examples

Plotting and printing things are not what you’d want to use R and RPy for. Instead, you’d want to use them for things that you can’t do in available packages for Python.

Here are some examples where R can really fill in the gaps in Python’s statistical functionality. Anything you can do in R, you can do from Python. Given the wide variety of packages available for R, this is some stupendous power at your fingertips. Now to learn how to wield it!

Linear models in R

Say I have a Python script already up and running, and it returns some data . . . and I want to know if the slope of two variables is significant. I haven’t found any statistics libraries for Python, but in R this kind of functionality comes standard, in the function lm().

Viewing the help for lm(), you can see that it takes a model specification, like “y~x” which means “y on x”. Now, the components of this model specification, y and x, can either refer to variables in the R workspace (which is separate from Python, remember) or they can be variables in a dataframe which is supplied in an optional argument to lm().

So first we need to figure out how to send the data to R; performing the linear regression should be trivial, then we need to get the data back out.

First, let’s set up some test data in Python:

import numpy as npy
x = npy.arange(10)
y = npy.arange(10) + npy.random.standard_normal(x.shape)

Now send it to R:

r.assign('x',x)
r.assign('y',y)

(exercise for the reader: instead of assigning x and y individually, how would you get them into R as a dataframe?)

In R, run the linear model and save it as a variable in R. Here, I’m simultaneously saving it as a Python dictionary (sneaky!)

LM = r('linear_model = lm(y~x)')

OK, here’s where it take a little exploring. The dictionary you get back may take some navigating. Looking at it for a little bit, you might notice the ‘coefficients’ key of the dictionary LM, which in turn has two more keys: ‘(Intercept)’ and ‘x’.

{'assign': [0, 1],
 'call': ,
 'coefficients': {'(Intercept)': 0.28490682478866736,
                  'x': 0.86209804871669171},
 'df.residual': 8,
 'effects': array([-13.16882479,   7.83039439,   1.22245056,   0.18398967,
         0.51108108,   0.8141431 ,  -0.45120018,  -1.1985602 ,
         1.54636612,   0.51341949]),
 'fitted.values': array([ 0.28490682,  1.14700487,  2.00910292,  2.87120097,  3.73329902,
        4.59539707,  5.45749512,  6.31959317,  7.18169121,  8.04378926]),
 'model': {'x': array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.]),
           'y': array([-0.64212347,  1.39389811,  3.06676323,  2.84957073,  3.99793052,
        5.12226093,  4.67818603,  4.7520944 ,  8.3182891 ,  8.10661086])},
 'qr': {'pivot': [1, 2],
        'qr': array([[ -3.16227766, -14.23024947],
       [  0.31622777,   9.08295106],
       [  0.31622777,   0.15621147],
       [  0.31622777,   0.0461151 ],
       [  0.31622777,  -0.06398128],
       [  0.31622777,  -0.17407766],
       [  0.31622777,  -0.28417403],
       [  0.31622777,  -0.39427041],
       [  0.31622777,  -0.50436679],
       [  0.31622777,  -0.61446316]]),
        'qraux': [1.316227766016838, 1.2663078500948464],
        'rank': 2,
        'tol': 9.9999999999999995e-08},
 'rank': 2,
 'residuals': array([-0.92703029,  0.24689324,  1.05766031, -0.02163025,  0.2646315 ,
        0.52686386, -0.77930909, -1.56749877,  1.13659789,  0.0628216 ]),
 'terms': ,
 'xlevels': {}}

So if all we were after were the slope and intercept, then

slope = LM['coefficients']['x']
intercept = LM['coefficients']['(Intercept)']

But what about a P-value for the slope? It’s nowhere to be seen in that dictionary. Turns out, you need the summary() function in R, and it takes as its input a linear model (among other possible inputs, but here we’re just using a linear model). So save it in R (just in case) and simultaneously save it in Python:

summary = r('LM_summary = summary(linear_model)')

Hmm.

{'adj.r.squared': 0.88847497651170382,
 'aliased': {'(Intercept)': False, 'x': False},
 'call': ,
 'coefficients': array([[  2.84906825e-01,   5.39776217e-01,   5.27823968e-01,
          6.11943659e-01],
       [  8.62098049e-01,   1.01109349e-01,   8.52639301e+00,
          2.75251311e-05]]),
 'cov.unscaled': array([[ 0.34545455, -0.05454545],
       [-0.05454545,  0.01212121]]),
 'df': [2, 8, 2],
 'fstatistic': {'dendf': 8.0, 'numdf': 1.0, 'value': 72.699377758431851},
 'r.squared': 0.90086664578818121,
 'residuals': array([-0.92703029,  0.24689324,  1.05766031, -0.02163025,  0.2646315 ,
        0.52686386, -0.77930909, -1.56749877,  1.13659789,  0.0628216 ]),
 'sigma': 0.9183712712215929,
 'terms': }

There’s the r-squared and adjusted r-squared,

R_squared = summary['adj.r.squared']

but no P value. What gives? Turns out Python can’t convert everything perfectly, and a little more exploration is in order. Try printing the summary from R:

r('print(LM_summary)')

Well, that makes more sense, and you can see the P value for the slope is 2.75E-5. But how to extract it from Python?

Call:
lm(formula = y ~ x)

Residuals:
    Min      1Q  Median      3Q     Max
-1.5675 -0.5899  0.1549  0.4613  1.1366 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)
(Intercept)   0.2849     0.5398   0.528    0.612
x             0.8621     0.1011   8.526 2.75e-05 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 

Residual standard error: 0.9184 on 8 degrees of freedom
Multiple R-squared: 0.9009,	Adjusted R-squared: 0.8885
F-statistic:  72.7 on 1 and 8 DF,  p-value: 2.753e-05

The trick is to match output from the summary printout in R with the dictionary returned to Python. Here, it looks like the key ‘coefficients’ in the summary dictionary in Python gives the numbers in the 2nd row, 3rd column:

P = summary['coefficients'][1,2]

Whew, and there you have it. See, it takes some digging around to get what you need, but now since I’ve done the work for you, you can now do linear regressions from Python. All together it looks like this (can be wrapped in a function or class for your own reuse):

r.assign('x', x)
r.assign('y', y)
LM = r('linear_model = lm(y~x)')
summary = r('summary_LM = summary(linear_model)')
slope = LM['coefficients']['x']
intercept = LM['coefficients']['(Intercept)']
P = summary['coefficients'][1,2]

Redundancy analysis

OK, say you have this data set to perform redundancy analysis (RDA) on. First, you need the package vegan installed, which is fantastic for multivariate stats. It’s probably best to fire up R proper (from a command line, or the GUI if you have it in Windows or OSX) and run

install.packages("vegan", dep=T)

Here’s a heavily commented script, rpy-demo.py, that will:

• load and format the data included in the script
• send the data to R
• perform an RDA in R
• plot the ordination
• save the ordination as a PNG
• print the variance explained by constrained and unconstrained axes as well as each RDA axis.

If you have RPy installed and the vegan package installed, you should be able to just run this Python script.

Often-run analyses that you need R for can be wrapped in a class or module to encapsulate your data analysis needs, so you don’t need to clutter your code with it. Once things are set up that way, it would be as easy as

from myRstuff import lm, rda
results = lm(x,y)
ordination = rda(data)

For much, much more see the online documentation for RPy, but hopefully I gave you enough to at least get started.

The trick is just to specify that you want polar coordinates when you create the axis. Then create a bar plot as normal.

from matplotlib.pyplot import figure, show
from math import pi

fig = figure()
ax = fig.add_subplot(111, polar=True)
x = [30,60,90,120,150,180]
x = [i*pi/180 for i in x]  # convert to radians

ax.bar(x,[1,2,3,4,5,6], width=0.4)
show()

Note that in the above example the “right” or “clockwise-most” edge is lined up with each specified x value. You can change this by subtracting width / 2 to each of the x values to center the bars on the x-values, like this:

from matplotlib.pyplot import figure, show
from math import pi

width = 0.4  # width of the bars (in radians)

fig = figure()
ax = fig.add_subplot(111, polar=True)
x = [30,60,90,120,150,180]

# Convert to radians and subtract half the width
# of a bar to center it.
x = [i*pi/180 - width/2 for i in x]
ax.bar(x,[1,2,3,4,5,6], width=width)
show()

Get funky . . .

The following is slightly modifed from the matplotlib examples:

import numpy as npy
import matplotlib.cm as cm
from matplotlib.pyplot import figure, show, rc

# force square figure and square axes (looks better for polar, IMHO)
fig = figure(figsize=(8,8))
ax = fig.add_axes([0.1, 0.1, 0.8, 0.8], polar=True)

N = 20
theta = npy.arange(0.0, 2*npy.pi, 2*npy.pi/N)  # random angles
radii = 10*npy.random.rand(N)  # random bar heights
width = npy.pi/4*npy.random.rand(N) # random widths

# Create the bar plot
bars = ax.bar(theta, radii, width=width, bottom=0.0)

# Step through bars (a list of Rectangle objects) and
# change color based on its height and set its alpha transparency
# to 0.5

for r,bar in zip(radii, bars):
    bar.set_facecolor( cm.jet(r/10.))
    bar.set_alpha(0.5)

show()

The tools

• ffmpeg

  (On Ubuntu: sudo apt-get install ffmpeg)

• gtk-recordmydesktop

  (On Ubuntu: sudo apt-get install gtk-recordmydesktop)

• Jeroen Wijering’s embedded Flash player wizard (brilliant!)

The process

1. Record a screencast

Fire up gtk-recordmydesktop. The screen looks like this:

Optionally press “Select Window” and click on a window to select it for recording.

Press record, and do your thing.

By default, it saves the file as out.ogg in your home directory. (more info on the Ogg format).

2. Convert the .ogg to .flv

You need to convert this into a Flash .flv format. Easiest way is to use ffmpeg:

ffmpeg -i out.ogg out.flv

or, optionally resize so that the output file will be 320×250:

ffmpeg -i out.ogg -s 320x250 out.flv

3. Upload the .flv to your site

Upload the new .flv to your site (and remember where you put it).

4. Use the embedded Flash player wizard

Then go to Jeroen Wijering’s embedded Flash player wizard. Paste the link to your newly uploaded .flv.

Double check that the file shows up in the preview of the wizard. When you’re happy with it, copy the source code shown in the wizard and paste it in your site.

A note on embedding in WordPress

Copying and pasting the code into the edit window for a post did not work properly at first in WordPress (even in the non-visual editor). The source code showed up as plain text. To fix this, I had to delete all the newlines so that the embed tag was all on one line.

from pylab import figure, show

# Create some data: one x, three different y
x = arange(0.0, 20.0, 0.01)
y1 = sin(2*pi*x)
y2 = exp(-x)
y3 = y1*y2

# Create a figure and add three subplots.
fig = figure()
ax1 = fig.add_subplot(311)
ax2 = fig.add_subplot(312, sharex=ax1)  # share ax1's xaxis
ax3 = fig.add_subplot(313, sharex=ax1)  # share ax1's xaxis

# Plot
ax1.plot(x,y1)
ax2.plot(x,y2)
ax3.plot(x,y3)

# Show the figure.
show()

Here’s a video of what the interaction is like with this figure (matplotlib automatically adds the pan, zoom, home, etc buttons to all figures):

Notice that the y-axis remained independent in each of the three subplots. As you’d expect, the add_subplot() method accepts a sharey keyword argument as well. You can even pass both sharex and sharey . . . this is most useful when two subplots show data with the same units.

I used it recently to calculate a year’s worth of sunrise and sunset times. Using something as advanced as PyEphem for something this astronomically simple might be overkill, but it works well. Here’s how I did it:

import ephem
import datetime

obs = ephem.Observer()
obs.lat = '38.8'
obs.long= '-75.2'

start_date = datetime.datetime(2008, 1, 1)
end_date = datetime.datetime(2008, 12, 31)
td = datetime.timedelta(days=1)

sun = ephem.Sun()

sunrises = []
sunsets = []
dates = []

date = start_date
while date < end_date:
    date += td
    dates.append(date)
    obs.date = date

    rise_time = obs.next_rising(sun).datetime()
    sunrises.append(rise_time)

    set_time = obs.next_setting(sun).datetime()
    sunsets.append(set_time)

To plot day length in hours over the course of a year, first run the above code. Then (assuming you have matplotlib):

from pylab import *
daylens = []
for i in range(len(sunrises)):
    timediff = sunsets[i] - sunrises[i]
    hours = timediff.seconds / 60. / 60.  # to get it in hours
    daylens.append(hours)

plot(dates, daylens)

# if you have an older version of matplotlib, you may need
# to convert dates into numbers before plotting:
# dates = [date2num(i) for i in dates]

xlabel('Date')
ylabel('Hours')
title('Day length in 2008')
show()

Installation

It is really, really easy.

If you have easy_install on your machine, running

easy_install sphinx

from the command line will do it.

If you don’t have easy_install yet, download and install it, then run easy_install sphinx from the command line.

Run sphinx-quickstart to automatically setup a directory structure

Navigate to the directory that you want to generate documentation for. Run sphinx-quickstart from the command line. You will get a series of questions that lead you through the process of setting up the directories and some files that Sphinx needs. Most questions have defaults that you can just accept no prob. When it comes to version name, you can just make something up.

If you accepted all the defaults (you conformist, you!) there are now a couple of new files and directories:

New directory contents:
/.build
/.templates
/.static
index.rst
conf.py

In a moment we’ll look at index.rst and conf.py. But first . . .

Create some content

Time to make some content. All content is created in plain text files. Create a new text file in the same directory as index.rst and conf.py. Call it chapter1.rst or something . . . what’s important though is that you give it the “.rst” extension (if you chose the default option for “Source file suffix” when you ran sphinx-build, otherwise the extension must be whatever you chose in that step). Sphinx will recognize files that should be included based on their extension. Since I tend to have random, non-documentation .txt files laying around which Sphinx will ask me about, I prefer to use .rst instead of .txt.

Inside chapter1.rst, type some stuff. You could be documenting the code that’s in the current directory, for example. Here are some formatting elements you can try for now. The Sphinx site has lots more info on formatting.

This is a header
================
Some text, *italic text*, **bold text**

* bulleted list.  There needs to be a space right after the "*"
* item 2

.. note::
    This is a note.

Here's some Python code:

>>> for i in range(10):
...     print i

Once Sphinx is set up, it will read in the plain text above and convert it into some nicely formatted HTML. The results of the above plain text will look something like this:

But first you have to run it through Sphinx in order for it to look that way, and before you can do that you have to tell Sphinx to include this document in its processing.

Tell index.rst what files to include

Now we have to edit index.rst to tell Sphinx that chapter1.rst should be included in the documentation. By default, index.rst looks like this:

Change this . . .

Welcome to my project's documentation!
======================================

Contents:

.. toctree::
   :maxdepth: 2

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

To this . . .

(Just add the chapter1.rst line)

Welcome to my project's documentation!
======================================

Contents:

.. toctree::
    :maxdepth: 2

    chapter1.rst

Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

The indentation of the chapter1.rst line is important. So is the blank line above, and the blank line below.

BE CAREFUL . . . this burned me the first time, and took a while to figure out. See the :maxdepth: 2 line? It’s only indented 3 spaces. It doesn’t have to be 3 spaces, it just happens to be in this auto-generated file. If you have your text editor set for 4 spaces, or anything other than 3 spaces, when you hit tab on another line your indentation will be inconsistent, resulting in errors.

Usually I just add a space to make the :maxdepth: 2 line a 4-space indentation.

Build the documentation

Back at the command line, make a new directory to hold your documentation. I’m going to call mine doc.

mkdir doc

Then, in the same directory as index.rst, run

sphinx-build . doc

You’ll get some output that tells you what it’s doing.

View the result

Now you can open up doc/index.html to view your newly generated documentation. It looks something like this:

Clicking on the “This is a header” link shows you the content you added in chapter1.rst, and it looks something like this:

So what’s the big deal?

The above example could have been done with a little work in Microsoft Word. Where Sphinx really shines, though, is when you use it to document Python code.

Sphinx can auto-document by reading in a module, then displaying the docstrings of objects in that source code. It hyperlinks modules, classes, methods, attributes, etc. It can even take the code you write as a tutorial and use it as doctests, which test your code to ensure that it is correct. With a little more setup, you can generate LaTeX (and then PDF files) of your code, along with the good-looking syntax highlighting. (see upcoming posts for how to do all of this)

I feel that other documentation tools, like epydoc, have too much of an auto-generated look, and it’s too easy to include extraneous content. In order to include tutorial info in epydoc documents, you have to add text to the beginning of a module. While epydoc has some nice formatting (allowing you to tag text as parameters or return values), it tends to look ugly–sometimes to the point of unreadability.

I like Sphinx because it allows you to insert auto-generated documentation when you need it but tends to focus on high-quality, handwritten, tutorial-style content which I think is the most effective kind of documentation.

Try this Python script, addtiddler.py, to insert tiddlers into an existing TiddlyWiki. You can optionally specify an image name (relative to the output file, see the documentation in the source code) to be embedded. You can use this script from the command line using options, or import it into another script.

I tried to add lots of comments so you can modify it for your own needs. Let me know if you find bugs so I can fix them.

By default, the key to sort by is the first letter of each string. Or the first item in a sequence if it’s list of sequences. But Python allows you to specify any key that you want, using the key parameter for the sort() function. The key is the name of a function.

The way it works is this: sort runs every item in the list through the function. Whatever the function returns is used to sort, overriding the default of using the first thing.

Example 1

So to sort a list of strings by the second letter instead of the first letter then we simply need a function that returns the second letter of a string. Here’s such a function, and how to use it as a key to sort. Note that key=secondletter, NOT key=secondletter(). We’re specifying the reference to secondletter, not trying to call it.

def secondletter(x):
    return x[1]

mylist = ['orange', 'banana', 'apple']

mylist.sort(key=secondletter)

# ['banana', 'apple', 'orange']

By the way, for simple one-liner functions like this, we could have used the lambda syntax instead of defining the secondletter function:

mylist = ['orange', 'banana', 'apple']
mylist.sort(key=lambda x: x[1])

Example 2

OK, while it’s a good first example, sorting on the second letter isn’t terribly useful. How about sorting a list of people’s names by their last name? We simply need a function to return the last name, and use the name of that function as the sort key.

def lastname(x):
    firstname, lastname = x.split()
    return lastname

presidents = ['Abraham Lincoln', 'George Washington',
              'Benjamin Harrison', 'Millard Fillmore']

presidents.sort(key=lastname)

#['Millard Fillmore',
# 'Benjamin Harrison',
# 'Abraham Lincoln',
# 'George Washington']

Of course, in practice you would have to be careful with this . . . if there’s a middle name in there, then it would break the lastname function. This one works better:

def lastname2(x):
        return x.split()[-1]

not_all_presidents = ['Abraham Lincoln', 'George Washington', 'Benjamin Harrison',
                     'Millard Fillmore', 'Prince', 'Madonna', 'Arthur C. Clarke']

not_all_presidents.sort(key=lastname2)

#['Arthur C. Clarke',
# 'Millard Fillmore',
# 'Benjamin Harrison',
# 'Abraham Lincoln',
# 'Madonna',
# 'Prince',
# 'George Washington']

…but if you have names like ‘King George III’, you’ll have to fix the function to deal with them.

Example 3

How about sorting a list of stocks by their maximum closing price for this week? (Use reverse=True so that highest are listed first)

stocks = [ [56, 94, 13, 90, 91], [33, 76, 22, 34, 105], [25, 28, 29, 30, 35] ]
stocks.sort(key=max, reverse=True)

# [[33, 76, 22, 34, 105], [56, 94, 13, 90, 91], [25, 28, 29, 30, 35]]

Example 4

You can get tricky…knowing that sort() changes the list in-place, sort individual items by stock price, then sort each stock by its max.

def mymax(x):
    x.sort(reverse=True)
    return x[0]

stocks = [ [56, 94, 13, 90, 91], [33, 76, 22, 34, 105], [25, 28, 29, 30, 35] ]
stocks2 = stocks[:] # make a copy, cause we're about to change it

stocks2.sort(key=mymax, reverse=True)

#[[105, 76, 34, 33, 22], [94, 91, 90, 56, 13], [35, 30, 29, 28, 25]]

Example 5

Or the sort a list by absolute value instead of paying attention to negative signs:

deviations = [10, -34, -5, 90, -87]
deviations.sort(key=abs)
# [-5, 10, -34, -87, 90]

As you can see, specifying the sort key can be pretty useful if you know it’s there. Coming up with these examples really helped me see where this technique would be useful. You can find more info on sorting on the Python wiki, and comparisons between Python and Perl sorting.

Recently I posted a couple of different ways to sort one list by another list. But how to decide which one is faster?

I like using the timeit magic function in IPython to time Python code. It takes, as its argument, a single Python expression. timeit then returns the time it took the CPU to run that code. Problem is, timeit takes a single expression, but those sorting methods are multi-line statements.

No matter, we just wrap them in a function, and call the function as a single expression. Here are the functions I’ll be testing:

import numpy
def method1(x,y):
    z = zip(x,y)
    z.sort()
    return zip(*z)

def method2(x,y):
    inds = numpy.argsort(x)
    return numpy.take(y,inds)

def method3(x,y):
    xa = numpy.array(x)
    ya = numpy.array(y)
    inds = xa.argsort()
    return ya[inds]

OK, we need some lists to use for the test sorting. How about the ones used from the previously mentioned post:

people = ['Jim', 'Pam', 'Micheal', 'Dwight']
ages = [27, 25, 4, 9]

And now, to test each method. If it’s a really fast bit of code, timeit will run it many times (here, 100,000 or 10,000 times) to get a good estimate of how long it takes.

timeit method1(ages,people)  # 100000 loops, best of 3: 3.6 µs per loop
timeit method2(ages,people)  # 10000 loops, best of 3: 63.7 µs per loop
timeit method3(ages,people)  # 10000 loops, best of 3: 36.8 µs per loop

These results suggest that the first method is an order of magnitude faster. But wait a minute, method3() converts the input lists into arrays . . . I bet that takes some time. How fast does it run if the data are already in an array? Time for another function. This one expects its arguments to be arrays already.

def method3a(x,y):
    inds = x.argsort()
    return y[inds]

And let’s convert ages and people into arrays ahead of time so we can use method3a:

array_ages = numpy.array(ages)
array_people = numpy.array(people)

The other methods still ought to run when the data are arrays. Here are the results on my machine for all the methods so far:

timeit method1(array_ages, array_people)   # 100000 loops, best of 3: 6.29 µs per loop
timeit method2(array_ages, array_people)   # 100000 loops, best of 3: 6.88 µs per loop
timeit method3(array_ages, array_people)   # 100000 loops, best of 3: 5.12 µs per loop
timeit method3a(array_ages, array_people)  # 100000 loops, best of 3: 4.02 µs per loop

So it looks like if the data are already in an array form, method3a looks like the fastest.

Let’s see if the results are consistent with a larger dataset, where you might actually perceive a difference in speed.

#The test arrays
xa = numpy.random.random(100000)
ya = numpy.random.random(100000)

# The test arrays converted into test lists
xlist = xa.tolist()
ylist = ya.tolist()

# Test the speed of sorting arrays
timeit method1(xa,ya)  # 10 loops, best of 3: 443 ms per loop
timeit method2(xa,ya)  # 10 loops, best of 3: 20.6 ms per loop
timeit method3(xa,ya)  # 10 loops, best of 3: 18.9 ms per loop
timeit method3a(xa,ya) # 10 loops, best of 3: 19.1 ms per loop

# Test the speed of sorting lists
timeit method1(xlist,ylist)  # 10 loops, best of 3: 391 ms per loop
timeit method2(xlist,ylist)  # 10 loops, best of 3: 51.3 ms per loop
timeit method3(xlist,ylist)  # 10 loops, best of 3: 55.1 ms per loop
# (can't test method3a since it won't accept lists)

Interesting. When your data is already in an array, method3 (which converts x and y into arrays) is actually faster than method3a (which assumes they are already arrays)! Not by much, but I wouldn’t have expected that two extra lines of code would actually make it faster.

The final results show that the answer depends on what form your data are in:

• For small lists, use method1.
• For large lists, use method2.
• For small arrays, use method3a.
• For large arrays, use method3.

In each case imagine we want to sort a list of peoples names by their ages.

Method 1

Zip the lists together, making sure that the one to sort by is passed first to zip(). The result is a list of tuples. When you sort a list of tuples, it sorts using the first item in each tuple. Then use the zip* trick to unzip the now sorted tuples into separate variables.

people = ['Jim', 'Pam', 'Micheal', 'Dwight']
ages = [27, 25, 4, 9]

agesAndPeople = zip(ages, people)
agesAndPeople.sort()
sortedAges, sortedPeople = zip(*agesAndPeople)

Note that if you want to sort in reverse, you can use agesAndPeople.sort(reverse=True).

Method 2

This method uses NumPy, and you don’t have to convert the lists into arrays. The argsort() function doesn’t return the sorted ages . . . instead, it returns the indices that each item would if it were in an already sorted array (try it to see what I mean). take() is a way of using useful NumPy indexing on a list. See the next example for something that might be more straigtforward for Matlab users.

people = ['Jim', 'Pam', 'Micheal', 'Dwight']
ages = [27, 25, 4, 9]

import numpy
inds = numpy.argsort(ages)
sortedPeople = numpy.take(people, inds)

Method 3

This method also uses NumPy, but first it converts the lists into arrays. Then it uses the argsort() of one to index into the other.

people = ['Jim', 'Pam', 'Micheal', 'Dwight']
ages = [27, 25, 4, 9]

import numpy
people = numpy.array(people)
ages = numpy.array(ages)
inds = ages.argsort()
sortedPeople = people[inds]