The tools

• ffmpeg

  (On Ubuntu: sudo apt-get install ffmpeg)

• gtk-recordmydesktop

  (On Ubuntu: sudo apt-get install gtk-recordmydesktop)

• Jeroen Wijering’s embedded Flash player wizard (brilliant!)

The process

1. Record a screencast

Fire up gtk-recordmydesktop. The screen looks like this:

Optionally press “Select Window” and click on a window to select it for recording.

Press record, and do your thing.

By default, it saves the file as out.ogg in your home directory. (more info on the Ogg format).

2. Convert the .ogg to .flv

You need to convert this into a Flash .flv format. Easiest way is to use ffmpeg:

ffmpeg -i out.ogg out.flv

or, optionally resize so that the output file will be 320×250:

ffmpeg -i out.ogg -s 320x250 out.flv

3. Upload the .flv to your site

Upload the new .flv to your site (and remember where you put it).

4. Use the embedded Flash player wizard

Then go to Jeroen Wijering’s embedded Flash player wizard. Paste the link to your newly uploaded .flv.

Double check that the file shows up in the preview of the wizard. When you’re happy with it, copy the source code shown in the wizard and paste it in your site.

A note on embedding in WordPress

Copying and pasting the code into the edit window for a post did not work properly at first in WordPress (even in the non-visual editor). The source code showed up as plain text. To fix this, I had to delete all the newlines so that the embed tag was all on one line.

I’ve been doing quite a bit of code documentation lately, and I decided to try and figure out the best tool to use. I found it. It’s called Sphinx, and you can see what the documentation looks like by checking out the documentation for Python itself (v. 2.6 and 3.0).
Here’s how to get started using Sphinx.
Sphinx is the name of the program that takes your plain text files and converts them into hyperlinked, nicely formatted HTML documents. It knows what to link and how to format based on simple formatting commands in the text files (specifically, it uses the ReStructured Text (ReST) markup language).

Installation

It is really, really easy.

If you have easy_install on your machine, running

easy_install sphinx -U

from the command line will do it.

If you don’t have easy_install yet, download and install it, then run easy_install sphinx from the command line.

Run sphinx-quickstart to automatically setup a directory structure

Navigate to the directory that you want to generate documentation for. Run sphinx-quickstart from the command line. You will get a series of questions that lead you through the process of setting up the directories and some files that Sphinx needs. Most questions have defaults that you can just accept no prob. When it comes to version name, you can just make something up.

If you accepted all the defaults (you conformist, you!) there are now a couple of new files and directories:

New directory contents:
/.build
/.templates
/.static
index.rst
conf.py

In a moment we’ll look at index.rst and conf.py. But first . . .

Create some content

Time to make some content. All content is created in plain text files. Create a new text file in the same directory as index.rst and conf.py. Call it chapter1.rst or something . . . what’s important though is that you give it the “.rst” extension (if you chose the default option for “Source file suffix” when you ran sphinx-build, otherwise the extension must be whatever you chose in that step). Sphinx will recognize files that should be included based on their extension. Since I tend to have random, non-documentation .txt files laying around which Sphinx will ask me about, I prefer to use .rst instead of .txt.

Inside chapter1.rst, type some stuff. You could be documenting the code that’s in the current directory, for example. Here are some formatting elements you can try for now. The Sphinx site has lots more info on formatting.

This is a header
================
Some text, *italic text*, **bold text**

* bulleted list.  There needs to be a space right after the "*"
* item 2

.. note::
    This is a note.

Here's some Python code:

>>> for i in range(10):
...     print i

Once Sphinx is set up, it will read in the plain text above and convert it into some nicely formatted HTML. The results of the above plain text will look something like this:

But first you have to run it through Sphinx in order for it to look that way, and before you can do that you have to tell Sphinx to include this document in its processing.

Tell index.rst what files to include

Now we have to edit index.rst to tell Sphinx that chapter1.rst should be included in the documentation. By default, index.rst looks like this:

Change this . . .

Welcome to my project's documentation!
======================================

Contents:

.. toctree::
   :maxdepth: 2

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

To this . . .

(Just add the chapter1.rst line)

Welcome to my project's documentation!
======================================

Contents:

.. toctree::
    :maxdepth: 2

    chapter1.rst

Indices and tables
==================
* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

The indentation of the chapter1.rst line is important. So is the blank line above, and the blank line below.

BE CAREFUL . . . this burned me the first time, and took a while to figure out. See the :maxdepth: 2 line? It’s only indented 3 spaces. It doesn’t have to be 3 spaces, it just happens to be in this auto-generated file. If you have your text editor set for 4 spaces, or anything other than 3 spaces, when you hit tab on another line your indentation will be inconsistent, resulting in errors.

Usually I just add a space to make the :maxdepth: 2 line a 4-space indentation.

Build the documentation

Back at the command line, make a new directory to hold your documentation. I’m going to call mine doc.

mkdir doc

Then, in the same directory as index.rst, run

sphinx-build . doc

You’ll get some output that tells you what it’s doing.

View the result

Now you can open up doc/index.html to view your newly generated documentation. It looks something like this:

Clicking on the “This is a header” link shows you the content you added in chapter1.rst, and it looks something like this:

So what’s the big deal?

The above example could have been done with a little work in Microsoft Word. Where Sphinx really shines, though, is when you use it to document Python code.

Sphinx can auto-document by reading in a module, then displaying the docstrings of objects in that source code. It hyperlinks modules, classes, methods, attributes, etc. It can even take the code you write as a tutorial and use it as doctests, which test your code to ensure that it is correct. With a little more setup, you can generate LaTeX (and then PDF files) of your code, along with the good-looking syntax highlighting. (see upcoming posts for how to do all of this)

I feel that other documentation tools, like epydoc, have too much of an auto-generated look, and it’s too easy to include extraneous content. In order to include tutorial info in epydoc documents, you have to add text to the beginning of a module. While epydoc has some nice formatting (allowing you to tag text as parameters or return values), it tends to look ugly–sometimes to the point of unreadability.

I like Sphinx because it allows you to insert auto-generated documentation when you need it but tends to focus on high-quality, handwritten, tutorial-style content which I think is the most effective kind of documentation.

SSH in.

ssh user@hostname.com

Once on the remote machine, set up a named screen:

screen -S myscreen

In another terminal, open another SSH connnection and start another screen:

screen -S mysecondscreen

You can see they are there by using, in either of the terminals,

screen -ls

This does NOT start screen, just lists the different screens.

You can now disconnect the SSH connections. When you reconnect, you can use

screen -r myscreen

or

screen -r mysecondscreen

to reconnect to the one you want.

sudo apt-get install subversion

Make a directory to store the repositories

mkdir /path/to/repository

Create the repository

svnadmin create /path/to/repository

Import existing files into repository

svn import /path/name/to/foo file:///path/to/repository

when you checkout this repository, it will create the directory foo. So to get the svn repository in my ~/work directory as ~/work/foo I would go to ~/work, then

svn co file:///path/to/repository

that is, don’t make a new dir dir called foo and import into there . . . it will make its own dir.

Check out locally

svn checkout file:///path/to/repository /local/workdir

Check out remotely through an ssh connection

svn checkout svn+ssh://user@hostname/path/to/repository/on/remotehost /local/workdir

Update local copy from SVN

svn update

Check what’s been changed

svn status

Resolve a conflict

svn resolved filename

Send these changes to SVN (editor will prompt for revision notes, must be non-empty)

svn commit

Send these changes to SVN, and specify logfile to send as comments

svn commit -F logfile

Edit /etc/ssh/sshd_config as root.

• Change the port (default is 22)
• Change “PermitRootLogin yes” to “PermitRootLogin no”
• AddUser username
• save and quit
• restart the ssh server: sudo /etc/init.d/ssh restart

More info here:
http://ubuntu-tutorials.com/2007/02/14/what-you-ought-to-know-about-securing-ssh/

ssh-copy-id -i ~/.ssh/id_dsa.pub root@fileserver01

More good info here:
http://ubuntu-tutorials.com/2007/02/05/unattended-ssh-login-public-key-ssh-authorization-ssh-automatic-login/

First, forward the local port 3307 to 3306. That is, when you access the local port 3307, it will redirect it to port 3306 on the remote host.

ssh -fNg4 -L 3307:127.0.0.1:3306 user@hostname

-f sends SSH to the background
-g allows remote hosts to connect to local forwarded ports
-N don’t execute a remote command
-4 this was key! Forces IPv4. Kept getting “bind: Address already in use” errors because I didn’t have this.
-L the forwarding magic happens here . . . syntax is localport:localhost:remoteport

Step 2: Connect to mysql on port 3307

. . . which will redirect to port 3306 on remote host.

mysql -u root -h 127.0.0.1 -P 3307 -p

and you’re in!

Use this to convert an eps into svg for editing in Inkscape:

pstoedit -f plot-svg input.eps output.svg

Then save the SVG in Inkscape as a “Plain SVG”, and use

convert input.svg output.pdf

Install the open-source PDF Toolkit

Get the PDF Toolkit package, available for Windows, Mac OSX 10.3+ (or build from source for earlier Mac OSs) and Linux.

It’s in the Ubuntu repositories:

sudo apt-get install pdftk 

Split pages from a PDF

To split out pages 10-15 and page 19 of whole.pdf into a separate PDF, piece.pdf, run

pdftk whole.pdf cat 10-15 19 output piece.pdf

Merge multiple PDFs into one

To combine first.pdf, second.pdf, and third.pdf into big.pdf, use

pdftk first.pdf second.pdf third.pdf cat output big.pdf

Other examples here

The basic idea

You’ll need the command line utility mencoder (which comes with mplayer) for this.

The command to create a movie out of a list of files is this (note this command should all be on one line, but it needs to be wrapped to be displayed here):

mencoder "mf://@temp.txt" -mf fps=25 -o output.avi -ovc lavc
 -lavcopts vcodec=mpeg4

temp.txt is a file containing a list of image files, and the new movie will be called output.avi.

A specific example

This a Python script for my future reference, but perhaps someone will find it useful.
The problem was this:

• I had many subfolders with up to 20,000 jpeg images per folder. Each image was named after the experiment and had a number at the end specifying the frame number.
• Some of those subfolders had up to three different experiments in them. Each experiment needed its own movie.
• It was OK to have multiple movies for a single experiment.

Directory structure was something like this;

workingDir
    -- day1
            -- a_001.jpg
            -- a_002.jpg
            ...
            -- a_21999.jpg
            -- b_001.jpg
            -- b_002.jpg
            ...
            -- b_487.jpg
    --  day2
            -- c_001.jpg
            -- c_002.jpg
            ...
            -- c_1478.jpg
    -- day3
            ...
    ...
    -- day20

I decided to use Python to parse an input file containing a list of the directories (created using ls workingDir > folders-to-run.txt). If I wanted to have a movie created from the contents of a folder, I marked that folder by putting a “*” as the first character on the line.

I parsed the contents of each marked directory, separated out files that had different base names, then sorted by the frame number. I decided on a movie length of 2000 frames to keep file sizes manageable. For each batch of 2000 images from an experiment, I wrote the filenames to temp.txt and fed that to mencoder.

folders-to-run.txt looked like this:

day1
*day2
*day3
...
day20

(Since only day2 and day3 had a *, they would be the only ones run)

In the end, I’ll end up with movies called:

a_0-2000.avi
a_2000-4000.avi
a_4000-6000.avi
...
a_20000-22000.avi
b_0-2000.avi
c_0-2000.avi
...

Those last two will be smaller than each of the “a” movies, since they have less frames, but that’s OK for my purposes.

Here’s the working Python script.

"""A Python script for creating movies out of directories of jpegs.
Requires mencoder to be installed."""

import os
import re

directory = 'images/workingDir''

# Parse the list of folders to run from the input file.
folderlist = open('folders-to-run.txt')
folders = [line.rstrip() for line in folderlist if line.startswith('*')]
folders = [i.replace('*','') for i in folders]

# Set up regular expressions for later
RE = re.compile('.*-(\d*)\.jpg')
fbRE = re.compile('(.*)-.*\.jpg')

# How many frames to use per movie
framestep = 2000

for folder in folders:

    print '\n\n\tlisting contents of %s . . .'%folder
    files = os.listdir(os.path.join(directory,folder))
    print '%s files found.\n\n' % len(files)

    # If a file is called "asdf-003.jpg", the basename will be 'asdf'.
    basenames = [fbRE.match(i).groups()[0] for i in files if fbRE.match(i)]

    # Get the set of unique basenames.  In the
    basenames = list(set(basenames))

    print '\t***There are %s different runs here.***' % len(basenames)

    # This loop will only execute once if there was only a single experiment
    # in the folder.
    for j,bn in enumerate(basenames):
        these_files = [i for i in files if bn in i]

        # Sort using the "decorate-sort-undecorate" approach
        these_sorted_files = [(int(RE.match(i).groups()[0]),i) for i in these_files if RE.match(i)]
        these_sorted_files.sort()
        these_sorted_files = [i[1] for i in these_sorted_files]

        # these_sorted_files is now a list of the filenames a_001.jpg, a_002.jpg, etc.
        for k in range(0, len(these_sorted_files), framestep):

            frame1 = k
            frame2 = k+framestep

            # Name the output file based on the folder, basename, a number
            # to indicate if it was a second basename in the folder, and the
            # frame range.  Then replace any spaces with "_" so as not
            # to confuse the mencoder command
            this_output_name = '%s_%s_%s_%s-%s.avi' % (folder,bn,j,frame1,frame2
            this_output_name = this_output_name.replace(' ','-')
            print '\n\n\toutput will be %s.' % this_output_name

            #Create a temporary text file that will contain pathnames.  This
            # will be passed to mencoder.
            f = open('temp.txt','w')
            filenames = [os.path.join(directory,folder,i)+'\n' \
                                for i in these_sorted_files[frame1:frame2]]
            f.writelines(filenames)
            f.close()

            # Finally!  Now execute the command to create the video.
            cmd = 'mencoder "mf://@temp.txt" -mf fps=25 -o %s -ovc lavc\
            -lavcopts vcodec=mpeg4' % this_output_name

            os.system(cmd)
            print '\n\nDONE with %s' % this_output_name
print 'Done with all marked folders.'