Keeping a lab notebook with org-mode, git, Papers, and Pandoc (Part II)

I promised a while ago that I’d post the code I used to make org-mode a powerful lab notebook, and here it is. In Step 1, I present the emacs lisp for the lab-notebook minor mode. In Step 2, I show you how to leverage Papers to work with Emacs and Pandoc. In Step 3, I show you how to hook Pandoc up with Emacs.

As I mentioned previously, these are all hacked-together solutions. My requirements and setup will likely be different from yours, and I have probably forgotten a few key steps. For instance, while I use a Mac, the only parts that are specific to Macs are the hooks with Papers, a reference library application. Emacs, Pandoc, and bibtex are all fairly cross-platform solutions, I think.

Step 1: lab-notebook mode in emacs

The code in the gist below provides a minor-mode for emacs that enables checkin-on-save and auto-export from Papers (using the Applescript shown in Step 2). Load this in your .emacs file and initiate by M-x lab-notebook-mode, or add -*- eval: (lab-notebook-mode) -*- to the top of your notebook file.

(defun ensure-in-vc-or-checkin ()
(if (file-exists-p (format "%s" (buffer-file-name)))
(progn (vc-next-action nil) (message "Committed"))
(ding) (message "File not checked in.")))
(defun export-bibtex ()
"Exports Papers library using a custom applescript."
(message "Exporting papers library...")
(shell-command "osascript ~/notebook/papers_bibtex_export.scpt"))
(define-minor-mode lab-notebook-mode
"Toggle lab notebook mode.
In this mode, org files that are saved are automatically committed by the VC
system in Emacs. Additionally, Papers library export to bibtex is hooked to
the <f5> key.
Future additions may hook a confluence upload or Org export to this mode as well."
;; initial value
;; indicator
:lighter " LN"
;; keybindings
:keymap (let ((map (make-sparse-keymap)))
(define-key map (kbd "<f5>") 'export-bibtex)
;; body
(add-hook 'after-save-hook 'ensure-in-vc-or-checkin nil 'make-it-local))
(provide 'lab-notebook-mode)
view raw lab-notebook.el hosted with ❤ by GitHub

Step 2: Papers auto-export and citekeys

Papers comes with a Citations tool that allows you to insert a citekey in any text field, including Emacs. A citekey is a short, unique reference to a publication or entry in your reference library, used by tools to automatically format your citations upon printing or export. To use, summon Citations using whatever keybinding you chose in Papers’ preferences and add a citekey like you would in a Word document. In our setup, we’ll be using Pandoc, so be sure to change the citekey type to ‘Pandoc’ (Papers -> Preferences -> Citations -> Format: Pandoc Citation).

Pandoc doesn’t talk to Papers, but Papers can export its library in a format that Pandoc will understand, called bibtex. We will want this to happen automatically before Pandoc attempts to render our notebook. The applescript below exports my Papers library as a bibtex file in the location specified by “path_to_notebook_folder” (replace with whatever you want). Save this file in the same location (or wherever) and update the part of the Gist above that starts (shell-command "osascript ... with the right path and name that matches whatever you saved it as.

tell application "Papers"
set outFile to "path_to_notebook_folder/library.bib"
export ((every publication item) as list) to outFile
end tell
view raw papers2bibtex.scpt hosted with ❤ by GitHub

The benefit of adding this into Emacs is that you can either press a key (by default, F5) to automatically keep your bibtex file up-to-date, or add a hook so that when you render your notebook using Pandoc, it automatically updates your bibtex file first before trying to build your citations.

Step 3: Use ox-pandoc in emacs to export a pretty version of your notebook.

While org-mode does include its own LaTeX and HTML export commands, I found it easier to use Pandoc and the ox-pandoc Emacs package. The benefits of Pandoc is that it exports to a much wider variety of formats and has a simple and easy citations manager, pandoc-citeproc. Citeproc can use the bibtex library we exported in Step 2- all we have to do is add the following somewhere in our notebook org file:
#+PANDOC_OPTIONS: bibliography:library.bib
Make sure there is a library.bib file in the same folder as your notebook file, of course. Now, to export to Latex with formatted references, simply use M-x org-pandoc-export-to-latex-pdf and it will produce a beautiful PDF copy of your notebook. Note: in case it wasn’t clear, you will need to install Pandoc and ox-pandoc since they are not built in. Only recent versions of Pandoc handle org-mode files, so make sure you’re up-to-date. ox-pandoc is available on the Emacs package repo Marmelade.

Example file

An example file. Since Github automatically renders org-mode files when they’re displayed, this is a link to the raw source so you can see the markup used.

Pandoc can export to Github-flavored markdown, so here is the same file rendered:

<2014-09-25 Thu>

Reading this (Wang et al. 2010), in particular Supp. report 4. Has to do with bootstrapping population estimates using the different enzymes used to find integration sites.

<2014-09-26 Fri>

Notes on (Wang et al. 2010) :popest:petersen:intsites:

The review (here A. Chao 2001) indicates sample coverage approaches may be most appropriate for models where the capture probabilities can vary both over time and between species (which is certainly the case for estimating populations from longitudinal T cell sequencing data). I'll need to read more (esp. (Tsay and Chao 2001) and (Chao and Tsay 1998)).

Also found the package Rcapture which appears to fit a Poisson regression to incident counts. I've gotten it to work (kind of) but R crashed trying to plot the model so I'll need to come back to this. If memory serves, it was substantially more conservative in its estimates than the Chao2 estimate.


Chao, A, and P K Tsay. 1998. “A sample coverage approach to multiple-system estimation with application to census undercount.” Journal of the American Statistical Association 93 (441): 283–93. doi:10.2307/2669624.

Chao, Anne. 2001. “An overview of closed capture-recapture models.” Journal of Agricultural, Biological, and Environmental Statistics 6 (2). Springer-Verlag: 158–75. doi:10.1198/108571101750524670.

Tsay, P K, and A Chao. 2001. “Population size estimation for capture - recapture models with applications to epidemiological data.” Journal of Applied Statistics 28 (1): 25–36.

Wang, Gary P, Charles C Berry, Nirav Malani, Philippe Leboulch, Alain Fischer, Salima Hacein-Bey-Abina, Marina Cavazzana-Calvo, and Frederic D Bushman. 2010. “Dynamics of gene-modified progenitor cells analyzed by tracking retroviral integration sites in a human SCID-X1 gene therapy trial.” Blood 115 (22): 4356–66. doi:10.1182/blood-2009-12-257352.

view raw hosted with ❤ by GitHub

Further reading

3 thoughts on “Keeping a lab notebook with org-mode, git, Papers, and Pandoc (Part II)

  1. Thanks for sharing this. I like the way you made a minor mode for the save hook. Seems pretty simple.
    However, you might want to take a look at Firestarter[1], which is an Emacs package that: “lets you execute all kinds of things after saving a buffer. You can run shell commands, execute interactive Emacs commands and evaluate arbitrary pieces of Emacs Lisp. Support for shell commands includes asynchronous process control, format chars in command strings and report of process on common process termination states.”


  2. Thanks for sharing this! For a while I was struggling with how to connect my lab notebook in Emacs with my papers in Mendeley. This also would have been super useful when I kept a reading list for qualifying exams in Org mode. I was able to implement your idea using the citation key in Mendeley (instead of Papers) on Linux.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s