I promised a while ago that I’d post the code I used to make org-mode a powerful lab notebook, and here it is. In Step 1, I present the emacs lisp for the lab-notebook minor mode. In Step 2, I show you how to leverage Papers to work with Emacs and Pandoc. In Step 3, I show you how to hook Pandoc up with Emacs.
As I mentioned previously, these are all hacked-together solutions. My requirements and setup will likely be different from yours, and I have probably forgotten a few key steps. For instance, while I use a Mac, the only parts that are specific to Macs are the hooks with Papers, a reference library application. Emacs, Pandoc, and bibtex are all fairly cross-platform solutions, I think.
Step 1: lab-notebook mode in emacs
The code in the gist below provides a minor-mode for emacs that enables checkin-on-save and auto-export from Papers (using the Applescript shown in Step 2). Load this in your .emacs file and initiate by
M-x lab-notebook-mode, or add
-*- eval: (lab-notebook-mode) -*- to the top of your notebook file.
(defun ensure-in-vc-or-checkin ()
(if (file-exists-p (format "%s" (buffer-file-name)))
(progn (vc-next-action nil) (message "Committed"))
(ding) (message "File not checked in.")))
(defun export-bibtex ()
"Exports Papers library using a custom applescript."
(message "Exporting papers library...")
(shell-command "osascript ~/notebook/papers_bibtex_export.scpt"))
"Toggle lab notebook mode.
In this mode, org files that are saved are automatically committed by the VC
system in Emacs. Additionally, Papers library export to bibtex is hooked to
the <f5> key.
Future additions may hook a confluence upload or Org export to this mode as well."
;; initial value
:lighter " LN"
:keymap (let ((map (make-sparse-keymap)))
(define-key map (kbd "<f5>") 'export-bibtex)
(add-hook 'after-save-hook 'ensure-in-vc-or-checkin nil 'make-it-local))
Step 2: Papers auto-export and citekeys
Papers comes with a Citations tool that allows you to insert a citekey in any text field, including Emacs. A citekey is a short, unique reference to a publication or entry in your reference library, used by tools to automatically format your citations upon printing or export. To use, summon Citations using whatever keybinding you chose in Papers’ preferences and add a citekey like you would in a Word document. In our setup, we’ll be using Pandoc, so be sure to change the citekey type to ‘Pandoc’ (Papers -> Preferences -> Citations -> Format: Pandoc Citation).
Pandoc doesn’t talk to Papers, but Papers can export its library in a format that Pandoc will understand, called bibtex. We will want this to happen automatically before Pandoc attempts to render our notebook. The applescript below exports my Papers library as a bibtex file in the location specified by “path_to_notebook_folder” (replace with whatever you want). Save this file in the same location (or wherever) and update the part of the Gist above that starts
(shell-command "osascript ... with the right path and name that matches whatever you saved it as.
|tell application "Papers"|
|set outFile to "path_to_notebook_folder/library.bib"|
|export ((every publication item) as list) to outFile|
The benefit of adding this into Emacs is that you can either press a key (by default, F5) to automatically keep your bibtex file up-to-date, or add a hook so that when you render your notebook using Pandoc, it automatically updates your bibtex file first before trying to build your citations.
Step 3: Use ox-pandoc in emacs to export a pretty version of your notebook.
While org-mode does include its own LaTeX and HTML export commands, I found it easier to use Pandoc and the ox-pandoc Emacs package. The benefits of Pandoc is that it exports to a much wider variety of formats and has a simple and easy citations manager, pandoc-citeproc. Citeproc can use the bibtex library we exported in Step 2- all we have to do is add the following somewhere in our notebook org file:
Make sure there is a library.bib file in the same folder as your notebook file, of course. Now, to export to Latex with formatted references, simply use
M-x org-pandoc-export-to-latex-pdf and it will produce a beautiful PDF copy of your notebook. Note: in case it wasn’t clear, you will need to install Pandoc and ox-pandoc since they are not built in. Only recent versions of Pandoc handle org-mode files, so make sure you’re up-to-date. ox-pandoc is available on the Emacs package repo Marmelade.
Example notebook.org file
An example notebook.org file. Since Github automatically renders org-mode files when they’re displayed, this is a link to the raw source so you can see the markup used.
Pandoc can export to Github-flavored markdown, so here is the same file rendered:
Reading this (Wang et al. 2010), in particular Supp. report 4. Has to do with bootstrapping population estimates using the different enzymes used to find integration sites.
Notes on (Wang et al. 2010) :popest:petersen:intsites:
The review (here A. Chao 2001) indicates sample coverage approaches may be most appropriate for models where the capture probabilities can vary both over time and between species (which is certainly the case for estimating populations from longitudinal T cell sequencing data). I'll need to read more (esp. (Tsay and Chao 2001) and (Chao and Tsay 1998)).
Also found the package Rcapture which appears to fit a Poisson regression to incident counts. I've gotten it to work (kind of) but R crashed trying to plot the model so I'll need to come back to this. If memory serves, it was substantially more conservative in its estimates than the Chao2 estimate.
Chao, A, and P K Tsay. 1998. “A sample coverage approach to multiple-system estimation with application to census undercount.” Journal of the American Statistical Association 93 (441): 283–93. doi:10.2307/2669624.
Chao, Anne. 2001. “An overview of closed capture-recapture models.” Journal of Agricultural, Biological, and Environmental Statistics 6 (2). Springer-Verlag: 158–75. doi:10.1198/108571101750524670.
Tsay, P K, and A Chao. 2001. “Population size estimation for capture - recapture models with applications to epidemiological data.” Journal of Applied Statistics 28 (1): 25–36. http://gateway.webofknowledge.com/gateway/Gateway.cgi?GWVersion=2&SrcAuth=mekentosj&SrcApp=Papers&DestLinkType=FullRecord&DestApp=WOS&KeyUT=000165844500002.
Wang, Gary P, Charles C Berry, Nirav Malani, Philippe Leboulch, Alain Fischer, Salima Hacein-Bey-Abina, Marina Cavazzana-Calvo, and Frederic D Bushman. 2010. “Dynamics of gene-modified progenitor cells analyzed by tracking retroviral integration sites in a human SCID-X1 gene therapy trial.” Blood 115 (22): 4356–66. doi:10.1182/blood-2009-12-257352.