A cognitive basis for academic workflows

During doctoral comprehensive exams, I paused to consider the effort it took to research and produce answers to questions that would only be used in a 2-hour comprehensive exam. Of course, if I was lucky, some of the writing might make it into my dissertation. And the time spent reading, thinking, and learning was potentially invaluable. But the process was painful. For each of four papers, I considered my question, amassed sources, crammed notes into Evernote, outlined, wrote, re-wrote, re-thought, re-searched, re-collected, re-assembled, re-drafted and finally spend a day inserting and formatting sources properly. That was just for a first draft.

In the end, all of my work was in disconnected Evernote notes and Word documents.

Like many other doctoral students, I came to the realization that my method was incredibly inefficient and could not scale. In fact, I realized that my largest problem was actually the ability to re-use my own thinking, knowledge, and experience.

So I read everything I could find on academic workflows.

What I found was huge discussion on the relative merits and trade-offs on organization (knowledge management) tools, brain-storming tools, note-taking, bibliography, and writing tools. Naturally, I downloaded a variety and began to explore. At first, I couldn’t see why there were so many over-lapping functionalities. For instance, various students and researchers would admit to using multiple brain-storming tools simultaneously. What the heck?! (I’ll get back to this later.)

Fairly quickly, I came to the conclusion that my best chance at settling on a workflow was to focus on studying the workflows of researchers who were very skilled. I reasoned that someone who publishes a lot probably was good at re-use of previous thinking and work. This post by Steven Berlin Johnson describing his use of DevonThink is representative.

While developing my own workflow, I ran across an article from Pirolli and Card (2005) that reminded me intensely of academic workflows! In fact, they had done a cognitive task analysis (CTA) of intelligence analysts. What they described was sense-making. In essence, sense-making is a set of processes for framing data to a model and also fitting data to a mental model (Klein, Moon, & Hoffman 2006). The first part (framing data) is posed as an “information foraging loop” while the second (fitting data to a model) as a “sense-making loop”. Another way to look at this is as an interleaving of bottom-up processes with top-down processes: search, filter, and understand versus find evidence, examine counter arguments, and re-evaluate. The diagram below is extracted from Pirolli and Card (2005).

My current workflow is represented below. In effect, it represents a simple CTA that generalizes the academic workflow that I now use.

Though I drew in arrows that indicate a sort of continuous back-and-forth process at a micro-level, there do seem to be larger foraging and sense-making loops. It seem so since I spend a day or days at a time doing focused work within specific tools. In fact, I spend a lot of time in Scrivener. Writing seems to require much more effort than note-taking in other parts of my workflow. Probably, before adopting this workflow, I spent most of my time in the foraging loop - and then losing or forgetting much of the learning I had attained. (Let’s face it, hunting is more fun.)

The major tools I use appear under the loops in the diagram above:

• Papers 2 - bibliography and PDF management. I do a lot of reading and note-taking directly in Papers 2.
• DevonThink - content management which gets feeds directly from RSS feeds and web clippings. DT is a big shoebox for everything I collect that is not a PDF.
• Skim PDF reader - free PDF reader with some very useful note-taking and export functionality.
• Tinderbox - thinking and brainstorming
• Microsoft OneNote - note-taking using pen and clippings on my Surface Pro. Skydrive sync lets me view and edit notes on the Mac, as well.
• Scrivener - writing. Even my blog is written using Scrivener. Any notes captured in tools above can be imported into Scrivener by converting them to PDF first.

Not shown is a production tool. I typically export from Scrivener to LaTeX (the subject of a future post).

Another way to look at this process is in terms of note-taking. At each stage in the process, my notes become more compact and inter-connected. When I read an article, I may outline summaries and extract concepts and quotes. But when I’m sense-making, I’m primarily working in Tinderbox or OneNote and what’s important is thinking and bridging – the relationship between notes. They are typically short, but need to link back to summaries, extracts, and sources. When I write prose, I work in Scrivener. The link back to intermediate sources (Tinderbox & OneNote) and original sources (Pages & DT) is important to maintain. Sometimes there is the need to go back and work on material in more depth. But, in the end, all of these intermediary products are re-usable.

Now, back to the question of overlapping functionality. Why is it that, at least on the Mac, there is so much overlap in functionality between “academic workflow” tools? Why would I want to take notes in multiple tools? It turns out that note-taking is a multi-faceted beast. Depending where I am in the sense-making process, I’m taking different sorts of notes. For example, below are different sorts of notes serving different sorts of purposes - which may be used in different parts of the sense-making loop. I adapted the scheme below based on an analysis of historians using DT.

Labels, tags, notes (summaries, maps, highlights, etc.) are all really just notes. We just use them in different ways. And some are obviously more compact than others. As Mark Bernstein of eastgate.com remarks:

• Notes are records, reminding us of ideas and observations that we might otherwise forget
• Shared notes are a medium, an efficient communication channel between colleagues and collaborators
• Notes are a process for clarifying thinking and for refining inchoate ideas

(These remarks can be found in Bernstein's talk on the youtube video link at the bottom of this post.)

There are also other small workflows which impact productivity in a cumulative sense. One of the most valuable has been to create a small workflow which is triggered whenever I save a PDF to the desktop. When this happens, a hazel script immediately imports the document into Papers2. This causes me to source the document right away so that when I need to reference a source downstream, its already entered into Papers2.

In the end, I hope to capture my own knowledge and experience over time so that I can build on previous work without having to re-think and re-organize every time I start a new paper. It’s an experiment in progress. FYI - I still use Evernote. But more for random odds-and-ends such as notes on programming and how to keep my blog running smoothly.

Here are some of the references online that I found particularly inspiring for my own workflow. Interestingly, I found lots of useful information about DT from historians, while historical fiction writers seem more excited by tools like Tinderbox. Scrivener, however, definitely crosses boundaries. Apparently, all of us are deeply concerned with writing and production. Enjoy!

• James Ostwald's blog - methodology (DT)
• Rachel Leow's blog "A Historian's Craft" - multi-part series on DT
• Parezco Y Digo blog - DT for historical research
• How Tinderbox can help writers - Mark Bernstein talks about Tinderbox on Youtube
• Welcome to Sherwood - Tinderbox chronicles
• Twig (Tinderbox) screencasts
• Ultra-learning - insights on active note-taking

Klein, G., Moon, B., & Hoffman, R. (2006). Making Sense of Sensemaking 2: A Macrocognitive Model. Intelligent Systems, IEEE, 21(5).

Pirolli, P., & Card, S. (2005). The sensemaking process and leverage points for analyst technology as identified through cognitive task analysis. Presented at the Proceedings of International Conference on Intelligence Analysis.