Emacs and Org posts are a dime a dozen on the internet, and they’re usually extremely specific, going into precise detail on how the author uses every tool in Org’s arsenal. I usually find these blog posts overwhelming and not particularly useful. Maybe I’ll pick up some tricks and integrate them into my own setup, but usually my eyes glaze over and I leave none the wiser.

So rather than explain my system in detail, I’ll just list some basic tricks you might find useful for your own setup. A lot of Org posts go into detail about actually using Org itself, but I found that I didn’t really use Org very much at all until I fixed some basic usability issues not directly related to Org at all.

What didn’t work for me

It took me a while to really get into Org mainly because of sync. I had a fair number of computers I needed to edit my Org files own (including my phone), so I needed to make sure they were always available on all the computers. My first solution was Dropbox, which sounded perfect on paper. I would simply edit the org files and Dropbox would sync them to the cloud and then my other devices. There was even an Android app, Orgzly, that could edit Org files and sync them with Dropbox: perfect.

Except not perfect. I immediately ran into sync issues. Files would become desynced with 2 clients having different sets of changes. Dropbox would resolve the issue by copying the file and adding (Conflicted version...) to the end. Without automatic merging, I had to diff the files, see which lines were missing from where, and manually and painstakingly merge them. Org files are actually very difficult to edit as text files. If they’re longer than like 50 lines, you absolutely need code folding for them (headers and property drawers, for example), but diffs were always plaintext un-folded, so you didn’t really have enough context to properly merge. This went triple for Orgzly, which clobbered my main Org file at least once. I relegated Orgzly to editing its own file where it couldn’t cause too much damage, but multiple isolated Org files was not really want I was after.

I also found Orgzly to be pretty clunky. I could never figure out how to capture a quick note from my phone with Orgzly, and the settings were dense and difficult to understand. Org setups tend to be very custom, leaving non-Emacs Org editors with real usability issues as they try and judge how much of Org they should support, and how to handle configuration. And everyone uses Org differently, so there’s a high chance that whatever subset they do implement is not the subset that you use.

Git

I directly identified that the issue with the Dropbox solution was merging, but it took me a while to think up a setup that would be simple enough for me to use on a daily basis. Because merging was my issue, tracking changes to my Org file with git was an obvious choice, but git’s ergonomics are heavily biased towards multi-user source control, not single user sync. I didn’t want to think of a commit message for all my arbitrary changes. I eventually “invented” the auto-commit script:

#!/usr/bin/env bash

# Add any changed files
git add -u
# All all new files from directories where I always want new files
git add roam .attach
git commit -m "Automatic commit at `date`"
git push

And it’s associated Emacs function:

(defun auto-commit ()
  (interactive)
  (shell-command "cd $(git rev-parse --show-toplevel) && ./auto-commit"))

I didn’t add this to any Emacs hooks, I just call it with M-x whenever I want to sync my changes. This unconditionally commits local changes before pushing to a remote repository. If this repo didn’t have the most up to date changes, you don’t have to worry about the remote changes clobbering local ones because they are already commited, and then you can deal with the merge using git’s full merge system and Magit’s UI, which makes things much easier.¹ I ended up placing these repositories outside of Dropbox because they are intended to be logically separate and Dropbox would only get in the way.

Capturing from my phone

So our Org files are now in a git repo and all changes are tracked. So accessing Org from my phone got much harder, right? It’s going to take a terminal emulator and git and all sorts of stuff to get this working, right?

Well, Orgzly can still sync with Dropbox. That didn’t work when it was syncing with several other computers, but what if it was the only client writing to Dropbox? Orgzly would write changes to Dropbox, Dropbox would sync them to a real computer, and that computer could read the changes and push them to the Git repo. Because all changes flowed from Orgzly, it wouldn’t really need to sync, right?

In practice, this did not work very well. Although this made sync easier to deal with and more structured, sync still had to happen. I would capture things with Orgzly, and commit them, but then I’d have to sync with git. So changes would still have to flow the other way. Orgzly had to care about whether any captured entries had been deleted because git cared, and while syncing was less nightmarish, it was not the effortless sync I was looking for.

So I gave up on phone capturing for a time while I tried to think of a way to organize the system. Everything I was capturing on the phone was meant to be processed later, either with more time, a better browser, or a different mindset. The phone never had to consume any entries, just produce them. It also didn’t have to happen immediately: I could commit the phone entries to the repo at my leisure, and I could use a full computer to do it with.

So I eventually settled on writing entries from my phone into a chat application (Discord in this case, but you could use any) and then collecting them later. The chat didn’t have to persist message forever, just long enough for me not to forget them.

However, I wanted to be able to capture messages into my Org file and then delete them later when I was finished with them. Being able to delete the chat messages would also be useful. So, I couldn’t rely on matching messages to individual Org headlines by ID because neither the message nor the Org headline were guaranteed to exist later. So I needed some kind of external persistent state² to track which messages had been processed into Org and which hadn’t. A hosted database was right out: I wanted to be able to run this from wherever I was and not bother with network authentication. Tracking the state with git was the best option, because then the output and the state would be tracked side by side. I decided to store the state in the properties of a headline Org file itself rather than a JSON file or SQLITE database for ease of administration.

So I eventually settled on the structure of a program that would read message state from an Org file, query messages from the chat app’s API, and then write new messages to the Org file. This design took me quite a while to settle on, but it was straightforward to implement. I picked Rust because my last cursed sync program³ was in Go and became unmaintainable after 3 rounds of revisions, and because Rust had a good Org parser and a good Discord client. It was also extremely fast. This didn’t matter initially, but the program scans the entirety of the file looking for “configs” to get messages from, and the file has gotten quite long.

I spent the bulk of my time writing the translator for turning Discord message embeds into Org tasks. I got very tired of all my Twitter links being entirely opaque, and the Discord embeds have at least their text. I called the program link_dump because it dumps links into my Org files.

I have the program and tokens set up in their own copy of the repo. This lets me start by pulling the latest changes, running the program to dump links, then running auto-commit to push the changes: guaranteed zero merge issues.

This program is not open source because it’s so specific to my needs, but if there’s a demand, I can think about open source it.

The pattern of requesting data and automatically rendering it into Org files while triggering the processing manually makes the process really easy to control and understand. This pattern can also be applied to a number of other tools. You could get RSS feeds, Twitter accounts, and Youtube subscriptions dumped directly into your Org file for efficient reading. Many chat apps, including Discord, also support webhooks, so you could use an automation platform like Zapier or IFTTT to get many platforms for the implementation of one. I mostly use it for notes to myself.