Automate a developer workflow in Emacs

During development of an application, multiple times I had to reset a process that the application depended. This reset required a cumbersome sequence of steps both in and outside of my IDE, Emacs. This note describes how I reduced that workflow to a single call to an Emacs Lisp function.

Introduction

For one of my clients I’ve been working on an application that creates a report in the form of a static website. The application retrieves data from several sources, processes it and reports the results through Markdown files, PNG files for plots and HTML files for tables. The static website generator Material for MkDocs, MkDocs for short, builds a static website from these files.

MkDocs comes with a small server that watches over the directory tree that contains the files the application creates. As soon as something changes in that directory tree, it regenerates the relevant parts of the static website. If your browser has a tab open with a page that is rebuilt, the server triggers a reload. This is really handy during development.

Every once in a while the server gets stuck, for example if the application has created incorrect Markdown. When that happens, I have to stop the server, fix the issue and restart the server. In some cases, I have to delete all the files to restart with a clean slate. Here’s the workflow involved:

1. I switch to the terminal that runs the server and stop it. If I want to start from a clean slate, I use this terminal to manually delete all files the application has created.
2. I switch back to Emacs to run the application to recreate all files - I use a Makefile for that, which I run from Emacs. When this build has finished successfully, I switch back to the terminal and start the server.
3. I wait until the server has rebuild the site to be sure the output of my application is valid.

This process requires a lot of keystrokes to switch between applications and execute all the required commands. It’s not a big thing, but it adds friction to an otherwise rather seamless development experience. So I set out to automate this workflow in Emacs.

Manage an external process in Emacs Lisp

For me the big unknown was how to use Emacs Lisp to 1) start and end the MkDocs server and 2) show the server output in a buffer. Some googling lead me to the start-process command and after some trial & error, I had the following expression to start the MkDocs server:

(start-process "my-report-process"    # name of the new server process in Emacs
               "*my-report-process*"  # name of the Emacs that shows the output of the server process
               "mkdocs"               # command to run
               "serve"                # 1st argument to command to run
               "--dev-addr"           # 2nd argument to ...
               "localhost:8000")      # 3rd argument to ...

If you want to get info about this process, you can just ask. For example, the following snippet checks if the process is already running:

(let ((serve-process (get-process "my-report-process")))
  (if (or (not serve-process) (not (process-live-p serve-process)))
      (message "Report process is running")
    (message "Report process hasn't started yet or has already stopped")))

The following expression ends the MkDocs server when it’s running:

(let ((serve-process (get-process "my-report-process")))
  (when (and serve-process (process-live-p serve-process))
      (delete-process server-process)))

Now comes the part that turned out to be a bit more difficult.

(Re)start the MkDocs server after a compilation

To run the application I use a Makefile and from Emacs I “make” the appropriate target using the compile command. So my first approach to run the application and start the MkDocs server looked like this:

(compile "make create-report")
(let ((serve-process (get-process "my-report-process")))
  (if (or (not serve-process) (not (process-live-p serve-process)))
      (message "start the MkDocs server")))

To my surprise, this didn’t work. The reason is that compile starts make asynchronously. So the snippet above would start the server while the build was still in progress. As the input for the server had not been generated yet, the server would abort.

I had hoped that compile would allow you to provide an Emacs Lisp function to call on completion, but that’s not the case. However, it does provide a hook for functions to call after a compilation, compilation-finish-functions. Unfortunately, these hook functions are called after every compilation that is executed, not just the one you’re interested in. An additional problem is that there are a lot of Emacs functions that use compilation, so the hook functions are called a lot…

The hook functions are passed the compilation buffer and a message that describes how the compilation finished. I found a blog post for a hook function that first checks if the compilation buffer is visible in the right Emacs frame and only if that’s the case, “does its thing”. What if I create a hook function that only starts the server when it’s for the right compile command? The following snippet shows how such a hook would look:

(defun my-compilation-finish-function (compilation-buffer msg)
  (with-current-buffer compilation-buffer
    (save-excursion
      (beginning-of-buffer)
      (when (re-search-forward "make create-report" nil t)
        (message "start the MkDocs server")))))

(add-hook 'compilation-finish-functions 'my-compilation-finish-function)

It turns out that this approach works.

Finetuning

Of course, this can and should be finetuned. First of all, you only need to start the server when the build was successful. I found out that in that case, the msg that is passed to the hook function is “finished\n”¹. This means I can wrap the body of my-compilation-finish-function in the following expression:

  (when (string= msg "finished\n")
    ;; body of my-compilation-finish-function
    )

Another adjustment is the regex to use to determine whether the compilation is for an appropriate build. You want to make sure that your hook doesn’t act on a compilation it shouldn’t act on. To reduce that risk, I added a tag to my compilation command and let the hook react on that tag. For example, instead of

(compile "make create-report")

I call

(compile "make create-report  # my-project: serve report")

and let the hook search for

        (when (re-search-forward "^make .*  # my-project: serve report$" nil t)

In hindsight, I don’t really need it for my current needs but I do have some other ideas on how to use this - more about that later.

The following snippet shows a simplified version of the final code:

(defun my-project-refresh-report()
  "Stop the MkDocs server and start a build."
  (interactive)
  ;; close the mkdocs server if its running
  (let ((serve-process (get-process "my-report-process")))
    (when (and serve-process (process-live-p serve-process))
      (delete-process serve-process)))
  ;; start the compilation in the my-project directory
  (let ((default-directory "/path/to/my-project"))
    (compile "make create-report  # my-project: serve report")))

(defun my-project-serve-report-on-success (buffer msg)
  "Restart the MkDocs server when the build has finished successfully.

This function is to be used as a compilation finish hook."
  (when (string= msg "finished\n")
    (with-current-buffer buffer
      (save-excursion
        (beginning-of-buffer)
        (when (re-search-forward "^make .*  # my-project: serve report$" nil t)
          (my-project-serve-report))))))

(add-hook 'compilation-finish-functions 'my-project-serve-report-on-success)

(defun my-project-serve-report ()
  "Start the MkDocs server when it is not running."
  (let ((serve-process (get-process "my-report-process")))
    (if (or (not serve-process) (not (process-live-p serve-process)))
        (let ((default-directory "/path/to/generated-report"))
          (start-process
           "my-report-process"
           "*my-report-process*"
           "mkdocs"
           "serve"
           "--dev-addr"
           "localhost:8000")))))

So instead of a workflow that requires a lot of keystrokes to switch between applications and execute all the required commands, I only have to execute my-project-refresh-report and I’m done.

There is an optimization I did not implement yet and that’s the use of re-search-forward. Currently it searches the complete compilation buffer for the make command and of course, that’s way too much. I haven’t run into a performance issue yet, but it would be better to limit the search. Looking at the compilation buffer, the make command always seems to be at the 4th line. It would be OK to limit the search command to the first 4, maybe 5 or 6 lines.

Final ideas

I can imagine I’m going to extend the pattern described here to make it easier to support other compilation commands. Suppose you maintain a list of end-of-compilation functions and let each function have a unique tag, the you only need to add a single function to compilation-functions-hook. That single function extracts the tag from the compilation command and if it finds one, executes the associated end-of-compilation function.