unsealed counsel
16 Apr 2022

Consult interface for dash-docs

A consult interface for a command is the usage of consult--read for completing read. This post is a list of issues encountered in the initial development of consult-dash.

1. Dynamic completing read

Unlike emacs' standard completing-read function, consult--read does not support dynamic completion tables. Such completion tables are completions generated on the fly in response to user input. As an example, consider a hypothetical command find-synonyms which produces choices of synonyms. Suppose the minibuffer contains ca and the user types in the characters p, e and r. The choices are produced by a dynamic commpletion function that lists synonyms of cap, cape, and caper. Note that the list of synonyms must be regenerated every time the input changes.

Libraries must use an asynchronous generator of completion results to emulate dynamic completion tables with consult. This approach is much nicer in terms of lag, but complicates library design significantly.

Edit: Daniel Mendler, the author of consult, kindly and correctly points out that the complexity is inherent in the usage of asynchronous facilities, and is not required by the use of consult--read itself. Indeed, the use of asynchronous completion makes the interface feel far snappier than the usage of dynamic completion tables. Additionally, consult--read brings with it multiple goodies, e.g., a simple way to set the default initial value to the symbol at point, without requiring a new wrapper function.

The basic pipeline used by dash-docs to identify candidates is to synchronously query dash SQLite databases each time a new character is input. There is some clever caching to avoid some redundant queries, but the fundamental operation is synchronous. When input changes, the generated candidate list is presented to completing-read using the dynamic completion table interface.

Faking asynchronous behavior using timers with a synchronous process does not work with consult--read. The simplest example then to follow for asynchronous command completions is consult-man from consult.el. The asynchronous pipeline consists of two parts:

  1. consult-dash--builder: a command builder given user input
  2. consult-dash--format: a formatter that converts the output of the command into candidates

With those parts in place, constructing a new async command is pretty straightforward:

(defun consult-dash (&optional initial)
  "Consult interface for dash documentation.

INITIAL is the default value provided."
  (interactive)
  (dash-docs-create-common-connections)
  (dash-docs-create-buffer-connections)
  (setq consult-dash--current-docset nil)
  (when-let* ((builder (consult-dash--with-buffer-context #'consult-dash--builder))
              (search-result (consult--read
                              (consult--async-command builder
                                (consult--async-transform consult-dash--format)
                                (consult--async-highlight builder))
                              :prompt "Dash: "
                              :require-match t
                              :group #'consult-dash--group
                              :lookup #'consult--lookup-cdr
                              :category 'consult-dash-result
                              :annotate #'consult-dash--annotate
                              :initial (consult--async-split-initial initial)
                              :add-history (consult--async-split-thingatpt 'symbol)
                              :history '(:input consult-dash--history))))
    (dash-docs-browse-url search-result)))

2. Asynchronous commands and buffers

Unlike consult-man, and unlike every other consult interface known to this author, consult-dash needs a command-line builder that is specific to the buffer from which it is called: the list of docsets used in any buffer is specific to that buffer. However, when the builder is called asynchronously by consult--read, it is run with a different buffer context. Hence the buffer itself needs to be saved when passing the builder to consult--read:

(defun consult-dash--with-buffer-context (func)
  "Ensure that FUNC is called with the correct buffer context."
  (let ((buf (current-buffer)))
    (lambda (&rest args)
      (with-current-buffer buf
        (apply func args)))))

Note that lexical binding is in effect above. In the invocation of consult--read, the appropriate function is constructed on the fly:

(when-let* ((builder (consult-dash--with-buffer-context #'consult-dash--builder))
            (search-result (consult--read
                            (consult--async-command builder
                              (consult--async-transform consult-dash--format)
                              (consult--async-highlight builder))
                            ...)))
  ...)

Similarly, the formatter is called with partial output (though always complete lines), and hence may need to retain state between calls. In this case, the state (consult-dash--current-docset) is stored a global variable. (Edit: The idea of using a let-bound variable, as in the builder, does not work, but can be fixed with a bit of effort.)

(when-let* ((builder (consult-dash--with-buffer-context #'consult-dash--builder))
            (current-docset t) ; t due to when-let
            (formatter (some-func current-docset #'consult-dash--format))
            (search-result (consult--read
                            (consult--async-command builder
                              (consult--async-transform formatter)
                              (consult--async-highlight builder))
                            ...)))
  ...)

Edit: It appears that the closure approach should work; still need to figure out the bugs in my earlier failed implementation.

3. Multiple commands

Consult's support for asynchronous command builders assumes a single command that is run. However, in the case of dash-docs, we need to run multiple commands, one for each docset. Ideally, we would have a sequence of commands which should be launched in a chain, i.e., each new command would be spawned when the previous one completes. However, it is unclear how one would incorporate such a process chain into consult's async functionality. Any ideas here would be greatly appreciated.

The alternate mechanism that was followed in consult-dash was the construction of a long shell command line consisting of independent statements querying each docset. Essentially the command run is equivalent to the following run under /bin/sh:

echo "DOCSET:" "DOCSET1"
sqlite3 "some DOCSET1 query"
echo "DOCSET:" "DOCSET2"
sqlite3 "some DOCSET2 query"
echo "DOCSET:" "DOCSET3"
sqlite3 "some DOCSET3 query"
...

The markers from the "DOCSET:" lines are used to identify the docset from which subsequent query results are found since the transformer processes the lines in sequence. This approach restricts consult-dash to only those systems with a POSIX shell. Any ideas on ways to execute these command sequences with pure emacs facilities would also be greatly appreciated.

4. Text properties

Attempts to add support for embark and marginalia indicated another problem: only the formatted string (shown as the candidate in the minibuffer) was presented to those functions, and not the actual non-formatted candidates themselves. consult-dash uses text properties attached to the candidate strings to transfer information to embark and marginalia annotators. consult--read itself returns the original non-formatted candidate. Any ideas on better techniques would also be greatly appreciated.

Edit: Daniel Mendler, the author of consult, provided a definitive answer that the completing read interface is the limitation, not the lack of techniques.

Tags: emacs, consult, dash, async
Creative Commons License
runes.lexarcana.com by Ravi R Kiran is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License .