Anonymous state variables in emacs lisp

Asynchronous programming typically involves the use of context-local state variables, e.g., threads can use thread-local storage. In my consult-dash emacs package, results from multiple searches were collected into a single state variable for further processing. The state variable would have been created on the heap in languages such as C, and a reference passed along to any asynchronous calls, typically via a void pointer in C. Each invocation of the asynchronous procedure would create a new anonymous state, which should then be destroyed by the consumer of the result of the asynchronous procedure.

Emacs lisp code typically does not use such state variables since the lisp interpreter is single threaded. Instead, a global variable is used when interactive commands internally use asynchronous features:

(defvar my-package--private-state nil
  "Private state used in my async code")

;; Much later in some command handler
(call-async-function
 (lambda () (setq my-package--private-state (some-computation))))

This approach works since the single-threaded nature of the interpreter assures that such a command completes (or is canceled) prior to a new invocation. However, for long-lived asynchronous state, it would be much preferable to use local state with lexical binding:

;; This does not work, and is intended as an example.
(let ((private-state)) ; anonymous private variable
  (lambda () (setq private-state (some-computation))))

private-state inside the lambda in the preceding example is different from the let-bound anonymous private variable. Shenanigans with symbol-name and intern also do not work since their interaction with lexically scoped variables is quite limited. The simplest workaround is a factory that generates closures with an embedded variable, which can then be modified or examined as a function:

(defun consult-dash--local-variable (&optional value)
  "Closure representing a local variable with initial value VALUE.

To set a new value, call the closure with it as the argument.
To get the current value, call the closure with no arguments."
  (let ((var))
    (setq var value)
    (lambda (&rest val)
      (when val (setq var (car val)))
      var)))

Armed with the definition above, we can now create new state variables and then refer to them in closures:

(let ((private-state (consult-dash--local-variable)))
  (lambda ()
    (private-state (some-computation)) ; set variable
    (private-state)                    ; examine variable
    (do-other-stuff)))

Each time the let-binding is called, a new private variable is now generated. This approach is conceptually similar to gensym, of course, but without the need to pass the variable name itself into the lambda.