personal web site of janos erdos

Vega-Lite charts with Clojure and GraalJs

Wed, 09 Oct 2024 07:00:00 +0000

Vega-Lite charts with Clojure and GraalJS

For a recent side project I needed a charting tool I could operate on the server-side. I initially used gnuplot which was really nice but had some limitations (for example, making plots clickable) and found the DSL somewhat complex. Then I discovered Vega and Vega-Lite and I really liked the elegant style and flexibility. But since these libraries are written in Javascript to be used in the frontend, I had to find a way to be able to integrate them in my backend-heavy Clojure application. This limitation led me to use a Clojure -> GraalJS -> Javascript interop approach to generate charts from the backend.

Setup

I got a hand on the Javascript files by just copying them from the node packages directly into the resources folder:

$ npm install vega vega-lite
$ cp node_modules/vega-lite/build/vega-lite.js resources/vega-lite.js
$ cp node_modules/vega/build/vega.js resources/vega.js

Next, here is a snipped of the contents of what deps.edn looked like. The configuration should be similar with other build systems also.

{:deps {org.clojure/clojure {:mvn/version "1.12.0-alpha10"}
        org.graalvm.js/js   {:mvn/version "23.0.4"}}
 :paths ["src" "resources"]
 :aliases {:run {:exec-fn core/-main}}}

For simplicity, all the code in the next steps should go to just a core.clj source file.

Calling GraalJs

To create the environment to run Javascript, we need a Context object that is going to hold the active value bindings:

(defonce context
  (-> (org.graalvm.polyglot.Context/newBuilder (into-array String []))
      (.allowAllAccess true)
      (.build)))

With the context established, we can define a function to evaluate Javascript code:

(defn- js-eval [source] (.eval context "js" (str source)))

Utilities

I found that the Javascript runtime provided by GraalJs lacks some features, but we can easily implement what we need:

Structured clone: The window.structuredClone is missing but we can define a simple implementation:
```
(js-eval "function structuredClone(a) { return JSON.parse(JSON.stringify(a)) }")
```
Converting Clojure data: And we need an utility to recursively convert Clojure data structures to Javascript-compatible types:

(defn- javascriptify1 [x]
  (cond (map? x) (org.graalvm.polyglot.proxy.ProxyObject/fromMap x)
        (map-entry? x) x
        (keyword? x) (name x)
        (sequential? x) (org.graalvm.polyglot.proxy.ProxyArray/fromList (vec x))
        :else x))

(defn edn->js [data]
  (clojure.walk/postwalk javascriptify1 data))

Callbacks: We need a way to handle Clojure functions as Javasript callbacks. It can be done by implementing the Consumer interface.
```
(defn- callback [f]
  (reify java.util.function.Consumer
 (accept [_ x] (f x))))
```
Timeouts: The window.setTimeout function is currently missing in GraalJs, so we will have to implement it ourserves.

Integrating Vega and Vega-Lite

We can go ahead with loading the Vega and Vega-Lite libraries into the context.

(js-eval (slurp (clojure.java.io/resource "vega.min.js")))
(js-eval (slurp (clojure.java.io/resource "vega-lite.min.js")))

The .toSVG() call return a Promise, which can be tricky to implement in Clojure. Instead, I defined a wrapper function to handle callbacks. This function takes three arguments: it parses and renders the Vega-Lite definition provided in the first parameter and calls the function passed in the success parameter with the rendered SVG image, while calling the reject parameter on error.

(def js-render-fn
  (js-eval "(function(input, success, reject) {
               const vegaSpec = vegaLite.compile(input, {}).spec;
               new vega.View(vega.parse(vegaSpec)).toSVG().then(success, reject);           
             })")

Additionally, Vega relies on .setTimeout calls in its resource loader. This mechanism is also invoked when using hyperlinks in the marks, a feature I intend to use.

To work around this, let’s simulate setTimeout() calls. Since actual timing is not required for this use case, the new implementation just registers the callback functions to a list that then can be executed sequentially. This is done in a helper something function:

(defn with-set-timeouts [f]
  (js-eval "globalThis.tasks = []; function setTimeout(f) { globalThis.tasks.push(f); }")
  (let [result (f)]
    (js-eval "while(globalThis.tasks.length > 0) { globalThis.tasks.pop()() }"))
    result))

This approach is taining the previously defined context, and is therefore not therad safe. Ideally, every thread should operate within its own context. For now, synchronizing the calls on the same monitor ensures the calls remain serializable.

(defn render-fn [data]
  (let [data       (edn->js data)
        result     (promise)
        on-success (callback result)
        on-error   (callback (fn [e]
                               (println :error (pr-str e))
                               (result (format "<H1>ERROR</H1><BR/><PRE>%s</PRE>" (pr-str e)))))]
    (locking ::render-plot
      (with-set-timeouts (fn [] (.execute js-render-fn (object-array [data on-success on-error]))))
      (deref result))))

Testing the implementation

To test the above code, the following example Vega-Lite definition can be used:

(def data
  {:$schema "https://vega.github.io/schema/vega-lite/v5.json",
   :description "A simple bar chart with embedded data.",
   :data {:values [{"a" "A", "b" 28}, {"a" "B", "b" 55}, {"a" "C", "b" 43},
                   {"a" "D", "b" 91}, {"a" "E", "b" 81}, {"a" "F", "b" 53},
                   {"a" "G", "b" 19}, {"a" "H", "b" 87}, {"a" "I", "b" 52}]},
   :mark "bar",
   :encoding {:x {:field "a", :type "nominal", :axis {:labelAngle 0}},
              :y {:field "b", :type "quantitative"}
              :href {:field "b"}}})

(println (render-fn data))

This will print the SVG chart to the standard output.

Faster tree-seq in Clojure with transducers

Thu, 11 Jul 2024 21:00:00 +0000

Faster tree-seq in Clojure with transducers

The tree-seq is one of my favourite functions in Clojure. It can be used to lazily iterate in any tree-like data structure in a depth-first order. The definition of the function goes something like this:

(defn tree-seq [branch? children root]
  ((fn walk [node]
     (lazy-seq
      (cons node
            (when (branch? node)
              (mapcat walk (children node)))))) root))

Let’s create a big tree and see how it works. Quickly create an n-ary tree of height h where n=4 and h=10.

(defn ntree [n h]
  (nth (iterate (partial repeat n) :leaf) h))

(def deep-tree (ntree 4 10))

How does such a tree look like? Calling (ntree 2 3) gives (((:leaf :leaf) (:leaf :leaf)) ((:leaf :leaf) (:leaf :leaf))) and calling (ntree 3 2) gives ((:leaf :leaf :leaf) (:leaf :leaf :leaf) (:leaf :leaf :leaf)).

Of course, (ntree 4 10) is much bigger. To see how much, run tree-seq, then count all nodes and also see how long it was running:

(->> deep-tree
     (tree-seq coll? seq)
     (count)
     (time)
     (println "All node count"))
; user=> "Elapsed time: 1125.355003 msecs"
; All node count 1398101

The result sounds correct, given the number of nodes equals N=(n^(h+1)-1)/(n-1). However, the performance could be improved, so lets do a rewrite with transducers. First of all, we need a function that will reduce a tree node with rf if it is a branch, and just skip it otherwise.

#_ ;; pseudocode, will not compile
(defn reduce-node [rf a x]
  (if (branch? x)
    (reduce rf a (children x))
    a))

Then comes the actual transducer: for each element in the collection, try to recursively further reduce it.

#_ ;; pseudocode, will not compile
(defn xf [rf]
  (fn inner
    ([]    (rf))
    ([x]   (rf x))
    ([a x] (reduce-node inner (rf a x) x))))

We are also going to utilize the preserving-reduced trick from clojure/core.clj. It helps make sure that a reduced value stays reduced even when passed through a chain of reduce functions. So the actual xf function is closer to something like this:

#_ ;; pseudocode, will not compile
(defn- preserve-reduced [rf]
  (fn [a b]
    (let [r (rf a b)]
      (if (reduced? r)
        (reduced r)
        r))))

#_ ;; pseudocode, will not compile
(defn xf [rf]
  (fn inner
    ([]    (rf))
    ([x]   (rf x))
    ([a x] (reduce-node (preserve-reduced inner) (rf a x) x))))

Of course it will not compile, because branch? and children are undefined. To resolve this, we assemble it into a single letfn form to give it some function and supply the missing functions as arguments:

(defn tree-transducer [branch? children]
  (fn [rf]
    (letfn [(step [a x]
              (if (branch? x)
                (reduce preserve-reduced a (children x))
                a))
            (inner
              ([]    (rf))
              ([x]   (rf x))
              ([a x] (step (rf a x) x)))
            (preserve-reduced [a b]
              (let [result (inner a b)]
                (if (reduced? result)
                  (reduced result)
                  result)))]
      inner)))

We have a transducer now and we can use it to build an eduction:

(defn tree-eduction [branch? children root]
  (eduction (tree-transducer branch? children) [root]))

Testing it yields:

(->> deep-tree
     (tree-eduction coll? seq)
     (vec)
     (count)
     (println "All node count with tree-eduction:")
     (time))
; All node count with tree-eduction: 1398101
; "Elapsed time: 54.186769 msecs"

Flatten

We can reuse the transducer to reimplement flatten as a transducer. Test it with a small expression counting the number of leaves in the tree. First, with just flatten for a baseline:

(->> deep-tree
     (flatten)
     (count)
     (println "Count of leaves with flatten:")
     (time))
; Count of leaves with flatten: 1048576
; "Elapsed time: 1128.76338 msecs"

The same calculation with transducers yields:

(defn flatten-transducer []
  (comp (tree-transducer sequential? seq)
        (remove sequential?)))

(->> deep-tree
     (transduce (comp (flatten-transducer)
                      (map (constantly 1)))
                + 0)
     (println "Count of leaves with flatten-transducer:")
     (time))
; Count of leaves with flatten-transducer: 1048576
; "Elapsed time: 76.234496 msecs"

Summary

We gained a bit of a speedup by getting rid of temporary object allocations. However, the new approach does not differ only by implementation. Like usually with engineering, a tradeoff was made: This new function returns an eduction instead of a sequence. To highlight the main differences:

An eduction is not a collection. Calling coll? will return false.
As shown in the code, we cannot call count on it without first converting it to a collection. Calling vec on it will reduce it into a vector.
The result of eduction is not caching. Calling seq on it (or reducing it, for example, into a vector) will evaluate the whole thing again and again every time and allocate a new sequence.

Keeping the differences in mind, I still sometimes optimize out parts of code to use reducers and eductions, which often proves to be a low-hanging fruit.

Reading List for 2019 December

Sun, 15 Dec 2019 21:00:00 +0000

Reading List for 2019 December

Clojure - testing private vars

Fri, 25 Oct 2019 21:00:00 +0000

How do we test private vars in Clojure?

How do we write unit tests for private vars?

1. Namespace for implementation

One approach is just not to use private vars. Private vars are hidden, because they are part of the implementation of a functionality. As such, they are not part of the public API of the software, and should not be accessed from the outside world.

You could introduce a new namespace containing the implementation details. Lets call it yournamespace.impl. The name indicates that this namespace contains implementation only and it should not be relied on directly. You can unit test the vars if you define all vars publicly in this namespace.

2. Make private vars visible for testing

An other approach is to use Clojure’s dynamic nature. You can access the namespaces and define vars on the fly for testing:

(let [target (the-ns 'testable-namespace)]
  (doseq [[k v] (ns-map target)
          :when (and (var? v) (= target (.ns v)))]
    (eval `(defn ~(symbol (str "-" k)) [~'& args#] (apply (deref ~v) args#)))))

This code snippet checks all functions in the testable-namespace namespace and creates an alias for them in the current namespace. The aliases will have a - prefix and you can use these names from your unit tests.

For example, if you have the following in testable-namespace ns:

(defn- add [a b] (+ a b))

They you can test it like this in the unit test namespace.

(deftest test-add
  (is (= 5 (-add 2 3))))

Note that we needed to write -add instead of -add in the test.

Clojure collection quirks - A REFERENCE

Thu, 19 May 2016 10:00:02 +0000

This is a reference of Clojure’s default data structures and the most common predicates.

Functions

This table contains the most common predicates applied to the core collections.

	nil	'(1 2 3)	(range 5)	(range) (lazy-seq) (cons 1 ())	Persistent Queue/EMPTY	[1 2 3]	(last {1 1})	(hash-map) (array-map)	(sorted-map)	#{}	(sorted-set)
		Seq				Vector		Map		Set
type	nil	Persistent List	LongRange	Iterate or LazySeq or Cons	Persistent Queue	Persistent Vector	MapEntry	Persistent ArrayMap or HashMap	Persistent TreeMap	Persistent HashSet	Persistent TreeSet
coll?	false	true	true	true	true	true	true	true	true	true	true
counted?	false	true	true	false	true	true	true	true	true	true	true
ifn?	false	false	false	false	false	true	true	true	true	true	true
associative?	false	false	false	false	false	true	true	true	true	false	false
map?	false	false	false	false	false	false	false	true	true	false	false
sequential?	false	true	true	true	true	true	true	false	false	false	false
list?	false	true	false	false	true	false	false	false	false	false	false
seq?	false	true	true	true	false	false	false	false	false	false	false
reversible?	false	false	false	false	false	true	true	false	true	false	true
vector?	false	false	false	false	false	true	true	false	false	false	false
map-entry?	false	false	false	false	false	false	true	false	false	false	false
map?	false	false	false	false	false	false	false	true	true	false	false
set?	false	false	false	false	false	false	false	false	false	true	true
sorted?	false	false	false	false	false	false	false	false	true	false	true

Interfaces

An incomplete list of some of the most commonly used default Clojure interfaces found in the clojure.lang package.

IMeta

The object has meta information accessible via the (meta x) function.

IObj

Indicates that meta information can be added to object (with-meta function).

Counted

This collection supports fast counting ((count x) function call).

IEditableCollection

A transient version of the collection can be created with the (transient x) call.

IFn

Indicates that he object can be called just like a function.

IKeywordLookup

A collection supports indexing by keywords.

ILookup

Collection supports access by index (via get function with default value).

IPending

Supports realized? function.

IPersistentCollection

Persistent collections have support for conj, empty, count, equals functions.

ISeq

Sequences have support for first, next, rest, cons functions.

Clojure collection quirks - Maps

Wed, 18 May 2016 11:00:02 +0000

Maps

Map data structures in Clojure are much like associative arrays in other languages. The abstraction is very handy and easy to use.

Creation

You can create maps with two syntaxes. Either by calling the hash-map function or by writing the {} map literal.

(def m1 {:one :a, :two :b, :three :c, :four :d})

One interesting fact when working with maps is that all maps with 8 or less entries maps will be created as a PersistentArrayMap an all larger maps will be a PersistentHashMap. The two differ only by the internal representation: small maps are better stored in an array.

Sequences

Calling seq on a map returns a sequence of its key-value pairs, called entries. They are instances of clojure.lang.MapEntry but they also act like vectors.

(seq m1)
  => ([:one :a] [:two :b] [:three :c] [:four :d])
(type (first m1))
  => clojure.lang.MapEntry
(vector? (first m1))
  => true

Access

You can acces a value for a given key in a map with the get function.

(get {:a 1 :b 2} :a)
  => 1
(get {:a 1 :b 2} :c)
  => nil
(get {:a 1 :b 2} :c :default-value)
  => :default-value

Maps also act like functions (they implement the clojure.lang.IFn interface). For example:

(map {true :even false :odd}
     (map even?
          (range 100)))
 => (:even :odd :even :odd ...)

Please note that calling a map on a missing key returns nil or the default value given as an optional third argument.

Modify

Use the assoc function to associate a new value with a given key. The result of the function call is a new copy of the parameter with a value replaced. The original map is kept intact.

Clojure collection quirks - Vectors

Tue, 17 May 2016 06:00:02 +0000

Working with Clojure collections is a pleasure. Yet, sometimes one may run into unexpected behaviors. In this series, we take a look at the most simple Clojure data structures.

Vectors

Vectors are sequential collections with constant time random access capability. One can also view vectors as integer to element mappings. Examples:

(def v1 [:zero :one :two :three])

(get v1 1) => :one
(find v1 1) => [1 :one]

Access

You can acces an element on the nth index with the get function or the nth function. Be aware that when a given index is not found the get function returns nil while the nth function throws an IndexOutOfBoundsException. You can specify a default value as a third argument that is returned when the index is not found.

(nth v1 1) => :one
(get v1 1) => :one
(get v1 -3) => nil

Vectors also act like functions (they implement the clojure.lang.IFn interface). An IOOBE exception is thrown when called on an uknown index.

(v1 1) => :one
(v1 9) => IndexOutOfBoundsException!!

Modify

We can also modify an item on an index of a vector:

(assoc v1 1 :One) => [:zero :One :two :three]

That’s right, one can change an item on an index of a vector just like with a map.

How is about removing an item from a vector?

(dissoc v1 1) => ClassCastException!!

You can not call dissoc on a vector because that could not remove an item without affecting the indices of all other items after the given index. To remove an item, first you need to split the vector into two parts (calling subvec) and then create a vector from the concatenation of the result.

(defn without [v index]
  (vec (concat 
    (subvec v 0 index) 
    (subvec v (inc index)))))

You can use the subvec function to return part of a vector.

(subvec [0 1 2 3 4] 3) => [3 4]
(subvec [0 1 2 3 4] 1 3) => [1 2]

Usage as queues

Vectors are often used as queues.

The conj function appends a value to the end of the vector.
The peek function returs the last item of the vector in constant time. (The last function is linear time thus much slower usually.)
The pop function drops the last item from a vector.

Sequences

You can turn a vector to a sequence with the seq function (note that (seq []) is nil). The rseq function returns a lazy sequence of items in the vector in reverse order. It is advised to favor rseq over reverse because of the constant time complexity and lazyness.

Indexed data structures in Clojure

Mon, 16 May 2016 10:00:02 +0000

When rumbling in the Java source codes of the Clojure language, I found the mysterious IndexedSeq interface. What is it good for?

Details

It provides the int index() method for collections to return the current index of the sequence on them. It extends ISeq, Sequential, Counted interfaces and thus IPersistentCollection and Seqable too.

Usage

One can easily create a wrapper class that gives an indexed functionality to any collection. The following example does just that.

(defn indexed
  "Creates an indexed data structure. You can access current index
  with the .index method call."
  ([s] (indexed s 0))
  ([s ^long init]
   (reify
     clojure.lang.IndexedSeq
     (index [_] init)
     clojure.lang.Counted
     (count [_] (count s))
     clojure.lang.ISeq
     (first [_] (first s))
     (next [_] (if-let [n (next s)]
                 (indexed n (inc init))))
     (more [_] (indexed (.more (seq s)) (inc init)))
     (cons [_ x] (indexed (cons x s) init))
     (empty [_] (indexed (empty s) 0))
     (equiv [this that] (= that s))
     (seq [this] (if (seq s) this))
     clojure.lang.Sequential
     )))

Testing

(def a1 (indexed (range 5 105)))

a1 ;=> (5 6 7 8 ... 104)

(.index a1) ;=> 0
(.index (next a1)) ;=> 1

(.index (take-last 2 a1)) ; => 98
;; ... etc.

Conclusion

The function shown can be used to wrap sequencial objects in order to keep track of the current index during iteration. This is a simple solution that comes handy when you would need to use map-indexed otherwise.