XTDB / Jul 16 2019

Graphing with Crux

Remix (fork) this in Nextjournal to get started with your own Crux notebook.

Introduction

Datalog is a powerful query interface that allows you to elegant express queries to retrieve information from the graphs of information contained across your documents.

"Imagine a user being part of different groups. A group can have different roles, and a user can be part of different groups. He also can have different roles in different groups apart from the membership. The association of a User, a Group and a Role can be referred to as a HyperEdge. However, it can be easily modeled in a property graph as a node that captures this n-ary relationship..."

This notebook example is originally based on data and queries from the Neo4J Cookbook, where a visual illustration of the sample graph can be viewed: https://neo4j.com/docs/stable/cypher-cookbook-hyperedges.html

Assuming you have some basic knowledge of Clojure, all you need for this notebook walkthrough is to add Crux to your deps which Nextjournal makes very simple. For more configuration details see here.

{:deps
 {org.clojure/clojure {:mvn/version "1.10.0"}
  org.clojure/tools.deps.alpha
  {:git/url "https://github.com/clojure/tools.deps.alpha.git"
   :sha "f6c080bd0049211021ea59e516d1785b08302515"}
  juxt/crux-core {:mvn/version "RELEASE"} ; "RELEASE" is the latest non-snapshot version from Clojars
  juxt/crux-decorators {:mvn/version "RELEASE"}}}
deps.edn
Extensible Data Notation
(require '[crux.api :as crux]
         '[crux.decorators.aggregation.alpha :as aggr])
11.7s
Clojure
Clojure

Create a Crux node

(def db-node
  (crux/start-standalone-node
    {:kv-backend "crux.kv.memdb.MemKv"
     :db-dir "data/db-dir-1"
     :event-log-dir "data/event-log-dir-1"}))

; alternatively, you can go with RocksDB for a persistent storage
(comment
  juxt/crux-rocksdb {:mvn/version "RELEASE"} ; add this to your deps
  ; define node as follows
  (def node
    (crux/start-standalone-node ; it has clustering out-of-the-box though
      {:kv-backend "crux.kv.rocksdb.RocksKv"
       :db-dir "data/db-dir-1"})))
5.7s
Clojure
Clojure

Define a graph

0.4s
Clojure
Clojure
(def nodes
  (for
    [n [{:user/name :User1
         :hasRoleInGroups #{:U1G3R34 :U1G2R23}}
        {:user/name :User2
      :hasRoleInGroups #{:U2G2R34 :U2G3R56 :U2G1R25}}
     {:role/name :Role1}
     {:role/name :Role2}
     {:role/name :Role3}
     {:role/name :Role4}
     {:role/name :Role5}
     {:role/name :Role6}
     {:group/name :Group1}
     {:group/name :Group2}
     {:group/name :Group3}
     {:roleInGroup/name :U2G2R34
      :hasGroups #{:Group2}
      :hasRoles #{:Role3 :Role4}}
     {:roleInGroup/name :U1G2R23
      :hasGroups #{:Group2}
      :hasRoles #{:Role2 :Role3}}
     {:roleInGroup/name :U1G3R34
      :hasGroups #{:Group3}
      :hasRoles #{:Role3 :Role4}}
     {:roleInGroup/name :U2G3R56
      :hasGroups #{:Group3}
      :hasRoles #{:Role5 :Role6}}
     {:roleInGroup/name :U2G1R25
      :hasGroups #{:Group1}
      :hasRoles #{:Role2 :Role5}}
     {:roleInGroup/name :U1G1R12
      :hasGroups #{:Group1}
      :hasRoles #{:Role1 :Role2}}]]
      (assoc n :crux.db/id (get n (some
                                   #{:user/name
                                     :group/name
                                     :role/name
                                     :roleInGroup/name}
                                   (keys n))))))

(crux/submit-tx
  db-node
  (mapv (fn [n] [:crux.tx/put n]) nodes))
0.4s
Clojure
Clojure
Map {:crux.tx/tx-id: 1600811123794945, :crux.tx/tx-time: function Date() { [native code] }[Tue Jul 16 2019 15:48:33 GMT+0000 (UTC)]}

Find roles for user and particular groups

(def db (crux/db db-node))

(crux/q db '{:find [?roleName]
             :where
             [[?e :hasRoleInGroups ?roleInGroup]
              [?roleInGroup :hasGroups ?group]
              [?roleInGroup :hasRoles ?role]
              [?role :role/name ?roleName]]
             :args [{?e :User1 ?group :Group2}]})
0.5s
Clojure
Clojure
Set(0) #{}

Find all groups and roles for a user

(crux/q db '{:find [?groupName ?roleName]
               :where
               [[?e :hasRoleInGroups ?roleInGroup]
                [?roleInGroup :hasGroups ?group]
                [?group :group/name ?groupName]
                [?roleInGroup :hasRoles ?role]
                [?role :role/name ?roleName]]
               :args [{?e :User2}]})
0.2s
Clojure
Clojure
Set(0) #{}

Define a Datalog Rule

;; a datalog rule
(def rules '[[(user-roles-in-groups ?user ?role ?group)
              [?user :hasRoleInGroups ?roleInGroup]
              [?roleInGroup :hasGroups ?group]
              [?roleInGroup :hasRoles ?role]]])
0.1s
Clojure
Clojure
user/rules

Find all groups and roles for a user, using a Datalog rule

(crux/q db {:find '[?groupName ?roleName]
               :where '[(user-roles-in-groups ?user ?role ?group)
                       [?group :group/name ?groupName]
                       [?role :role/name ?roleName]]
               :rules rules
               :args '[{?user :User1}]})
0.3s
Clojure
Clojure
Set(0) #{}

Find common groups based on shared roles and count the number of shared roles

This uses the aggregation decorator, which wraps the default `crux/q` and looks for `:aggr` instead of `:find`

(aggr/q db {:aggr '{:partition-by [?groupName]
                    :select
                    {?roleCount [0 (inc acc) ?role]}}
            :where '[(user-roles-in-groups ?user1 ?role ?group)
                     (user-roles-in-groups ?user2 ?role ?group)
                     [?group :group/name ?groupName]]
            :rules rules
            :args '[{?user1 :User1 ?user2 :User2}]})
0.2s
Clojure
Clojure
Vector(0) []

What's next?

Try adding additional :hasRoleInGroups values (e.g. `#{:U1G1R12 :U2G3R56 :U2G1R25}`) to :User1 by submitting a new version of the document

Learn more at https://juxt.pro/crux