Rich Hickey on REPLs

Adapted from the Clojure mailing list.

Let's please keep this discussion to the proposed feature and its suitability to various tasks. I don't want this to become a discussion about nREPL.

REPL stands for something - Read, Eval, Print, Loop.

It does not stand for - Eval RPC Server/Window.

It is not something I "feel" about. But I do think we should be precise about what we call REPLs.

While framing and RPC orientation might make things easier for someone who just wants to implement an eval window, it makes the resulting service strictly less powerful than a REPL.

Let's look at some of that power:

'read' is a protocol. It works on character streams, one of the most widely available transports. It uses delimiters (spaces, [], (), {} etc) to separate forms/messages. Readers implement this protocol and return data upon encountering complete forms. read is shipped with Clojure and multiple implementations are available. It is available for use in programs, including programs you invoke in a REPL. A substantial subset of evaluable+read-able forms are supported by edn, which has even wider support. The reader format is extensible. So, when you get read, you are getting a lot.

When embedded in a REPL, even more is implied by read. There must be some source (stream) of things read. That source is the same source that should be used should the code being evaluated call 'read'. If that source is a terminal with a human, they can continue their interaction in place.

'eval' implies the full power of Clojure. Take a form as data, evaluate it, return a result as data.

'print' takes the returned data and prints it on an output stream as characters. It is true that not all things might print read-ably, which can pose challenges for programs sitting on the other end of a REPL that would like to call 'read' on its output. But we have improved Clojure in this area - now the default print-method prints something that is read-able (with an appropriate tag handler installed). Another area where many REPLs presume a human consumer and print un-read-ably is when exceptions occur. This socket REPL will default to printing exceptions read-ably, but you can switch to a human oriented formatter of your choice.

Again, as with read, when embedded in a REPL, even more is implied by print. There must be a destination (stream) of things printed. That destination is the same destination that should be used should the code being evaluated call print. If that source is a terminal with a human, they can leave their reading focus in place and see all output targeted towards them.

'loop' - do it again. Note however that a REPL will not attempt to read/print while evaluating.

The stream binding and nestability is important. It means that you can, in the code you evaluate in your REPL, launch a nested interpreter, or Eliza-like thing, or a text-adventure etc, and those things will get input from the same source and print to the same target. This is what someone who knows what REPLs are expects and demands.

You can, if you desire, even launch an eval RPC server in a REPL, but not vice-versa.

REPLs work over the most primitive connections, terminals, and can be pipe, rerouted, etc with all tools. They accomplish a lot over a single connection, which will often be all you can get.

But, we now have nice editors/IDEs often running on the same machine, with the ability to use sockets. Multiplexing their needs with those of a human REPL consumer over the same connection is going to make things bad for one or both of them.

With the socket REPL, the recommendation is that tools will open more than one connection, and use a dedicated channel for most editor/IDE operations that might want to interact with the running process. With the improved read-ability of REPL output, I think an edn-based REPL makes a fine RPC server for program-to-program communication, especially since the set of 'verbs' supported is bounded only by what you can evaluate, and the types of things you might discover is open.

These are the intentions of this feature.

Thanks for the feedback,

Rich

I think it is important, with tooling, to have the broadest, open notion of the types of 'use cases' for Clojure, and for workflows of people using Clojure, lest you actually limit them. I hope people are able to write their languages and DSLs, test embedded control/debug consoles, design next-generation REPLs and debuggers, grab data dumps etc, all from their REPL. That's the heritage of Lisp. Many people are doing data science, languages, genetics, rule-systems etc type things with Clojure, not just web apps.

In any case, it's great to hear support might be coming. I think you should also question your presumptions about needing to completely give up your Cursive functionality in such a REPL. You can examine/prepare anything you want on the way to sending it over the REPL. You can toggle per-character and per-form transmission. People will only need the former when utilizing a non-reader-based stream consumer. You can package any form in a wrapper expression that wraps the result in something you can detect on the return, thus being able to distinguish responses to IDE-issued requests from other output. You can provide IDE controls for switching namespaces that allow you to track that. Etc. Of course you need do little more than support eval-form and load-file, while staying out of the way of the streaming, in order to win me over from inferior lisp mode :)

As far as the namespaces, again yes, it is important to think broadly about the power of multiple sessions, even from the same user. At first we are talking about 2 sessions, for human and IDE. But it is quite common in rich Common Lisp IDEs like Lispworks for people to have multiple listener (REPL) sessions open at the same time (against the same runtime). You can launch long running processes that produce streaming output, switch to another listener and continue interaction and development, have separate listeners for separate contexts/state/command-history etc. There's a lot to Lisp development that's not RPC. And of course a socket REPL server could support multiple simultaneous users, where 'users' might be programs.

Have you ever worked with an environment like Lispworks or Allegro? If not, I would definitely recommend trying it. I like IntelliJ, and am a long time customer, but it still can't touch some aspects of the experience of a rich Lisp environment. IntelliJ is much more of an editor/code management system than a live programming environment. Hopefully Cursive won't settle for giving Clojure programmers the power of Java :)

Note multiple listeners, data inspectors and more. Maybe not the latest GUIs, but the capabilities should be a source of inspiration for any Clojure IDE.

We have options and tradeoffs around ns per session and are considering them. Sharing by default can lead to conflict by default. Having session/ns affinity allows for automatic cleanup, else that also becomes a user problem. People who want to create namespaces that survive sessions can do so, and can then manage them. The biggest argument against per-session namespaces might be memory use in constrained environments. Even then, there would likely be at least one user ns, so it might just be a matter of configuration to ensure that the first socket REPL session creates it. Still thinking about this.

If you think users will want to end up in 'user' every time, just have Cursive send an in-ns on connect before handing it over to the user. But please make that something I can turn off :)

I'm not trying to set your priorities, and I know you are working on this (thanks!). But I will continue to object to this characterization of which people require nested read and the things it can be used for. It is not about text adventure games. I've given several serious use cases, including languages and DSLs and control/debug interaction prompts.

Here's a wonderful demonstration of the power of nestability:

"Programming Should Eat Itself" by Nada Amin

And it's not just about hardcore Lispers who know its heritage.

Consider students. Here's one of my daughter's first high school programming assignments:

prompt user for their gender read the gender prompt user for their height read the height print "your ideal weight is:" the result of a calculation using gender and height

In Java this requires defining, compiling and running a whole program with a main. How can we claim that Clojure is dynamic and interactive if we can't run this in a REPL, and test its component functions there?

===== Connecting to local nREPL server... Clojure 1.7.0-master-SNAPSHOT nREPL server started on port 53446 on host 127.0.0.1

(defn homework \[\] (let \[_ (println "enter your gender (m/f):") gender (read) _ (println "enter your height:") height (read)\] (println "your ideal weight is" 42))) => #'user/homework

(homework) enter your gender (m/f): DEBUG: unknown status need-input

(defn homework [] 

  (let [_ (println "enter your gender (m/f):") 

        gender (read) 

        _ (println "enter your height:") 

        height (read)]

    (println "your ideal weight is" 42)))

Note that nested interactive input has 2 modes - form-oriented 'read'ing and arbitrary eager character-by-character reading.

Ideally, you would continue to maintain form orientation (allowing completion and structural editing) as a default. Structural editing and form correctness checking will still be a win for those whose nested reader is calling 'read' (which won't care if the form was fully prepared before being sent). Only in a (necessary, but could be primitive) mode would you have to allow character-granularity transmission, for arbitrary stream-consuming functions.