λ-2D: An Exploration of Drawing as Programming Language, Featuring Ideas from Lambda Calculus

The area of non-verbal programming languages has not been unexplored. There are ASCII-based languages such as Befunge and asciidots, as well as image-based ones such as Piet, just to name a few. Both inspired and challenged by these work, I set the following goals for my new language:

• To take advantage of the fact that the program is drawn, to include features that is otherwise unfeasible with text-based programming languages.
• To have neither too few instructions, nor too many: For the former it becomes laborious to construct even the simplest programs, and for the latter it becomes non-minimalistic and difficult to do computer vision on.
• To be able to draw programs that look visually appealing themselves, such that someone would want to put a frame around them and hang them on the wall.

Though I'm normally an imperative and low-level person, I was compelled to use lambda calculus, or functional programming in its most primitive form, as the basis for the language, for it seemed to me that the ideas bear a lot of resemblance to a typical drawing. It has no concept of "execution" but only that of "evaluation", much like how the viewers' eyes can linger on any part of a drawing in no particular order, and by following many dots they see the line, and lines the form, and forms the composition.

Conceptual blabber aside, I started with a grid-based system. While the user can draw continuous lines across many grids, each grid end up as one of a finite set of symbols. It seems like a good compromise between the ease of drawing for the human and that of parsing for the computer.

Lambda calculus is such a concise language that it only has two instructions: that of function application and that of function definition. I quickly came up with working symbols for each: a "cup" shape for the former (for the silly intuition that applying the function is like putting the argument into the "cup"), and the eponymous greek letter for the latter (which is a bit unimaginative and arguably overused, but at least it's clear). Just like vanilla lambda calculus, functions always take one argument and produce one output; to get more, you can chain multiple functions together, known as "currying".

Then came the wires to connect the symbols and through which data can flow. Technically the language is Turing complete at this point, but will be excruciatingly laborious to use, violating my design rule #2. Therefore, I added a lot more other symbols you would expect from your favorite programming languages, things like numbers and math operators. Consider them mere syntactic sugars: you can still stick to Church numerals (they're pretty cool) and other "pure" lambda calculus constructs if you'd like.

I used to enjoy how easy it is in Scratch, to draw some sprites in the same editor and immediately use it for the program. Since my new language is entirely drawn, it should be even more natural to incorporate such kind of feature. Additionally, I wanted to be able to sketch the shape of a mathematical function and use it (e.g. for animating stuff), without the extra step of figuring out the equation (manually or otherwise). So I introduced the idea of "frames": fence any area of the canvas with wires, and put an indicating symbol at upper left; anything doodled in the area can be used as data. In a similar spirit, I wanted to be able to draw sliders (and perhaps other GUI elements in the future), which can be dragged at run-time to parametrically control the program.

Initially I sketched my ideas on a dotted notebook, trying to construct some example programs in my (then imaginary) language. (I found this approach useful in my previous programming-language-design experiences). Then I figured it was time to code a parser for it, to see if it really "works". Since the computer vision part to scan the program from the paper was not ready yet, I decided to first make a simple editor software to let the user digitally draw programs.

Each symbol is made to 5x5 pixels, conveniently stamp-able on the gridded canvas, while the user can also draw "free hand" with a "pencil"-like tool. What initially started as a temporary measure grew to an almost full-fledged editor with many features. 

One interesting problem that I did not anticipate while imagining the language was that it turned out so purely functional and absolutely state-less, that it becomes impossible to implement a "print" statement, for to print is to change state, to expect some things to be printed in some particular order is to assume that some expressions will be evaluated in some order. The solution was a functional re-thinking of the definition of "printing" as passing a piece of empty canvas to some function and receiving a new canvas with altered pixels resembling text (or whatever scribble desired) on it. (Replace "canvas" with "string" and "pixels" with "characters", if you wish, but you probably figured having read thus far that this is an anti-strings and pro-pixels language).

The baseline parser works by transpiling (translating) an entire λ-2d program to a javascript equivalence. The resultant javascript one-liner is a single horrendously inscrutable mega-expression that contains enough parentheses to make a lisper shudder. But the coolest part about it is that it works (albeit inefficiently)!

I work to improve the language by constructing more example programs, and as I do I discover design flaws to be corrected. It was a lot of fun, for I am myself unfamiliar with my own creation: I only know the base rules, and that in theory it should work, but as to how to actually program in it I am as clueless as any other new learner of programming languages. Gradually I start to grasp its temperaments, of what it is like to program in this very strange, drawing-based language. In the beginning I was using the syntax clumsily, trying to bend things to my will; later I become more artful and expressive in it. Coding in λ-2D is somewhat like playing Minecraft or Factorio, but it's even better because I can call it research.

Below you can the comparison of two fractal tree programs, one in notebook doodles, and the other in refined digital form. (Please forgive the copious bugs and logical inconsistencies in the former, for I know people who can program on a piece of paper and get it right in one shot, unfortunately I'm not one of them and learned programming the "rogue" way: by running stuff over and over again and see if I can get any errors to pop up).

I thought it must be cool and wondered what it would look like, to visualize the execution of a program written in this unusual language. The current parser spits out javascript and your browser's super-optimized javascript engine takes it over from there, so it is difficult to visualize the actual execution. However I can easily visualize the parsing, which should look similar to the path taken by a tree-walk interpreter executing the program.

And the animation did turn out quite fun to watch. But what if it makes sounds when going over different symbols? We can then "listen" to a program as it is being run, as if it were a song! I'm no musician myself but theoretically it should be possible to compose something musical with this kind of system.

It ended up sounding like a whacky computer game from 8-bit era. You can check it out in the online demo (Menu > Program > Animated Run).

λ-2D started as a part of a larger research to design a system where the user draw programs with pen and paper, and receive interactive feedback through augmented reality. However, it grew increasingly interesting that it became a full project on its own.  Though I'm quite proud of this neat little language, it is yet to fully meet some of the initial goals. For instance, the programs look too much like circuit diagrams and not enough like, well, drawings. Also, I'm not too optimistic about how easy it is for a human (excluding myself) to learn it, and for a computer vision system to scan it without error.

Therefore, after I refine λ-2D, I plan to design more potential programming languages that can be incorporated into the drawing-as-computation system I am developing, using knowledge and experience I've since gained.

You can try out a beta version of λ-2D online here. The source code for the parser and editor will shortly be available on GitHub.