Seme | Stack | Space

We can think of applications as being the product of three main levels of abstraction-seme, stack, space. These levels organize the content, containers, and contexts of the application.

The decision to break down into three levels derives from two sources: Engelbart’s famous phrase "thought vectors in concept space”, and Hal Abelson/Jay Sussman's description of language as having “primitive expressions, a means of combination, and a means of abstraction”.

From these, we can map:

Pattern 1: Documents as Semes

A common pattern in many new media softwares is to use a notion of blocks as the primitive for constructing media documents. By linking these blocks together, one can create larger and larger media-scapes, in which associations and hierarchies exist as fundamental parts of the structure.

Instead of using blocks to describe this material, we’ll borrow a term from semantics—seme— the smallest unit of meaning. We can think of semes as the unit of discretization, or decomposition, in our documents. In order to express higher-level meanings, we construct structures and networks from these low-level blocks.

Modularity is the human mind's lever against complexity. Breaking down a complex thing into understandable chunks is essential for understanding, perhaps the essence of understanding.
Bret Victor

This idea is already familiar to us. We break our songs into choruses and verses, books into chapters, essays into paragraphs, movies and play into scenes, reports into sections. When we create media, we do so as a composition of parts. Blocks feel more ergonomic because they are more ergonomic.

In having a uniform type serving as the foundation for the database, we can create media with a consistent material. If we consider the idea of blocks in two types of softwares—coding “journals” like Jupyter Notebooks, and notation systems like Roam and Notion—we see how the block changes our interaction with the documents.

In Jupyter, we no longer have to run programs as a full unit, but instead we can decompose it into comprehendible parts, and run these in a serial fashion. We can intersperse our code with other media—text and graphics—in order to more appropriately represent the nature of the inputs and outputs of a program, as well as the program itself. Creating, deleting, editing and other interactions with these blocks are uniform, so we can apply the same patterns to different types of blocks.

In Roam and Notion, we no longer write documents as monolithic structures, but instead as the composition of small units of text or other media. This pattern of using nesting, links and transclusions allows us to fold our documents onto themselves, creating interconnected—rather than independent—experiences.

Blocks give a consistent interface of interaction. We understand a block, what we can do with it, how we can combine it with other blocks. To make an oft-made reference, these blocks serve in the same way as Legos, allowing us to build up and tear down creations of media units.

Pattern 2: Arrangements as Stacks

In addition to a system of blocks, we also need useful ways of visually binding these together into groups. To express meta-block properties like nestings, transclusions and links, it is not enough to have a backend that can represent these features—we need meaningful ways of presenting these groups of blocks on screen that emphasize and represents the structure of their connections.

What we need is not just visible connection, but a rational and extensible structure behind it-- a way of grouping parallel documents and managing their visible interconnections.
Ted Nelson

In dealing with digital objects—plural—we need ways of aligning and ordering them; mechanisms for selecting, sorting, filtering and searching them; ways of handling operations that can be performed on them; and controls for interacting directly with them by dragging and dropping.

The benefit of stacks is their ability to impose a dimensionality (x, y, z) on a set of objects, such that they live in alignment within a single container. We bind objects to some backing structure, such that we can have a consistent formatted display, and interact with the objects as a group for selecting and manipulating.

Briefly, we can consider the three dominate forms of stacks within interfaces: the list, the grid and the canvas. Of these, we can consider the list and the grid to be fixed, while the canvas can be said to be free, in terms of the layout of the contained elements.

We see the list in many forms: in timelines of content, file systems, and more. For much of the text-based media content on our devices, we prefer a one-dimensional stacking of rectangular blocks, like the lines on a page.

We see the grid in fewer forms, mainly to display media items like photos or videos, or to present a set of icon-based controls/content (like the iPhone home screen). We use grids to pack more items per unit of screen space, while we use lists to present fuller shapes for text-based media.

An alternative to lists and grids is canvases, or spaces in which objects exist in a free, rather than fixed state. While lists and grids allow us to hand over the task of lining up objects, canvases afford a higher-level of user control over the size and location of our objects.

For a heterogenous set of objects (in terms of their size and shape), this type of layout is preferred. Fixed interfaces perform best over homogenous sets of objects, though there are ways to handle a diverse set of object forms.

Pattern 3: Perspective Maps as Spaces

There has been a recent rise in discussions on spatial software, which has helped to awaken a sleeping dragon, as many have been recently re-inspired to design for the potential of digital

What is not lost in the transition from the art of memory to the art of interface design is most of all the dimension of mental space. The stage where the play is enacted is idea space, regardless whether the mental image is evoked by printed, pictorial or sound signs, or a Wagnerian multimedia Gesamtkunstwerk.
Volker Grassmuck

What I think is missing from these discussions is the making good of the “location” metaphor. If we have spaces, we have places. If we have places, we have maps. The act of using maps, either internal or external, to navigate spaces is central to the nature of interacting with them. We must consider the ways that we build abstractions over spaces, in terms of places, paths, landmarks, areas, and the way we might describe these objects with names, aliases, properties and classes. We abstract locations and sets of locations, building understandings through the locality and arrangement of these abstracted pieces, stitching them together into understandings of larger and larger landscapes. We might filter our views based on certain properties or classes, and construct the landscape in a reduced dimensionality.

If we are to create worlds, then so too must we create maps. It is not simply enough to give a user a space to exist in, you must also provide tools to document, explore, label, classify, spatialize—map— the current digital place they are in. It is also not simply enough to create a map, but as recursive trees. There is an adage in computer science, that all problems can be solved with another level of indirection. In a different way, all problems can be solved by nesting maps in maps.

An ideal map would contain the map of the map, the map of the map of the map, endlessly.
Alfred Korzybski

While there have been numerous offerings giving new forms of spaces on computers, most seem to miss on the fundamental privilege that computers afford—the ability to represent information in arbitrary shape and scale. Most new designs for information spaces are still fixed on a physics that does not maximize the abstracting and recursing power of computers.

I’d like to redirect attention back a few decades, to some more “primitive” designs, so to speak. Two such designs are Boxer and Pad++, each centering on the displaying of objects with multiple levels of abstraction.

Boxer

Boxer, researched by a team from MIT, Hal Abelson and Andrea diSessa, focused on representing “boxes” in hierarchical and recursive ways, similar to the way that LISPs represented information through nesting lists. Key to the design was the ability to nest boxes within boxes, such that each box represented a branch that could hold both leaves (media) and other branches (boxes). In their words...

All computational objects are represented in terms of boxes, which are regions on the screen that contain text, graphics, or other boxes. Boxes within boxes represent hierarchical structures.
Hal Abelson and Andrea diSessa

By now, people are well-aware of this notion of displaying information (computational objects) in terms of recursive containers through products like Notion and Roam. But I’d venture to guess that most understand these as vertically linear arrangements, a stack of “blocks”. Unique about Abelson and diSessa’s design was the use of both horizontal and vertical nesting, and the emphasis on interactivity. These were not meant to be solely media documents, rather they were designed to be dynamic, interactive, computational experiences.

Pad++

Pad++ was researched by the team of Ben Bederson and James Hollan, and continued the work of Ken Perlin and David Fox (Pad). Bederson and Hollan describe a “much richer world” of “dynamic persistent informational entities”. Central to their design is a semantic zoom, which reshapes objects based on how “close” they are to the user in “3D” space. We are familiar with this dynamic in our mapping technologies, which show more, specific and more complete, labels and features as we zoom in, and higher-level, more regional labels and features as we zoom out.

Pad++ is a natural substrate for representing abstraction of objects using what we term semantic zooming…The physics need to be designed to exploit semantic relationships explicit and implicit in information intensive tasks and in our interaction with these new kinds of computationally-based work materials.
Ben Bederson and James Hollan

Distillation: Maps as Arrangements of Media

Finally, we’ll end with a note on what these spaces should be for: to arrange our media such that it is easy to access and interact with. For the same reasons we use shelves to keep books, music or movies, we want similar interfaces in our digital worlds.

In one sense, it is enough at an individual-level to construct personal libraries of content, suitable for storing away one’s own records and notes. But in order to truly develop a meaningful hypermedia language, a Gutenbergian task, we must also consider the ways the humans connect and coordinate to communicate meaning, and the different artifacts that are shared to mediate this process. The task of design must privilege the artifact of hypermedia as maps, communicatory tools for meaning and representation. The goal is spaces in which one can do many things—interactive, computational and communicative things.

While the attention has recently been renewed, this pattern is not new.

The Greeks had made an art of it...the idea of using loci and imagines agentes (places and active vivid images) for dynamic, active storage of res and verba (things and words).
Volker Grassmuck
Not just hypertext, but graphics, multiple panes, efficient navigation and command input, interactive collaborative work, etc. An entire conceptual world and world view.
Doug Engelbart

The closest contemporary are apps like Muse and Kosmik. But all the new contenders I see miss on what I believe to be an essential ingredient. Hypermaps need to be able to different semantic levels within the same space. Projects like Pad++ and Archy by Jef Raskin allow for LoD (level of detail) mechanics, which allows you to view different levels of information in the same space. Instead, for the lot of apps I see offered today, the process of moving between levels is delegated to “embedded spaces”, in essence a more graphical form of hyperlinks on the web. This is not enough for me. Zooming in should reveal more information, and zooming out should reveal more compact information.

To send you off, I’d ask you to play around with your preferred maps app and imagine a media library underneath that. Maybe its your entertainment collection. Maybe its your notes. Maybe its your reference library, or your dashboard, or any other assemblage of your documents, files, media…your digital stuff.

Zoom in and out, and imagine what would grow and contract before you.