Last month, I went to a concert at Artists Space by Yasunao Tone (*1935), a Fluxus artist known for his experimental, glitchy compositions. Even the analog pieces performed that night were less like songs than programs: “The Seminar on the Purloined Letter” (1998), for example, improvises an interpretation of a series of Lacan-inspired diagrams for clarinet, guitar, and trombone. “AI Deviation II” (2015), developed in collaboration with a team of machine-learning researchers, responds in real time to input from a laptop camera, which Tone, seated in front of the audience at a small table, swiveled around to point at us – rather mischievously, I thought. Indeed, even through earplugs, the sounds produced by his silver computer were amazing, but also loud and piercing, like sonic acupuncture. The crowd at Artists Space was really, really into it: people were literally jumping up and down, hungry to interact with the noise. Although it was cool to watch the audience (many of them, like Tone, possibly in their eighties) wave around their iPhone flashlights, I honestly couldn’t tell you whether, or how, their gestures affected the music. The interactivity and community built into liveness and shared space are part of the appeal of any concert; another is the opportunity to see an artwork at work : to track a musician’s movements and map them onto sounds.

When those sounds are made by software, of course, there isn’t much to see; though I was close to the stage, Tone’s laptop screen was too small for me to observe in action. But at some point, I noticed the huge disco ball suspended a couple yards above our heads. It wasn’t spinning, and the space was well-lit, so it didn’t throw off much light. But the many mirrors tiling its curved surface still reflected the room, reducing us to tiny squares of color, like pixels wrapped around a sphere. As people squirmed in their seats, I could see the mirror-pixels changing color, responding to their inputs. It was a beautifully synesthetic interface for the performance: the shimmering ball, being played as a visual instrument, producing the kind of effect that takes hours to replicate digitally. Instead, thanks to certain built-in behaviors of the Artists Space basement, like the laws of light and optics, the device worked perfectly – no HDMI cables necessary.

Yasunao Tone: Concert #3, performance documentation, 16 March 2023, Artists Space, New York. Courtesy: Artists Space. Photo: Destiny Mata

This is basically the idea behind Folk Computer, a small human-computer-interaction (HCI) research project run out of a converted auto shop in East Williamsburg, Brooklyn, that I visited one afternoon in March. Its founders, Omar Rizwan and Andrés Cuervo, want to make an interface that is haptic and three-dimensional: no longer an image of a desktop, but a real one. Folk’s hardware consists of a tiny, monitor-less Intel computer, a ceiling-mounted camera, a projector, a printer, and lots of paper spread out on an IKEA table. Each sheet is printed with a QR-code-like marker (called an AprilTag) that corresponds to a program written on a laptop and stored on the Github cloud. The camera reads the tag and executes its instructions, which, for now, are fairly simple – “draw a green box,” for example, causes the projector to cast four green lines onto the table. Folk is part of a larger “screenless” trend in HCI, but whereas Elon Musk’s Neuralink would sew electrodes activated by proprietary software into brain tissue, Folk would weave computation into our external environment. One day, the system should be able to control a robot, produce a song, manipulate masks in Photoshop, visualize results from a physics experiment, or spatialize an architectural blueprint. Their philosophy is straightforward, but quite radical: that humans think best when manipulating real objects in real space, rather than looking at small symbols on two-dimensional surfaces.

Another crucial Folk design principle: that users exercise actual control over their technologies. When the project goes open source later this year, anyone could theoretically set up a camera-assisted workspace of their own and tailor it to whatever project they’re working on. The code on which the Folk operating system runs is just a couple thousand lines, making it easy to adapt and expand, while its programming language is simple and the physical materials composing its interface are familiar ones. And because each piece of a larger program is self-contained on a sheet of paper, its effects are easy to visualize and understand – everything is literally out on the table. So even non-coders (or so the thinking goes) could master Folk much more easily than a programming language, just by interacting with it casually: moving stuff around, IRL, maybe with friends.

Folk owes much to Alan Kay, the computer scientist whose pioneering work on “window”-based “desktop” interfaces inspired the Apple user interface that would first make personal computers accessible to a wide range of consumers. “Although thinking goes on in one’s head, external media serve to materialize thoughts and, through feedback, to augment the actual paths the thinking follows,” writes Kay in “Personal Dynamic Media,” a 1977 paper, cowritten with Adele Goldberg, that enthusiastically describes the computer’s potential for interactivity and open-ended functionality as a “metamedium,” or a device that can simulate all other media technologies Fifty years later, a handful of Apple products have indeed replaced, or subsumed, almost every form of media: The software that runs on them can simulate anything from a mechanical egg timer to an orchestra.

We could continue to asymptotically approach replacing physicality with its VR reproductions. We could also continue to augment our “reality” with Snapchat or Instagram filters that place dancing hot dogs in videos, when they’re really just augmenting our iPhone cameras. But why spend so much energy engineering drop shadows on buttons, or drag-and-drop gestures, or animations that look as fluid as we actually feel? As HCI-visionary Bret Victor (the founder of Dynamicland, a project to which Folk owes much of its design and philosophy), likes to say: “in the real world, you get all that for free.”

A table of hardware during a workshop at Folk Computer, Inc., Brooklyn, March 2023. Courtesy: Omar Rizwan and Andrés Cuervo

Right now, Rizwan and Cuervo’s research is focused on getting their sensors to recognize images and radio-frequency identification (RFID) tags on 3D objects, rather than just black-and-white codes on paper. This would allow any object to become a Folk programming tool, radically expanding the project’s aesthetic possibilities. You could put an RFID on your cat’s collar or your coffee cup. If you really wanted to, you could put a dancing-emoji sticker on your hot dog and have the computer play a dance mix whenever you wiggle it around (a stupid example, to be sure, but “stupid” is what made Snapchat a billion-dollar company). The point is that, with this leap into “animating” real objects with computing power, Folk is a fully realized metamedium, in the literal sense of the word “meta”: a beam of light hovering above objects and connecting them to existing digital media, imbuing rather than replacing them with just a little more interactivity.

As for me: Being both an object-oriented writer and a collector of pointless, beautiful things, I like the idea of enchanting my possessions with an extra layer of information, agency, or meaning, a translucent glaze of functionality applied, like lip gloss, to the surface of things. If I had Folk, I think I would turn my bedspread into an interface for the apps I use most – Weather, Instagram, Adobe Acrobat – controlled using my satin hair ribbons and cool Death Valley rocks.

Executed instructions during a workshop at Folk Computer, Inc., Brooklyn, March 2023. Courtesy: Omar Rizwan and Andrés Cuervo

Shortly after visiting Folk, I went on a tour of the Emily Dickinson House, the famously reclusive poet’s historic mansion in Amherst, Massachusetts. The museum recently underwent a massive renovation, having received over 300 props and costumes from the set of Dickinson (2019–21), an Apple TV+ series starring Hailee Steinfeld as a fun, bisexual Euphoria -esque Emily. The tour guide compulsively referenced the show: “Apple, Apple, historical, Apple, historical,” she’d say, pointing at curtains, lamps, newspapers, desks, and faded daguerreotype portraits that were actually produced in 2020. It was strange, walking around these simulations of objects the poet once really used. I imagine that Dickinson would have developed a particularly strong personal relationship to the specific “external media” surrounding and enabling the realization of her thoughts: hand-sewn paper chapbooks and inkwells, maybe, but also plants and rolling pins, white muslin dresses, the tiny cherrywood writing desk facing Amherst’s Main Street. Indeed, many of Dickinson’s poems use objects to symbolize her work. My favorite begins:

I dwell in Possibility –
A fairer House than Prose –
More numerous of Windows –
Superior – for Doors –

Here, Dickinson sets up a contrast between Prose and the “fairer” form of language in which she dwells: Poetry, with its multitude of interfaces by which to view the outside world (Windows) and different structural or syntactical mechanisms for navigating the range of meanings writing can produce (Doors). Poetry, moreso than prose, permits one-to-one substitutions of language (here, “Poetry” with “Possibility”), enabling the poet to make fuller use of language’s Possibilities, hacking words and bending grammar to create new meanings. Poetry, envisioned simultaneously as a built environment or an operating system, simulates realities both real and imagined. Dickinson, whose experimental style and syntax (unorthodox capitalization, weird rhythm, lots of dashes) were way ahead of her time, paid a programmer’s attention to punctuation and line breaks. She even devised a (very digital-looking) notation system or handwriting interface: Her pre-edited manuscripts are often marked with “+” signs that denote a series of multiple choices for some words. Positively defined, indeterminate, and unreliant on absence or negation for meaning, the + not only documents her creative process, but suggests an astonishingly open-ended relationship to semiotics. Just as Dickinson wrote her own recipes – her rye loaf was prize-winning – and choreographed her own, idiosyncratic way of living among things – like a specially rigged basket for dropping gingerbread to children outside – she also customized language itself.

This, to me, is Programming. In fact, a more technical way of describing Folk’s open-endedness is to say that it is a programming language, not just an interface. If Dickinson invested the windows and doors of the real architecture she inhabited with multiple immaterial “meanings,” then Folk, like her House, is a (literal, physical) space of Possibility for users to determine the functions of the words and objects that compose their worlds.

Walking through Dickinson’s real-yet-reconstituted house, I trailed my fingers over the shiny sofas. The psychedelically patterned floral wallpaper looked new and garish, like something onscreen. The house began to feel less like the home of a real dead woman than a hallucination produced by a TV show, or a palimpsest of dying media forms: Netflix; television; the Springfield Republican, which first published many of Emily’s poems. Then again, I thought, why not? Dickinson didn’t care for conventional grammar; why shouldn’t Apple give her a lesbian love affair and a technicolor carpet? “Think different,” indeed. If there was a ghost haunting the house, it wasn’t Emily, but Possibility, and all the potentially inaccurate televisional interpretations that come with it: a multiplication of possible past lives, of probable meanings given to facts.

Dickinson Bedroom at the Emily Dickinson Museum, Amherst. Courtesy: Emily Dickinson Museum. Photo: Kaelan Burkett

My favorite part of the tour, a curious analog device set up for visitors to explore Dickinson’s poetry, expressed a similar sentiment. The display was basically a wooden board printed with the poem “A chilly Peace infests the Grass,” only with a couple of sliding knobs that could change the words Emily had marked with “+,” suggesting a set of other possible word choices. Moving these knobs, so eerily reminiscent of digital sliders, I made “chilly Peace” become “lonesome Peace” become “warning Peace.” It was a rare, embodied way to interact with the technology or medium I know best: words. It was like feeling language at work, meaning being made in real time. It was like Yasunao Tone’s disco ball. Honestly, engaging personally with Folk has been challenging; although I’m intrigued by the design philosophy and see how the system could transform pretty much everyone else’s work – architecture, early education, meteorology – I have often found it hard to imagine how it might be applied to my own. Writing, after all, might be the most “virtual” medium out there: Reading basically entails imputing a kind of holographic world into a small black mark; its very power comes from this immateriality. But in Emily Dickinson’s AppleHouse, as I switched “astray” to “abroad,” noting the way each adverb changed the read of the line, the endless possibilities of that tool I use daily took on, for the first time, a physical form. I felt writing come alive under my hands.

Poetry Board at the Emily Dickinson Museum, Amherst. Courtesy: Emily Dickinson Museum. Photo: Patrick Fecher

___