TITAA #32: Haunted Liminal Places
Phantom Settlements - Custom Text2Image Models - Zooming - Crazed CEOs
I’ve been introduced to “phantom settlements” (“not to be confused with ghost town or abandoned village,” notes Wikipedia) in the wonderful The Cartographers by Peng Shepherd. (Of which more below in the most important section, Books). Phantom settlements, also known as “paper towns,” are sometimes “copyright traps,” in which a fictional element is hidden in a map to expose fraudulent copying. Sometimes they’re just a mistake, like when an ink spot gets a spurious label. In printed works, where fictitious items might be added to a compendium of facts (—as a data scientist, this terrifies me—), these copyright traps are sometimes known as “mountweazels.” No, I’m not making that up.
The map in The Cartographers and the real map that inspired it shows Agloe, NY, which doesn’t exist—or does it? It does if you have a map to it, the book says. The map creates the territory. Historically, Agloe was a map trap, and then later there was a general store there and maybe even more of a town, a real place growing into the fictional place on the fictionalized map. The store and town are now gone, so now Agloe is a historical fictional real place. This article offers a good overview of the whole topic. I tried “Agloe, NY” in Google Maps, and got a location pin for the closest current real place, I guess.
The famous non-place Null Island has a buoy, by the way. So there is a real thing there, even if it’s not really the most visited place in the world. That fact about the buoy is just a data glitch. But maybe someone should build a floating resort there to cash in on all those visits. It would be a great cover story for someone on the lam: “Obviously I wasn’t at Null Island, that’s a well-known data error.”
Speaking of fictional maps, a couple months ago I forgot to bookmark and share the link to this amazing post analyzing map styles generated in Disco Diffusion by Mathieu Jacomy (@jacomyma). If you like RPG maps, satellite maps, network maps, and map styles, plus text2image promptcraft, go check it out.
AI Art & Tech Tools
Welp, lots going on here this month as text2image models are fine-tuned and custom models are released. Lots of research too, but let’s stay with the immediately accessible.
Zoomy Paintings
DALL-2 is in “beta,” meaning you now have to pay for usage. Sadly, there is no API, which would make some applications and tools so much easier and more fun.
My favorite DALLE-2 works are related to in-painting and out-painting. For instance, I love this Rube Goldberg generation zoom animation by Michael Becker. This Dragon Village video by DouweAndTonja is also very good. But don’t miss this powers-of-ten, zoom out and in animation by Adam Packard (watch to the end), which shows the prompts at each of 57 generation stages. (Last month I linked to an artist using EndlessPaper.app to make hand-drawn infinite zooms, @theorgana; here’s a charming French one about vacations, by @Vaskange. I’m currently “on vacation” in France, but I brought all my digital devices.)
A couple of people have published some tooling for making zoomy animations with the results from the hand-creation process in DALLE-2 — Michael Becker’s repo here has info and scripts and deKxi has a repo for a notebook-based set of tools for zoomy imagery. For info on how to do the out-painting and in-painting, see deKxi’s Guide to Outcropping with Photoshop and other image editors. The Dalle-2 Prompt Book has some info, too. And here’s an article by Douwe Osinga on how to use a python lib to stitch together infinite zoom pictures.
If you don’t want to use DALLE-2 for the out-painting, Robert Gonsalves has code and an article on using other open source tools (Dalle-mini and VQGAN) to do it.
Nicholas Rougeux has been making huge, intricate DALLE-2 pics of towns via in-painting; here’s a thread on how and a new cool vertical one with thread too.
Finally, Microsoft has announced NUWA-Infinity, which will also do text2image and text2video, but there’s no code yet.
Custom Diffusion Models
People are releasing custom trained diffusion models for special purpose art. This is a future I want, if only we could all, uh, afford to make these things.
Recent custom diffusion models with notebooks (all based on Disco Diffusion) for achieving different styles:
Floral Diffusion from jags
Liminal Spaces by BrainArtLabs
Fantasy Diffusion by lavista9008
Ukiyo-e Portrait Diffusion by avantcontra
KaliYuga_ai’s Medieval, Lithography, Watercolor, Pulp Sci-Fi, Textile models (her Pixel Art Diffusion was in last month’s newsletter, along with her article on how to train these things; she’s a hero).
The big downside of all these sweet custom models is the generation time and experience in a Colab, even with Colab Pro. It feels so slow, if you’re used to the hosted models like Midjourney or DALLE-2 or various bots. The results aren’t always better than a big general model; e.g., I tried Liminal and Midjourney with creepy rooms in haunted castles:
Midjourney is hard to make photoreal, by design; while Liminal really is built for generating creepy photos but the details feel less creative.
Big relevant news: There’s a poll up today on Midjourney asking how many images we’d be willing to rate to tune the model on our own custom style.
This is good - “AI Art is Challenging the Boundaries of Curation,” in Wired. It’s about the data sets as well as the models and the prompt craft process of creating with them, among other aspects.
New Big Models
The new, hot “#stablediffusion” appearing on Twitter is apparently a handful of different related models, trained off user ratings a là the Simulacra Aesthetic Captions dataset, under the funding support of Stability.ai. It is in testing on a private Discord server. I have tried one of the models, it is very nice at some things and is definitely different than both DALLE-2 and Midjourney. It defaults to photoreal with no style given, like DALLE-2 but unlike MJ.
Meanwhile, Midjourney just released a new model (“v3”) with various settings for stylization and quality; it’s very good. Just look at my dramatic penguins in Venice! (But the penguin in my game for Google Arts & Culture is nowhere near so Batman-like.)
Misc Other Creative / ProcGen Links
A good article on shape-packing for generative art in p5.js on Gorrilla Sun. Also see their article on making irregular grids which I really loved.
A lovely interactive article about shaders in React-Three Fiber.js by @MaximeHeckel.
Buildify, a plugin to Blender to make buildings, from Pavel Oliva. It will also work with the well-rated Blender-OSM plugin for making use of OpenStreetMap data for generating building footprints. To make, say, Paris.
Chris Ried did a good installment on 3D generative art tools, especially Blender, in his recommended newsletter.
This is a cool video exploration of using Midjourney to make textures and put them into Blender, by Curtis Holt. (I forgot this last month.)
Animating lots of three.js particles with GSAP, an Observable notebook demo by Dea Bankova. But it makes my laptop struggle a bit.
Emoji Kitchen mashups, a great post from Jennifer Daniel. I was yet again tempted to get an Android phone; but there are online websites and a Twitter bot too.
NLP & Data Science & Data Vis
Text Generation with Text Editing Models, slides for the tutorial at NAACL ‘22.
An Intro to Text Style Transfer and follow-on articles from Cloudera FastForward Labs.
How to Build Your Own GPT-J Playground, an article on using AWS, Streamlit and Hugging Face by Heiko Hotz. (Code) “The folks at Hugging Face have made it super easy, barely an inconvenience, to deploy a 6B parameters model like GPT-J on Amazon SageMaker.”
Holmes 4.0 is out and blogged about by Explosion. This is a smart info extraction and topic search toolkit for English and German, built on spaCy. For instance: “You can tell Holmes that a company takes over a company is the idea you are looking for, specify strategies for recognizing company names and synonyms of take over, and leave it to the library to find complex examples without having to write a large number of extra rules.” So good.
Visualize decision trees in Python, a lib by Stef van den Elzen. Would be good in classes.
Jupyter Scatter looks great. “An interactive scatter plot widget for Jupyter Lab and Notebook that can handle millions of points and supports view linking.” Demo examples use pandas.
Bulk, a cool visual clustering and “bulk” labeling tool from prolific Vincent Warmerdam. I use UMAP displays to look at text data clusters regularly, but adding in a way to lasso select and label the groups is a big win. I will definitely use this.
😅 I loved How I made Advanced BI Queries Child’s Play with Scratch Puzzle Pieces by Tommy van der Vorst. Evidently Scratch is a Big Deal, says a good article by Bryan Braun. Scratch is the “for kids” visual programming language where you use blocks to connect logic loops and make games and puzzles. Evidently the community online is now huge and thriving.
Games & Narrative
Liz England reviews Brian Upton’s Situational Game Design. “It looks at games as a series of situations (a moment in which a player makes a choice), moves (the set of actions a player can make), and constraints (which limit and inform their potential moves).” Also:
“One of the chapters defines the playfulness of a system (choice, variety, predictability, consequence, uncertainty, and satisfaction), and much later there’s a chapter on narrative that discusses how reading a book can be playful by using the same heuristics, and that because you can evaluate and design gameplay and narrative under the same framework they aren’t truly separate.”
Related to constraints, although in a different way, this is a fun article from Failbetter Games on their storycrafting system for Mask of the Rose.
A useful overview of Text Worlds (e.g., interactive fiction games) for embodied language understanding research by Peter Jansen. “This systematic survey outlines recent developments in tooling, environments, and agent modeling for Text Worlds, while examining recent trends in knowledge graphs, common sense reasoning, transfer learning of Text World performance to higher-fidelity environments.”
Designing Game Feel, A Survey article by Martin Pichlmair and Mads Johansen. (I forgot this last month.) For those interested in juiciness (and other UX attributes).
A good written-up thread on narrative and story in games, from a talk by Ernest Adams in Leiden a few years ago (by Christian Roth). Good reading for folks interested in AI narrative but may be old hat for some.
Emily Short reviews Hannah Nicklin’s Writing for Games book.
A good article on AI-assisted writing from Verge, focused on SudoWrite and some marketing copy tool called Jasper. As a fan of (and boggler at) the blurbs for cozy mysteries, I was especially interested in experience of cozy paranormal writer Leanne Leeds who sometimes works with SudoWrite. (NB, I like SudoWrite myself, but haven’t written anything significant with it yet. And I have a few more tools to try as they keep springing up.)
Off-topic, but Leanne and I exchanged a few tweets about her work and she tells me she is considered a slow writer, producing only 6-10 books a year, while “the bestseller in the genre produces 2 novels a month.” I spoke to a writer pitching a story on AI-assisted writing recently, and I more or less said, “There’s probably a lot out there that we don’t know about.” Which works, in a telephone game (see below), are going into the next training sets for text models. I wonder if mountweazels would help?
I’m unable to stop thinking about the demo video of a useful NPC in Minecraft from “Craft an Iron Sword: Dynamically Generating Interactive Game Characters by Prompting Large Language Models Tuned on Code,” by Volum, Rao et al. Code with demos, it’s all just promptcraft with Codex. This was presented at the Wordplay workshop where I gave a talk too. 👇
I gave an invited talk for the Wordplay workshop on text and games at NAACL 2022, and put my slides up here. My focus in the talk is on my personal attempts to make fun (and ideally funny) toylike apps using text and data and images, and the value of the glitch in the system that provides an interesting creative angle. There are discussions of games, story, toys, and types of fun in there. I think of my work as kind of AI-assisted mashups, more on the toy end than game end. The talk slides include lots of “telephone games” where narration moves from language to language, or image to language and back, using translation APIs, object recognition APIs, poetry and game data, captioning models, and text2image apps (D2, MJ, and DD). The fun of the toys is often in the “drift” due to how models see the world imprecisely.
Making personal AI art projects, I often run into these problematic tool issues, which I think afflict most AI artists right now:
The need for costly, memory-intensive compute (ok during dev, not so okay making apps for others to use)
Access to APIs for the best construction work (I have sadness that OpenAI hasn’t published a DALLE-2 API)
Ever-changing AI libs and hence version issues and dead repos (research and personal code is FRAGILE now)
The horrible framework & build world of javascript coding right now… and I theoretically know javascript.
Cost/difficulty of making personal, custom models. Although, right after my talk, KaliYuga_ai published 4 more custom models (see above), so that’s terrific!
But I promise the slides show fun projects in progress and fun bugs in games.
Books
The Cartographers, Peng Shepherd (lit fantasy). A fun read about a cartographer disgraced for wanting to take a junk driving map seriously at the New York Public Library, in what is called The Junk Box Incident. This alone warmed me to the book, before we even learn that her father and her deceased mother were part of a brilliant group of young grad school cartographers who discovered that this map — and others like it —has magical properties. (See intro.) Meanwhile, a reclusive, obsessive CEO is building a predictive maps product and wants access to all maps at the NYPL. This isn’t great literature, but it’s a fun read and I loved the map traps.
Locklands, Robert Jackson Bennet (fantasy). The finale to the Founders trilogy which I loved. I enjoyed this very much, although it has a lot of magicians throwing cities from the skies. All your Clef origin questions are answered. I cried at the ending. Maybe we could see the hierophants as crazed CEOs of (un)natural laws?
Ordinary Monsters, JM Miro (fantasy). This is on all the fantasy recs lists; it’s a fine Dickensian adventure novel about kids with weird powers and a couple very bad dudes, including a proxy CEO, trying to open a portal to the undead world. Some good heroines, too.
⭐ A House Between Earth and Moon, Rebecca Scherm (lit sf?). A couple years in our future: Climate-related disasters occur daily, there is a smartphone made by Sensus that is embedded and tracks your every move, and the obsessive CEO of Sensus is building a luxury space station and beta-testing it with scientists. Sensus sends up an observer to watch the scientists’ every move as input for building a predictive behavior algorithm. The main characters are a scientist on the station who is unable to replicate CO2-reduction results; his teen daughter, who is in trouble over a deep fake viral animation she made; and the reluctant researcher who gets hopelessly addicted to the reality show she observes for the algorithm. I was truly sucked in to the human stories here, and was 100% sold on the semi-functional, scruffy station that does not look at all like the brochures. My only complaint was the suddenness of the ending.
The Steps of the Sun, Walter Tevis (sf). Another crazy CEO story about an asshole getting fuel from other planets; I wrote it up here and have deleted the details because it’s not awesome. But good for completists, because Tevis is readable even when you hate the main character.
The Gradual, Christopher Priest (fantasy). A composer in a repressive, war-torn regime learns that people in the nearby, unmapped archipelago live by very different rules, including temporal ones. The slow (“gradual”) unfolding of his discoveries of the nature of these other places and their artistic meaning for him made a mystical, thoughtful vacation read. But it remains ineffable to the end.
TV
Still enjoying For All Mankind, despite the recurrence of the substance-abuse plot-driver and my mixed feelings about the not-really-that-nice CEO with the private space company.
Mostly this month I caught up on 2.5 seasons of Westworld, because I was in the mood for NPCs taking revenge and paranoia about who and what is real. And Yet More Crazy CEOs making predictive algorithms! I liked bits, and some of it was cathartic, but I’m now tired of the way guns are brought to wifi battles just to make them more interesting. I’ll stick out s4 to see what happens with game designer Dolores.
Games
Like the whole world, I’m playing Stray, the lost kitty game. It is gorgeous. I love that people are adding mods to play their own cat (WaPo article), but can do without the Garfield mod (although maybe it would make me less stressed when Kitty is in trouble).
I have also been playing Tentacular in VR, a cute game in which you are a kraken, or at least you are 2 tentacles and you are enormous. You must help tiny people on islands with various construction projects. The dialogue and design are funny and fun. You can splash around and throw things if you want (I do).
Poem
The room is
almost all
elephant.
Almost none
of it isn’t.
Pretty much
solid elephant.
So there’s no
room to talk
about it.
—Kay Ryan, “The Elephant in the Room”
Whew, another long one. Last month I floated that this was long and I think people don’t love that, but I got mixed signals. Someone said “just cover less,” which sure, would make it shorter. But maybe also less awesome. One person said, “Recommendo does six things,” which, er, would be a lot less awesome. OTOH, they publish once a week. Other friends said, “Why reduce? tell them to strap in and read faster.” Which I appreciate. If you made it this far, thanks for reading and stay cool.
All the best, Lynn / @arnicas
Keep the length, reading through it was fun! Thanks for posting your slides on Worldplay!
Most distressing part of Stray is 100% when the kitty is in trouble. (And another vote for keeping the newsletter exactly as long as the things you feel like putting in it!)