TITAA #57: Invisible & Generated Cities

Mythogeography - Gen Video - Game Poetics - Various Venices - Search Tools

Artwork from Karina Puente for Invisible Cities

Housekeeping before a lovely intro post: I’m moving the TOC to the top, to make it easier for people who want specific section jumps. And I’m rearranging the recs down to the bottom, because supporters get a separate mailing of them in more detail, and can therefore bail at that point!

• Thoughts on generative & fictional cities via Invisible Cities and mythogeography resistance

• AI Arty News (3D Places, Video, Creativity/Narrative, Misc AI)

• Misc Procgen/Web/Tools

• Games News

• NLP & Data Science & Vis

• Recs (Books, Games)

• A Poem

Thoughts on Invisible Cities & Drifting

If you’ve been following for a while, you know I’m a fan of odd maps (e.g., one post), psychogeography (another post), and generative and narrative landscapes, in and out of books, games, and art. This past week I revisited Calvino’s Invisible Cities, a collection of travel narratives about fictional cities as told by Marco Polo to Kublai Khan. The book is fantastical, a travelogue for a would-be conqueror of fictional lands, a critique of tourism, and a commentary on memory and language. It’s plotless, but the cities themselves are narratives. Kublai says, “Tell me another city.” We can assume travel is endless and circular—it starts in mid travel, with a dangling referent: “Leaving there and proceeding for three days toward the east…”

There is a host of academic literature on the book (e.g., this search page). The book is a postmodern classic. Marco Polo tells Kublai Khan about all the cities he’s seen (or not), and yet: “Every time I describe a city I am saying something about Venice.” Kublai asks, “Is what you see always behind you?” or rather, “Does your journey take place only in the past?”

You will learn a little about it from this english wikipedia page, which is a terrible link; the writer was evidently obsessed with an opera set in a train station based on the book. It links to a lot of dead or out of date pages, including a very old blog post of artistic representations of the cities, most of which are 404s. The excellent artist in my header is not there. Has the Internet become one of Polo’s cities, dissolving and mirroring itself in various AI training sets and the Wayback Machine?

“To make the leap from life to death less abrupt, the inhabitants have constructed an identical copy of their city, underground. [But… ] actually it was the dead who built the upper Eusapia, in the image of their city. They say that in the twin cities there is no longer any way of knowing who is alive and who is dead.”

I am a huge fan of the labyrinth of Venice, and 5 years ago I did a Nanogenmo project, Directions in Venice, using Tracery gammars plus GPT-2 trained on public domain historical Venice guidebooks because: “When I was in Venice in mid-November, my AirBnB host tried to give me directions to some places he liked using a paper map. Every time he circled a square and paused, I thought, "This is where it is," and it wasn't.” They all start in media res, too, and I am eerily reminded of Calvino’s text in some places, even when it’s word salad. “There are, of course, many ways of getting to Venice from other points.”

Now you will arrive at the Rio del Piombo. You see people talking, laughing, and reading. And as I sit here, I see the same old sense of fun in all these people's faces, and I know that everybody wants to be in Venice, but I don't know how. If I were to tell you that I knew, you'd laugh too. For one thing, I know that you don't want to believe me, but I'm not lying. I know that you don't want to believe me, but I don't know why. You want to believe me, but I know that you don't want to believe me.

(For comparison, Calvino’s text here — “All this so that Marco Polo could explain or imagine explaining or be imagined explaining or succeed finally in explaining to himself that what he sought was always something lying ahead.”)

Last week I was particularly struck by a gen AI project from DeepMind and Stanford, “Streetscapes,” which uses a trajectory on a map to generate a city scene, as video and 3D render:

… And it can be styled as another place. I could make any map city look like Venice. Or a poor facsimile. “These are all imaginary streets,” their video says. They are all empty, as well.

Calvino seasoning: “The world is covered by a sole Trude which does not begin and does not end. Only the name of the airport changes.”

Now, I do think that Streetscapes work is Awesome, but: How do we resist that sameness, especially after reduction in a limited data set and model? I am a fan of the latent wander, and the weirder the better, but the real world one is good too. From a lovely book (The Infinite Playground), I found mythogeography, with its embellished principles of wandering sideways in a landscape and resisting the normative story. It’s meant to be a form of resistance against the standardized “official” framing of cities and heritage sites, an alternate way of travel. The book by Phil Smith is a gem of arcane weird humor and meta-ness. The manifesto for a drift (or dérive in psychogeography) says:

Start with some kind of theme – look for traces of rebellion or snuffed-out difference, for wormholes, for powerful symbols, for voids, for where things are interwoven. If the drift diverts you onto another theme, that’s fine. The drift may begin to tell a story and you can look out for things that will develop the narrative.

And “To drift well, it may be quite enough to know the situationist definition of psychogeography and simply take it from there. On the other hand, you may end up in a pub, collapsing back into the everyday and missing the whole point.” Keep in mind there are places you might not think of as places, “Z worlds”:

Z Worlds: these are apparently self-contained worlds confined within a small area. They can be found in discarded boxes, in sheds, in abandoned rooms, in rock pools, in shop window displays, in decrepit horse troughs, on lichen-covered walls, at the end of cul-de-sacs, in front gardens, in the corners of clubs. Anywhere a small space has become a self-contained universe. They may be prefigurative, they may be warnings.

In conclusion: “Many are the cities like Phyllis, which elude the gaze of all, except the man who catches them by surprise.” — Calvino the mythogeographer.

Before moving on to the bonanza of news, did you miss the great opening intro article two weeks ago on simulations and consciousness?

TITAA #56.5: Escaping the Sim in a Tank

AI Art Tools

Opening with a few on architecture or 3D scenes, one of my fav topics…

"HoloDreamer: Holistic 3D Panoramic World Generation from Text Descriptions" - love it, it generates panoramas, then uses a depth model to make 3d Gaussian splats for the scenes.

"Streetscapes" — amazing work using map data (and streetview) to generate city scenes, and then render as 3D from the video. More above in the intro.

"Splatfacto-W: A Nerfstudio Implementation of Gaussian Splatting for Unconstrained Photo Collections." This allows you to restyle a monument visible in photo collections (like the Eiffel Tower) in NerfStudio in real time, if you are able to run the whole shebang (looks like a remote machine is the best way to go?). Frame from one of their videos of photo picking:

Creating 3D House Wireframes — this paper does text 2 layouts… (h/t Dreamingtulpa.)

But I was powerfully reminded of this recent xkcd on house inputs and outputs:

And in Invisible Cities, Armilla “has nothing that makes it seem a city, except the water pipes that rise vertically where the houses should be and spread out horizontally where the floors should be.”

Today the incredibly efficient InstantSplat, promising 3d splats in seconds, was code-released.

Creativity/Narrative

"Magical Interactive Storybook That Reacts to Your Touch," a story with mostly links to X video demos. Made with Rive. I should probably check this app out (I saved a link in Feb).

X link by Lance Snider

FairyLandAI: Personalized Fairy Tales utilizing ChatGPT and DALLE-3. Paper describing the Android app (site) that generates “personalized” fairytales for kids with images. Yet another children’s story generator app… It integrates educational and “moral” guidance. There are example prompts, and the main issue they report is a failure of Dalle-3 to remain consistent in character generation (not a surprise).

The GPT-WritingPrompts Dataset: A Comparative Analysis of Character Portrayal in Short Stories. — this is a project comparing GPT3.5 generated stories with human stories (very short ones), to check biases and differences using NLP tools and methods. Fans of David Bamman et al’s work would like this. “Our analysis shows that LLM-generated stories tend to be more positive, less active (lower arousal), and more in-control (higher dominance) when compared to human-written stories…” but they show consistency with human writers in gender biases with respect to female characters (more focus on appearance and less powerful).

Free Audiobooks from Listenly Public Library.

Video Gen Tools

Runway Gen3 Alpha now has image to video! I find it hard to make it work with specific directions on non-photoreal content. Like, I couldn’t get my horned goddess ghost (from MJ) to turn around and walk out the door, it prefers to leave by the wall, which — ok, maybe this is right for it! (When this kind of weird reality is impossible, we will have really lost something. But it also makes these hard to “control.”)

It also doesn’t interpret the figure’s front and back correctly, it becomes a Janus figure with two valid fronts. I actually sort of love that. (Is this Durelas, the god of dreams and doors, in The Bright Sword by Lev Grossman?)

Kling is open for non-Chinese accounts now! With image to video, and end-key-frames too. It is definitely better than Gen3 at several images I tried to animate. The ghost goddess has more relevant motion, but still has the back-front issue:

Kling also succeeded in animating these watercolor cats, although the scene wobbles at the end (Runway just panned across the still image):

banodoco/Dough: Dough is a open source tool for steering AI animations with precision. Out now! It’s installable with Pinokio locally — which I did, but it will require much more work to feed and tend than I have time before this issue is due (today). It looks like a fun August holiday app to play with, though.

Luma AI Dream Machine has added end-key-frame and looping generation options. I also had less luck with it on my antlered figure.

For some more AI videos to watch, in Forbes: "Civit AI Film Contest, A New AI Feature, Dishwashing Cats, More AI Videos". Also see Cine1’s AI movie showcase, h/t Aurora Straton at Google.

Misc AI Links

Image (and video!) object segmentation update to SAM: "facebookresearch/segment-anything-2: The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.

Awesome Artistic Typography — a repo and paper reviewing AI-generated artistic font and text styles including animated ones. Certified Awesome by moi. There are a bunch of projects I hadn’t seen (I usually put them in this newsletter when I do). Including recent KineTy: Kinetic Typography Diffusion Model.

"The Fabrication of Reality and Fantasy: Scene Generation with LLM-Assisted Prompt Interpretation" h/t DreamingTulpa. Relevant to my interests, they introduce a benchmark for image generation output — “the Realistic-Fantasy Benchmark (RFBench), which blends scenarios from both realistic and fantastical realms.”

Misc Web, Procgen, Tools

Lu Wilson (SPC Demo Night) - YouTube talk: TL;Draw demos of MakeReal, the thingy that uses LLMs to make UIs from your sketches, which I’ve mentioned several times in anticipation of a whole new world order for web dev.

AnswerDotAI/fasthtml: The fastest way to create an HTML app - Jeremy Howard’s release. This is another python-based web page constructor and server thing. I’ve tried a few of these and always need more UI customization, so ymmv, and I haven’t played with it yet.

Ray Tracing in One Weekend Book Series.

SVG.wtf — Amelia Wattenberger’s GUI for SVG shape editing. So very cool.

A node-based Webgl Shader Editor and tutorial.

Py5 - I didn’t realize the python version of p5.js was renovated for python 3.9! h/t GorillaSun.

Game News

⭐️ "'Elden Ring' Makes Everyone A Poet And A Reader | Defector" — A poetic and thoughtful tour of messages and the meanings they convey in a punishingly hard game. “Understanding Elden Ring is an act, most of all, of reading.” Players can use templates and limited words to leave notes in the game world, as an asynchronous multiplayer activity. They only get 10 messages. “It's the element of creativity within a set of rules that's alluring.” You may also read the bloodstains left by player deaths. In Calvino’s typology, this game’s world goes in “Cities & Death,” I suppose

“Any message anywhere in the world of Elden Ring could be anything, and you can't know until you read it. They are charged, loose filaments of possibility; any one of them could be someone trying to craft a signpost, or two haunting lines of verse.”

(H/t Florence Smith Nicholls who has written about this academically, too, who I’ve cited before on this.)

Walking sims — after that Defector piece, I found this good 2019 Vice history of games often called “walking sims.” I’ve always loved them for their archaeological “landscape narrative” structure, and didn’t realize they had been misogynistically gender coded during GamerGate. I was not paying full attention to the discourse in a life saving way.

At the beginning of a walking sim, the player is thrown into an unknown space: an empty house, a dry forest, a space station. As the game progresses, the player works their way through the labyrinth of that space’s secrets. They come to understand the people who inhabit that space, and what those people want, and why. They understand why that space is ugly or beautiful, peaceful or foreboding.

Game Tourism — (via the above article) turning off the baddies. Strongly reminds me of Pippin Barr’s Gamething podcast noclip episodes that I’ve rec’d before.

"Fallout: London fan-made mod out today | Eurogamer.net" - I love fan mods. It sounds a bit complex to install. I’m still making my way through Fallout 3 as a tv-show-come-lately, then have to play Fallout 4 before this one.

CRAWL —“crawl is a series of 512 unique, interactive artworks generated by @travess's python-code system, where each artwork is a "level" to be explored through custom software, and multiple levels can be merged into a new, combined artwork, super creative work.” On a crypto gallery site, actually interactive (though I might like the woodblock versions better?).

"A Text-to-Game Engine for UGC-Based Role-Playing Games" — that’s “user generated content.” An ambitious program outline. One detail of note:

Josh Gram’s useful writeup notes on Stacey Mason’s “Responsiveness in Narrative Systems.” “Games and narrative are not at odds. Or at least, they do not need to be. [...] I believe that agency within a narrative is the true magic of our medium.”

Grow Kingdom latest, a video multi-ending CYOA game on Youtube. (h/t v.) Wow, I think you should play this.

Resources:

• Game design tutorials/talk videos from TRUSU (Game Dev Summer School), h/t Matt Guzdial.

• Game Design Humble Bundle from MIT Press. 26 books, ends in 8 days (I grabbed it).

• "Artificial Life Conference Proceedings | MIT Press" 2024.

NLP & Data Science & Vis

Search: MindSearch — Search Engine + LLM Agents = Answer Engine — A search engine research project, open sourced. Not open source, but a cool search engine using embeddings: Exa Search, which turned up some things I didn’t get from more generic engines on a hard query.

Search: alexfazio/OpenPlexity-Pages: OpenPlexity Pages is a content creation engine powered by AI. It serves as an open-source alternative to Perplexity Pages. — another STORM-like?

AI Systems: "Building A Generative AI Platform" — a new epic post from the impeccable Chip Huyen.

Training: Unsloth peeps continue to kill it with performative code for fine-tuning tunable models: "Llama-3.1 8b + Unsloth 2x faster finetuning.ipynb - Colab" — also post on Hugging Face with details.

NLP Prompts: A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Tasks.

Knowledge Graphs: "Triplex — SOTA LLM for Knowledge Graph Construction - SciPhi AI"

Jupyter Tables: manzt/quak: a scalable data profiler for quickly scanning large tables in Jupyter - still in dev, but yet another cool tabular data tool for Jupyter (my quest is never ending).

Pandas Queries: "stanford-futuredata/lotus" — “LOTUS provides a declarative programming model and an optimized query engine for serving powerful reasoning-based query pipelines over structured and unstructured data! We provide a simple and intuitive Pandas-like API.” This is super strange but intriguing.

Vis & AI Design: "We had an AI attempt to make a data-driven story like we do at The Pudding" — this is with Claude Sonnet 3.5. Super interesting experiment well documented.

Recs

I’m moving the recs to the bottom with the poem, so that subscribers who get the separate more detailed recs post can just stop here :)

In my reading this month, there were 2 or 3 books that synced with the opening intro themes of invisible cities and drift wandering. The Book of Hidden Things is particularly concerned with cracks and rifts and boundaries; and the quests in The Bright Sword are a bit mythogeographical. There are buried gods and saints all over the place in these books. And Antimemetics is full of invisible cities.

Books

⭐️ There Is No Antimimetics Division, by qntm (sf). “Antimemes” are weird entities, ideas, or sometimes aliens which cannot be remembered and/or perceived. A govt org fights them. Top notch. Play Control if you liked this.

🏙 Invisible Cities, Italo Calvino (lit fantasy?). I said a lot above. I find it very relevant in this age of AI and as a tourist.

Echo of Worlds, by MR Carey (sf). Book 2 of the Pandominion. A solid ending, if more scattered than the first book. Humanity vs. machines and the Mother Mass.

⭐️ Ghostdrift, Suzanne Palmer (sf). Book 4 of the Finder novels. I loved these books, solid space opera. A pirate kidnaps Fergus, weird alien space hijinks follow.

🐍 Exordia, by Seth Dickinson (sf). Far longer and talkier than I expected (not in a good way)— but also deeper. And sometimes funny. Aliens with “sin” are evil.

⭐️ Book of Hidden Things, by Francesco Dimitri (fantasy/horror). Fabulous if disturbing. A group of Italian men reunite in their hometown in Puglia once a year, but this time their local friend doesn’t show up. Reminded me of the game Mediterranea Inferno.

⭐️ The Bright Sword, by Lev Grossman (fantasy). Can the land be re-united after Arthur’s death? Scattered remains of the Round Table plus a new knight go on quests. I really enjoyed it. Relevant Invisible Cities: “but the gods who live beneath names and above places have gone off without a word and outsiders have settled in their place.”

Games

• Oxenfree II — a solid if very similar sequel to the original. Good for fans of pretty art, talky characters, ghosts and strange radio signals in misty seaside landscape.

• The Stanley Parable— obviously a classic of capitalist work stress and meta narrative, played well on the Steam Deck! I guess it’s a walking sim?

• VR: Vampire: The Masquerade — Justice is fantastic if you like Venice, and you know I do. Similar to Assassin’s Creed Nexus in that you perch on and leap from buildings to assassinate, but with vampires in Venice at night.

A Poem

An open door says, “Come in.”
A shut door says, “Who are you?”
Shadows and ghosts go through shut doors.
If   a door is shut and you want it shut,
     why open it?
If   a door is open and you want it open,
     why shut it?
Doors forget but only doors know what it is
     doors forget.

— Carl Sandburg

Hope the heat is not kicking your butt, it’s kicking mine. I am going to be a bit sparse in mid-month edition, due to travel, but I promise some solid weird AI content news for the supporters. Keep drifting, all,

Lynn (@arnicas on the sfka xitter, mastodon, bluesky and Threads)