TITAA #33: Hunger Stones and Stability
Drought - Stable Diffusion - Big Data Vis - Swinging Overhead - Hard Work in Space
Hello, new subscribers, and welcome :) This is a longer one than usual because every week of August was like a month in AI art. It’s too long for gmail, so read on the web. Feel free to skip ahead to get to the good book/tv/game recs!
A few weeks ago there was a call for poetry related to climate change by American poets, and all I could think about was dead bodies showing up in dry Lake Mead. Then there was a foot found in Yellowstone hot spring. American lake finds are records of accident, murder, and mob hits. Not super poetic for me, and anyway I am dried up, too, when it comes to creative writing.
But in Europe, the drought could inspire epic poems. Rivers and reservoirs have sunk to their beds. The central European “hunger stones,” stones marked with dates and messages from previous droughts, have surfaced again, as they did in 2018. “If you see me, weep,” says one of the most famous in the Elbe. In Germany, another says, “When this goes under, life will become more colorful again.” Yet another turns the drought into women’s work: “Girl, don't weep and moan, if it's dry, water the field.”
In a study of these stones in the “Czech lands”, they record messages like…
1369: “Better wine in Bohemia than in Austria due to severe drought” [oh noes]
1371: “Frequent fires”
1393: “Possible to cross the River Vltava on its bed, water “as green as grass””
1473: “Streams dried up. Forest fires. Poor harvest of cereals and other crops.”
Meanwhile, in Spain, flooded village Aceredo is now visitable, due to the low water levels in a reservoir. Other submerged ruins that have surfaced include the village of Portomarín, the Roman fort Aquis Querquennis, 11th-century Sant Romà de Sau church in Catalonia, and Berich (or “Edersee-Atlantis”) in Germany.
The fantastic Dolmen of Guadalperal, a megalithic monument known as the “Spanish Stonehenge” (2000-3000 BC), has surfaced in the Valdecanas reservoir in the province of Caceres (wikipedia).
Here’s a good Guardian overview of European sites surfacing, and here’s a NY Times one. There are also bodies and plane wrecks coming out of melting glaciers, the mountain-top version of Lake Mead, I suppose. Finally, an excellent map of the dry rivers in Europe, the worst drought in 500 years:
But in all seriousness, the impact of empty, hot rivers means we can’t ship things on them, hydropower is limited, and we can’t cool nuclear power plants in France that produce clean energy. It’s a cascading crisis, even apart from the obvious dry fields and dead crops. I want to drop in an offering to the river gods.
Meanwhile, in tech…
AI Art
Image Generation
The big news of the month is Stable Diffusion’s public release. For those new here, this is a text-to-image model, such that I can type “a wall covered in roses and thorns in front of a mysterious tower, fantasy, moody, deviantart” and get image(s):
It was largely paid for (and announced) by hedge-fund dude Emad Mostaque of Stability.ai, who has also been funding LAOIN and EleutherAI on other open source efforts, like gigantic image datasets and large language models. Emad had some early relationship with Midjourney, but parted ways, it seems. Stability.ai has recently hired some of my favorite creative AI developers, including Katherine Crowson/@RiversHaveWings and @Kaliyuga_ai.
There are many ways to experience or build on Stable, only growing:
a beta website, Dream Studio, on which you will pay for images after trial
An easy SDK lib for using the API
A website to browse prompts and their output, Lexica
In-painting and image2image code (colab for image2image by patil-suraj)
a release of a web ui for it in a docker container, so you can run it yourself locally by mallorbc; another webui here by hlky; a how-to for running on an M1 Mac (my case) by James Reynolds
The release of the weights and code means new applications and workflow demos are also dropping, like a new Photoshop plugin called Alpaca that wants beta testers. AI dev artist Gene Kogan tweeted a very popular collage in-painting concept a few weeks ago. Here’s a concept for a Figma plugin by Antonio Cao. The apps keep coming out. Stable is also being plugged into various other creative tools like AI Dungeon, Night Cafe, NovelAI, and ArtBreeder.
HOWEVER 😱. Along with the release, NSFW (porn etc) content became an immediate problem. The first Midjourney-tuned version of the model was taken down after 24 hours due to producing so many realistic porny images, and several Reddits were shut down over porn content. The model’s NSFW filter is easy to disconnect. You can read a few thoughts on this problem from Waxy/Andy Baio (skip down past the DALLE2 stuff) in “The Pandora’s Box of AI Art.” Here’s a long article by Bakz T Future about his arguments with Emad about whether it was ready to release as-is.
My personal position is that I want an open-source model and code, but not one trained on porn. Stable didn’t need to be trained on porn. Presumably the model includes adult content because of profit possibilities there. It’s expensive to train things this big, and this model will be around for a while having an impact on the world. I’m a volunteer moderator at Midjourney and I don’t need this kind of extra work.😕
How Does it Compare?
If you want to see some comparison of Stable Diffusion with Midjourney and DALLE-2 output, AI artist @fabienstelzer has a great series of comps on a long tweet thread here. I did a few publicly too. My take on it is that it’s harder to get a beautiful image from raw Stable Diffusion without a lot of “promptcraft,” as compared to Midjourney which has been tweaked for artistry over photo-realism. (MJ is currently user-testing tuned models that support more photorealism while still excluding porn.) DALLE-2 is better at rules of image composition but worse than everyone at artistic output.
Google Parti is even better than DALLE-2 at composition, and Jason Baldridge at Google has achieved some lovely artistic output too. Parti and Imagen tools have been merged, according to Jason, although it’s not clear this is relevant to anyone since no one can get access or code. (I’m a consultant Google Arts & Culture and can’t get access.)
FYI, here’s Parti+Imagen doing a great job on my complicated prompt “a penguin in a carnival mask beside a canal in Venice”, an image I needed for my AI art story game released here. It looks horrific from almost all other models, although it comes out very atmospheric when rendered imprecisely in Midjourney (shown in last month’s newsletter).
Parti, an autoregressive model using a GAN, is better able to construct an image from a prompt, leading to more coherence. Parti works from large transformer encoder and decoder models to produces image token segments. It doesn’t use a CLIP text-image encoding.
The CLIP-interrogator (that’s a colab notebook) from @pharmapsychotic is a really interesting toy that may depress you as you see what CLIP models make of a subtle image. CLIP (developed at OpenAI) is a model technique that embeds text and images into a searchable space, more or less. You can embed text with CLIP to retrieve images or use an image to see what CLIP thinks describes it. For instance, using the interrogator notebook and this image of an oil painting by Hugh William Williams…
The CLIP vector matches of the image using the CLIP VIT-L/14 model used by Stable Diffusion and the ResNet 50 model report these top matches:
And it comes out with a suggested prompt construction from these top scoring terms: “a painting of a castle on a hill, a detailed matte painting by Joseph Wright of Derby, featured on deviantart, modern european ink painting, matte drawing, matte painting, flemish baroque.” If I feed this into Stable Diffusion, I get these not shabby results, frankly — although note how color scheme is in no way represented:
For more examples of this operation using CLIP interpretations of images to make new images (a kind of “telephone game”), this is a fun post from Shardcore.
I don’t know what this is (I’m not going back to the “internal language debates”), but @dribnet can find words like “splafulated” that return stable results across models, presumably something to do with CLIP tokenization?
Artists, Copyright, and Search Spaces
Some people are unhappy about these models being trained on web-posted images by artists. From the latest twitter discussions, I suggest these popular reads: A short succinct legal perspective from Technollama, a longer ranty-style one by Daniel Jeffries covering some of the same ground with more examples, and pointing out that artists learn by copying other artists. Note— that castle ruins picture above had a link on it to “order as a handmade oil painting” of the same picture.
🤔 One of the more interesting points was made in a Twitter thread by an actual artist (Marc Simonetti), saying that they are uncomfie with search results for work by them returning AI art with their name in the prompts: “‘Dark matter angel by Marc Simonetti” — when they had nothing to do with the artwork except as an influence. How will people know the difference? Won’t it potentially tarnish their reputation as an artist if the results are poor? Most people using search engines won’t be able to distinguish between real work by the artist and AI art results. This is a very good poser of a question.
Other AI Art News
Textual Inversion, the code is coming out! This would help maintain stability in terms of character content. I’ve seen one attempt to plug it into code with Stable Diffusion but it didn’t look like it worked very well? Related, Google’s Dreambooth, a paper with no code or demo.
If you want to be following Yet More AI Art news, Guy Parsons (the Guy who did the DALLE-2 prompt how-to book) has a newsletter which is not bad! He finally tried Midjourney too (he has a good article and how-to, with some comparison to Dalle-2), and will be talking more about Stable Diffusion, I am guessing.
Midjourney was also a star in the Last Week Tonight With John Oliver segment on AI art.
🎨 Disco Diffusion with in-painting (colab notebook), and some good video examples in the thread, from @cut_pow.
Links to lists of “awesome AI art image synthesis” tools.
🎴 Lots of people make tarot cards in image generation tools! Wolfe’s Arcana, or Midjourney, Meet Gene Wolfe. (Thanks, @Somnai). This is a very cute Minion Arcana made in Midjourney (thanks, Moritz Stefaner).
🎵 Musika music generation. (There’s a ton of stuff in music gen happening, I admit I don’t have time to pay full attention.)
Neural Human Radiance Fields from a single video, code.
Other Tech Art Links
🌏 A book site tour made in Google Earth by David Hewson: Locations in his novel set in Venice, the Garden of Angels. (True story: I met David at a bar in Venice, we ended up chatting over drinks. He lives my ideal fantasy life: He travels and writes fiction while he travels.)
py5.py - a modern python version of Processing.
A giant tweet thread about the mysterious Voynich manuscript, from Lisa Fagin Davis.
This hieroglyphic journal of a voyage to the Caribbean, like early emoji writing, in Public Domain Review:
Art from Code, in R, a tutorial workshop from Danielle Navarro. She’s killing it, she’s also below.
Vertex Art Gallery - “a virtual art gallery that shows 34 new art pieces every day.” The artworks come from the Art Institute of Chicago’s API.
NLP & Data Vis
ConcepCy, a spaCy wrapper for ConceptNet. ConceptNet has some junk in it and some serious gaps, but this is still cool.
A nice overview article of spaCy pipelines and tips from Ahmed Besbes.
Big data visualization: A problem I always have. Danielle Navarro on How to Visualize a Billion Rows in R with Arrow. Also using Arrow, Ben Schmidt has made some great improvements in his Deepscatter library and can also use Arrow (see an Observable demo here). Check out his Gaia demo here. See also: a HoloViz tutorial for big data, from KDD 2022.
Creating beautiful river maps with Python by Adam Symington, mapping Africa.
Some slides of mine on Artistic NLP for a group of artists organized by Google Arts & Culture, which cover text generation, algorithmic/simple generative formats, poetic coding, multi-modal, data-driven poetry, and more. It’s a smorgasbord!
Game-Related
Social Simulacra, research inventing personas on text community spaces who argue etc. Intriguing. “Can a designer understand how a social system might behave when populated, and make adjustments to the design before the system falls prey to such challenges? We introduce social simulacra, a prototyping technique that generates a breadth of realistic social interactions that may emerge when a social computing system is populated.”
Aaron A Reed’s keynote talk at Narrascope is superb, reflecting on lessons from the 50 Years of Text Games posts and upcoming book. The section on lessons from LambdaMOO made me connect again, which actually caused some anxiety as I hit “go.” Weird. Other talks are also now online, it seems!
The Minecraft generative settlement design competition for the year has been judged, and the results show that few of the winning entries are using AI and most are revs on past year’s efforts. Very interesting nonetheless. Why don’t I have more time to play!
Awesome-R-Games. Games written in the R language.
A curated collection of awesome game dev resources.
Books
The Women Could Fly, Megan Giddings (lit fantasy). In a modern world where women might be witches and witches are still burned, women have to register with the state, swearing to having no powers, and ideally they marry a male protector. There’s a lot of racial bias here, too, because guess who is more often accused of witchcraft. Our (black) heroine’s mother disappeared when she was young, and now Josephine gets a mysterious invitation to an island that might hold answers. It’s not plot-driven so much as reflection driven, which I guess is why it’s “lit fantasy” to me. I especially enjoyed the description of cursed artworks by registered witches; there is one featuring a woman crying alone with salad. Also some lovely magical realism.
Diaspora, Greg Egan (sf). An interstellar radiation disaster causes an AI polis in space to begin looking for how to avoid a finite future; they find aliens who communicate in strange math and doorways to new realities. Mathematical descriptions I couldn’t follow and theory-heavy, but mind-expanding. It’s a good follow-on to Permutation City that I newsletter-rec’d this summer, but more intensely abstract.
The Anomaly, Hervé Le Tellier (lit fantasy). A plane lands for the second time, a few months after it did the first time, with all the occupants duplicated. One of them is a novelist who has written a book called The Anomaly before killing himself (because this is French and meta). Sometimes funny, especially in the collection of scientists and philosophers (vis, French) who argue about what it means, and propose a kind of Greg-Egan-esque simulation theory. There is also a very stupid American president. (CW: a dog dies, suicide.)
⭐ Gallowglass, SJ Morden (sf). I liked his Frank Kittredge books about convict labor on Mars, and this is terrific too. Starts off exciting and does not let up, but is also thoughtful and anti-corporate in the right ways. Rich kid runs away from home to work in space, gets hired to navigate to a giant asteroid for mining rights. I had many feels over the crappy captaining of the scrappy ship where anyone who is dangerous or dead weight gets spaced. It felt like a horrible startup. (“Your schedule is stupid because it does not take into account the work,” was one of the moments here.) If you like “hard” sf, with lots of stuff about piloting in the debris field around a boulder in space, this is for you.
TV
Sandman — Netflix. Enjoyed, although Morpheus’s bedhead and pouty lips irked me. His coat made up for it. I have not read the graphic novels, and was a bit weirded out by the 2 arcs in this; the Dream Vortex thing felt dumb to me even if I liked her. Visuals and guest character bits were excellent.
⭐ Black Bird — Apple TV+. Really thoughtful and grim. Amazing performances and casting. Cocky guy arrested for drugs and guns takes a deal to go into a maximum security prison to try to get a confession from a possible serial killer (or serial confessor) who is mentally very, very “off.” (CW: Rape and murder of girls.)
Keep Breathing — Netflix. I binged this, but it will not be to everyone’s taste. Antisocial, workaholic woman crash lands in Alaska and has to survive alone. Meanwhile, a ton of flashbacks and family stuff about how she got this way, complete with mentally ill mother who abandoned her. If you like wilderness survival and can tolerate the family and personal drama, give it a shot.
Irma Vep — HBO Max. Very odd and meta! Occasionally hilarious. American actress comes to Paris to star in a remake of a classic b&w series about a criminal gang; the obsessed French director very much dislikes people, which as, his psychiatrist notes, is a problem in his job. The leader of the fictional gang is catsuit-wearing, high-heeled Irma Vep who vamps around on Paris rooftops. The spirit of Irma takes over, too, and soon the actress is walking through walls. It’s hard to describe; the ending is a bit inconclusive, but overall this was very fun! (The bondage consultant scene: LOL.)
Games
Citizen Sleeper. Not my usual thing, with a lot of game-play apparatus (dice rolls, etc, more like a TTRPG), but I’m totally enjoying this. You are a rescued robotic worker in a huge space complex, trying to get enough money to survive and escape a hit on you. You do hard, risky labor to pay for medication and food. Lots of choices about who to help and where to focus your efforts, tons of story details on the inhabitants of the massive, ruinous, indigent space city.
Gibbon: Beyond the Trees. I got this to try out the feeling of flying through the trees as a little gibbon monkey, after playing the VR game Resist (see below). It’s really beautiful art, and mesmerizing. Truly.
Stray, the cyberpunk kitty game. I made it half-way thru before a bug I can’t get around hung me. Other people have it, too, so I’m waiting for a patch. I very much enjoy climbing around and exploring and being a cat in a scifi city, but enjoy much less the button-pushy fights with the rat critters. My take on those kinds of sections is that there ought to be an option to skip past them, if you’ve tried 10 times and failed. The next game offers that! See, I’m not crazy and lazy!
Red Dead Redemption 2. So many feelings: I bought this on sale because I heard about the VR mod and saw how beautiful the scenery is, but right after I bought it, the mod was DMCA’d and taken down. I’m pretty mad about that. I did start the game, and it is very cinematic: Credits at the start, lots of cut-scene videos. It does indeed have beautiful scenery, and lots of on-screen UI tips and help for the controls. More importantly, it has the option I mentioned above; if you want to skip a challenge, you can. So smart and welcome.
VR Games
Resist, Oculus Quest. This is a low-fi, Spiderman-y espionage game, which is so much fun. You are an agent, swinging by ropes among the skyscrapers of the city, shooting drones and mechs. There’s a Sean Bean-alike giving you your jobs by smart-watch. Not for the queasy! Truly fun if you like flying. I was tickled that I could land on a boxy car and go for a ride on it.
Space Docker VR. I like outer space games, a lot. This one is just about navigating a shuttle and doing delivery stuff in zero-g, which is amazingly difficult, and thank goodness the devs have been releasing more help modes and controls to make it easier. I’m still going through the huge number of tutorials, one of which meant I was off in featureless skies 25K from where I was supposed to be… and had to trust radar to get back. It took a while.
Red Matter 2 is out in VR!!! I bought it immediately, haven’t played yet. I loved the first one.
Poem
I've known rivers:
I've known rivers ancient as the world and older than the
the flow of human blood in human veins.
My soul has grown deep like the rivers.
I bathed in the Euphrates when dawns were young.
I built my hut near the Congo and it lulled me to sleep.
I looked upon the Nile and raised the pyramids above it.
I heard the singing of the Mississippi when Abe Lincoln
went down to New Orleans, and I've seen its muddy
bosom turn all golden in the sunset.
I've known rivers:
Ancient, dusky rivers.
My soul has grown deep like the rivers.
—Langston Hughes
Well, this was a very long one. If you made it through, thank you for sticking with me! And thanks again to Claudia for the Recomendo post, which more than quadrupled my readers. It takes a lot of time to write this free newsletter, so say hi on Twitter or buy me a coffee ☕.
Best, Lynn (@arnicas)
Wow, fantastic post. The hunger stones resurfacing are an intriguing and horrifying bit of culture. And good work keeping up with all the work in the image generation field this month. It's been crazy!