TITAA #39: Moon Dots and SpamBots
The Bonne Saison - Stereo Generation - VR - Prompt Libs - SF Recs - Bees
I love citizen science discoveries. My favorite in the past month was the tentative identification of the dots and lines in lots of Paleolithic cave paintings of animals. Ben Bacon, a furniture conservator in London, had an idea, found archaeology faculty who did not think he was crazy, and co-wrote a paper with them in the Cambridge Archaeological Journal (link). They present a data-heavy hypothesis that these dots and lines are lunar calendar markings denoting animal breeding and migration patterns.
The Verge article notes: “The lunar calendar they envision would not track time across years, but would be informally rebooted each year during a time in late winter or early spring known as the ‘bonne saison.’” The bonne saison, or good season, is spring: it’s “a French zooarchaeological term for the time at the end of winter when rivers unfreeze, the snow melts, and the landscape begins to green,” says the CAJ article.
There is no consensus yet on whether this constitutes written “language.” The most interesting symbol may be the “Y” character that appears in many line or dot sequences, thought to be marking contextual events like births. The Y might tip this from counting into something more, at least proto-writing. In any case, “It implies that a form of writing existed tens of thousands of years before the earliest Sumerian writing system,” says the article. Quite a discovery, if so!
TOC for the linkfest below (these will be in-page links in the web page view):
Games & Narrative Links (articles & papers, VR stuff)
NLP & Data Science (lots of prompt tools and some Jupyter)
A Poem (and an amazing thread)
As usual, please consider becoming a paid subscriber if you aren’t: it’s the cost of a coffee per month. I take time off work to write this up at the end of each month. Right now I’m sitting in a hotel room instead of drinking cocktails in Paris, because I’m dedicated (and lame).
AI Art News
Smoother, softer latent videos with “Latent Blending” from Johannes Stelzer (there’s a colab and code). I really like these. I don’t know if my gif of my mp4 really shows it off, but there are fewer jerky jumps than the usual latent vector animations!
GLIGen’s demo is freaking adorable. And fun. You supply a prompt, locations of the items in the image, and it generates a cute picture. The in-painting isn’t bad either! Obviously I wouldn’t like it so much if the output weren’t cute, so ymmv. My prompt here was “a boy and a girl dancing under a streetlamp.”
InstructPix2Pix is out in code and demo on HuggingFace. With this model, you can supposedly edit a generated image in words, like “Make it snow.” I admit, I struggle to get what I’m after but other people are scoring with it.
⭐️ My favorite 3D AI art hack this month: CuddlySalmon/@nptacek has a thread on generating 3d stereoscopic panorama images in Stable Diffusion. It works, although the seams don’t always match perfectly (this could be fixed by hand afterward). My distillation of their tips:
Set image size to 1024x512 and use the v2.1-768 model (available in the Dreamstudio app or downloadable model)
Use a prompt like “3d panorama | stereoscopic Stereo Equirectangular” + content words.
View in an online pano viewer like RenderStuff’s. (Works fine without the upscale.)
For Quest/VR: Upscale your results times 4 at least. There are free sites for this now (or look for ESRGAN, here’s a colab tut).
Copy onto a Quest 2 via PC. Viewing on Quest via an app is easiest, like immerGallery (which has a demo version for free on the app lab site) or Cuddly says Pigasus VR Media Player works too (make sure you have the latest Quest OS installed for it).
The DeepFloyd image gen model can spell (or render text). Which means logos, signs, posters, etc. This will be open-sourced, thanks to Stability AI.
😧 I confidently said at cocktails last night that these models can’t reproduce any of the input they were trained on, and then this paper by Carlini et al. said I was wrong. This is bad news. I guess we’ll see some fallout.
Some Good Art 🎨
Stelfie the Time Traveller is a dude taking historical selfies used a Dreambooth-tuned selfie model and Stable Diffusion. Article, Twitter, Reddit. What a great idea!
Jonathan Hoefler’s (IG) Midjourney-generated fantasy machines and their fictional histories are outstanding.
mrk.ism’s animated AI art of biological critters continues to be DO NOT MISS (I said this in a past newsletter where I linked one of his making-of posts). Also do not miss this one.
Misc Useful Threads
Some useful Twitter tip threads, yeah, I’m sorry it’s Twitter but the art tech people have not all moved nicely or are hard to find on masto:
Twitter thread by @Mort_Vgart on how they animate their images in PS.
Twitter thread by @angrypenguinPNG on creating 3d assets using AI tools (they suggest generating a voxel shape with text2image, then using that with a white background in an image-to-3D model like OpenAI’s Point-E (HF demo).
Also how to train a Dreambooth fine-tune model on HF for cheap, by @angrypenguinPNG.
Google’s unreleased but cool MusicLM generator. (And geez this timeline of work on audio gen papers just in 2023 alone! thanks akhaliq.) I was going to be blasé like “sure, ok, we knew this was coming and certainly it’s useful” but I was actually very into the painting-to-music generations. I think that’s fun. (See also last year’s demo using Mubert to do image to music, by fffiloni.)
💀 Nick Cave’s takedown of ChatGPT lyrics is brutally on point. Most of the creative writing it produces is pretty banal and generic. “Writing a good song is not mimicry, or replication, or pastiche, it is the opposite. It is an act of self-murder that destroys all one has strived to produce in the past.”
Other Art and ProcGen News
The Migraine Art Competition Collection. I love very personal art.
A piece on the popularity of immersive animated art exhibits from The Guardian (popular in UK and Europe, and I’ve now seen 2 posts about them in Boston and NY).
SpamBots typing an AI-tuned version of Brave New World together, by Neil Mendoza. Lol, so cute. Each bot has 4 keys and together they can type out text.
Infinite City generation by Lin et al. “Starting from the bottom to the top, we synthesize multi-modality infinite-pixel satellite images, perform octree-based voxel completion to create a watertight voxel world, then finally texturize with voxel neural rendering.”
Creating Paint-by-Numbers images in R with paintr, by cj-holmes. It traces and polygonizes images.
“Summer Afternoon”, a beautiful, small webgl 3d game by Vincente Lucendo, much loved by many posters.
Shader Light Effects by Maxime Heckel. I follow like 2% of this, but maybe it’s useful to you!
TomLikesRobots had a good twitter thread on people using EBSynth, the free tool for image animation that many love. Plus his mysterious tips.
The animation of talking-avatar tool everyone is using is d-id. The speech synthesis (few-shot, any voice) project is VALL-E from MS with an unofficial PyTorch re-build.
Learn computer graphics with ScratchAPixel 3, open-source course, including 3D. And an ebook and online course for making 2D javascript games with Kaboom.js.
Games & Narrative Links
How the video game Fallout became a backdrop for live Shakespeare shows, an article in the Guardian. This is lovely, it’s basically Station 11 in a multiplayer world. The metaverse is actually here, and it’s in big online games.
Jussi Kempainnen’s posts about making a 2.5D game with AI art help are really fun. I say again, there are lots of tools in these pipelines, lots of true technical art skills at work, too. There’s no just “make me a game.” Echoed in this Wired piece: “Generative AI Won’t Revolutionize Game Development Just Yet.” Meanwhile, Scenario.gg got funding for making game assets with AI tools. And HuggingFace is continuing their cool blog posts on Creating a Farming Game with AI in 5 Days.
InWorld AI character and modding grants offer a bunch of money.
“Cozy city builders are finally letting me design pretty places in peace,” a good article in PC Gamer.
“Where does Dwarf Fortress stand after a decade defined by its own successors?” An article in RPS considering other deep simulators and UI accessibility. I finally tried Rimworld, and it is, yes, a slightly more accessible version of DF gameplay with scifi elements, while not being as deep.
Ink intergration for Godot game engine. Btw, evidently Case of the Golden Idol, a game I rec’d last month, was made in Godot engine.
🌃 I finally watched “I Built Starry Night in Minecraft” (the ChrisDaCow video) and it is funny and fast, if you want to see a look at the details of the painting and his tribulations making it in a voxel (3D pixel) tool.
Text game remakes: Zarf’s review of Colossal Cave remade with images by Roberta and Ken Williams (evidently also a VR version) and a video of Zork illustrated by Imagen but not an accessible app because it’s still Google non-OSS. (I’m especially irked because I wanted to do this myself.) These kinds of translations remind me of issues turning games into movies/tv. Btw, I am liking The Last of Us on HBO despite almost all my friends saying “we’re done with zombies.”
⭐️ Marissa Crane on the Finer Points of Experimental Fiction, on Lit Hub.
Narrative Papers
Can Very Large Pretrained Language Models Learn Storytelling With A Few Examples? by Xie et al. They find that prompted GPT3 quality is higher than other SOTA models from 2021ish but find some light plagiarism going on.
More discourse-controlled generation experiments: Sequentially Controlled Text Generation by Spangher et al. “More structural awareness results in higher control-accuracy, grammaticality, coherency and topicality, approaching human-level writing performance.” It looks tractable to do what they did.
Visual Story Generation Based on Emotions and Keywords by Chen et al has code. It looks like the work may be a year or two old (they’re using Disco Diffusion) but it has some keyword-based control stuff that looks interesting.
TropeTwist: Trope-based Narrative Structure Generation by Alvarez and Font (from last year but I missed it). Evolving narratives with trope graphs using MAP-Elites, a search space method from a paper by Mouret and Clune. I need to digest all this.
VR Related Game Articles
The Future of Text 3 is a massive multi-disciplinary volume with a focus on text in AR/VR. This looks amazing, I need time to read it.
An epic long post by Matt Ball on the history and state of VR/AR/XR investment and development, largely focused on hardware issues.
⭐️ Twine-Space, a game tool project by PS Berge, Daniel Cox, Jack Murray & Anastasia Salter. This was at ICIDS 2022 and I forgot to include it last month. It’s a Twine extension to allow VR/AR experiences via A-Frame and AR.js with Twine interactive story authoring. I’m very intrigued. You can include 3D panoramas and 3D objects along with the story options.
NLP and Data Science
It’s a ton of LLM links this time, I’m afraid! Plus some Jupyter developments.
LangChain is a tool to make working with LLMs (large language models) in multi-prompt contexts easier, and here is entity memory in LangChain - a way to record entities and use them in future prompts.
See also Promptify, which is a LLM prompt wrapper for NLP tasks like entity recognition and classification. I haven’t tested it out yet.
How to implement Q&A against your documentation with GPT3, embeddings and Datasette - knowledge/retrieval augmentation example for GPT3, done practically, by energizer bunny Simon Willison. Also a use case in the OpenAI Cookbook (and many other places right now)…
OpenAI Cookbook, online recipes for tons of stuff using their APIs.
Prompt Engineering guide from DAIR-AI.
“Text Pattern Extraction, Comparing Human-in-the-Loop with GPT3” by Maeda Hanafi. The HITL tool is from IBM, Pattern Induction. Interesting article. Relevant for rule-based ML, see also the CRAML Beta which looks like an interesting weak-supervision method. “The goal of CRAML is to take txt, csv, xml, and other text inputs, and output a unified and labeled dataset characterizing the contents of the text.”
Using Entity Rules to Guarantee Classification with SpaCy - I wrote up a section of my NormConf NLP tips talk on how to ensure that a classification is positive for “NSFW” using porn star names in entity rules while classifying image generation prompts. (It took me a little while to figure this out, so sharing.)
Jupyter developments:
Yet more Jupyter widgets: anywidget.
Streamlit-Jupyter: “preview and develop streamlit apps in jupyter notebooks.”
JupySQL, a better (big data) SQL for Jupyter.
Books!
So many good ones this month. I could put stars on most of them.
Under Fortunate Stars, by Ren Hutchings (sf). Very unusual time-travel book, on many best-of lists. Two ships meet in a weird space zone, one a ship evidently from the past of the other, hailing from the era of an interspecies war. The past ship’s crew are supposed to somehow become galaxy-saving heroes, but they don’t seem to quite match the historical stories.
The Wild Hunt, by Emma Seckel (fantasy). Truly atmospheric, a bit gothic. On a remote Scottish island after WW2, two sad people meet up while the angry crow spirits are gathering. How can they be appeased, and what happened to the boy who disappeared one night? The despair of the woman whose beloved father died, who has no income or job prospects, really cut at me.
The Immortality Thief, by Taran Hunt (sf). Space opera fun. Frenemies try to find a hidden cache of data on a space ship infested by genetic monsters. I gobbled this up, where’s the next one!?
Light from Other Stars by Erika Swyler (sf). Ooof. This is very good but quite heavy and features some body horror (CW). Lovely writing, but I cried a lot. One of those past and present interwoven stories, with an astronaut on a ship with a failing engine remembering her father’s botched experiment that created a time anomaly in their small Florida town. A lot about family here.
Singer Distance, by Ethan Chatagnier (sf). More fantastic writing. Alternate timeline: Mars has some kind of advanced life who communicate with us via math puzzles etched into their surface, which get increasingly difficult as humans solve them. A genius math grad student thinks she has the answer to the next one, and she and her friends take a road-trip to write it out on the desert in secret. After that, things get (very) emotionally difficult.
The Honeys, by Ryan La Sala (fantasy-horror). A non-binary, gender-fluid kid’s twin sister comes home from camp, tries to kill him, and dies tragically. Mars enrolls in the camp to look into what happened, despite the sexism and homophobia they experienced there as a child. They’re ambivalently welcomed by the girls who keep bees and live apart from the rest of the camp. This is gorgeously written and creepy and enraging.
TV Recs
Three Pines (mystery, Amazon Prime). Solid treatment of Louise Penny mystery set in Québec, with surprisingly creepy cute town of people with secrets and book clubs. Plus a lot of indigenous peoples’ pain which makes it not at all cozy.
Avenue 5 (sf comedy, HBO). I finally gave this a shot, and quite liked it! It’s a dark comedy version of the bleak Swedish sf movie I have recommended before, Aniara. A luxury private space hotel run by a puppet crew of actors is damaged and unable to return home. Mayhem ensues, some of it funny, some of it quite a bit darker.
Interview with a Vampire series (horror, AMC). This time it’s explicitly queer horror, with a lot of abusive relationship (CW, also for gore). But I really enjoyed this.
Emily in Paris (Netflix) and Sex Lives of College Girls (HBO). Trying to avoid the darkness, I binged these (I am not proud). S3 of Emily was quite trying, actually, and S2 of SL has a few issues too especially re the poor girl who makes bad choices and is stuck with the college bill.
Game Recs
I’m still playing Wanderer in VR, it’s huge. Again recommending if you like adventure puzzle games!
⭐ The Spirit and the Mouse is adorable. This is a cozy 3D game with a little mouse who has to solve various small puzzles around a cute French town to help the inhabitants get happier. It’s not too hard, it’s pretty, and sometimes funny. (Fine for kids, but really needs a controller to play.)
Poem
⭐ This thread, in response to chenchenwrites asking, “what’s a poem that’s saved your life / healed your heart? i need one of those right now”. I cried, but also for the dying of Twitter, as I read through it.
Snow is Falling
The funerals keep coming more and more of them like the traffic signs as we approach a city. Thousands of people gazing In the land of long shadows. A bridge builds itself slowly straight out in space.
— Tomas Tranströmer, translated by Robin Fulton, via poetryforge on IG
All the best for your next month as we head towards spring, and now I am going out of this hotel room,
Lynn / @arnicas on mastodon and sometimes Twitter
OK, the historical selfie thing is clever and I like it a lot. And the spambots are adorable 😭
I don't know how you make the time and brain space to find all this stuff, let alone read/watch and comment on it. I remain in awe :)