TITAA #51: Tool People & People Tools

Tools with Personality - SVG - Creativity Measures - Pals - 13 Clones - UEVR - Text Pipelines - Intense TV

Max Siedentopf campaign for Parco (site source)

I’ve decided I have to follow my synchronicity in themes I’ve encountered for these intro pieces. The latest synchs are a series of encounters on the tools-persons spectrum. From the header image: Max Siedentopf, a surreal sculptor/artist, did a campaign for Japanese brand Parco, featuring happy human appliances.

Throughout, a series of well-dressed people are either given attributes to turn them into an appliance – an on-off switch, a fan and lampshade placed upon one’s head, vacuum cleaner tubes for arms – while some of the subjects seem to ‘become’ the appliance. One figure is simply a head popping out of a hi-fi system, while an armchair is both the chair, and the person watching TV. Mind-boggling. In the short promo video, each anthropomorphised appliance is shown going about their respective roles – seemingly content with their dual existence. “The devices in our home become smarter every day,” says Max. “Here the idea was to imagine a near future where our machines take on a life of their own.”

You can see animations of them on his IG account (where I saw them first). I’m sure many of you are thinking of Douglas Adams’ “Genuine People Personalities” from Hitchhiker’s Guide to the Galaxy (and read Ars Technica in 2018 about personalities infecting our gadgets).

Non-human tools with personality are obviously starting to appear more often, especially with AI support. Hrbrmstr’s Daily Drop (a tech newsletter I recommend) recently featured the weather app from Carrot, which has a personality slider: “From the hilarious dialogue to the delightful animations, CARROT has character in spades. You can even customize CARROT’s personality to your liking.”

Meanwhile, in Large Language model (LLM) land, we keep seeing more and more pieces about model personality and role-playing. GPT4 was updated this week to make it less “lazy”. In the paper “Assessing and Understanding Creativity in Large Language Models,” the authors found that AI agents working together produced more creative results, much like teams of people do; and that having an LLM role-play a scientist improved creative output as well. (See more details below.)

I’ve posted a lot in this newsletter (2 weeks ago: 4 frameworks) about the “characters as agents” Sims-style tools being developed with AI backends. Park, of the famous Stanford Agents paper, has a TED talk out now, in which he reflects on how simulations can teach us about humans (not a new concept in sociology or economics). But another common meaning of “agent” is just a semi-autonomous code tool. There are also a ton of papers and frameworks for making that kind of agent system, including wiring up crowds of them to solve tasks in pipelines. We don’t usually think about their personalities in that context, although it’s becoming clear that good prompting requires at least some of concern for this. You want your LLMs to be motivated, after all!

In fiction-land: In the wonderful and funny Thirteen Ways to Kill Lulabelle Rock, Maud Woolf’s protagonist is the 13th clone of actress and model Lulabelle Rock. Lulabelle created her to kill the other clones in order to distract from expected bad press for her upcoming film of Medea. It turns out that all the other clones were tools: created to perform various jobs ranging from “explore my artistic side” to “go to brunch for me today because I’m bushed.” It’s funny and weird, with chapters based on Tarot cards. (One of the clones, a secretary, insists she is unique because “I can do Excel.”).

So now let’s talk a moment about real human behavior, human to human, in which humans are seen as disposable tools: the layoffs continue, across tech and the games industry. Nevertheless, exec salaries are astronomical at most of these places. AI “replacing” people is both a constant fear and occasional explanation in media coverage, but as game scholar Mike Cook notes in an excellent article, we already see the truth that “the dehumanisation of game workers and the brutal treatment of outsourced work, … that many roles in the games industry are already treated as if they were automated.” The threat of AI may be in fact a bit orthogonal to the ongoing threat of crappy labour conditions driven by short term market valuations. This situation has been going on for a long time.

Anyway, some cool articles and tools below!

TOC (links on the site, and separate recs going to financial supporters):

• AI Art & Tool News (3d, Video/SVG, Creativity & Writing, Arty/Procgen)

• Games News

• NLP & DataScience

• Recs: Books, TV, Games (VR)

• A Poem

AI Art & Tool News

RPG-DiffusionMaster makes extremely detail-specific images with complex prompts (there’s code).

Midjourney has tiling and outpainting/inpainting in v6 and is working on a new style tool.

3D and Depth Related

There’s actually been a ton of papers on editing nerfs and splats, replacing objects, etc. I simply don’t have time to go over them when I’m a day late. Drop me a note/comment if you want links to those. Meanwhile, this research tracker of papers on 3D Gaussian Splats on Hugging Face is the best I’ve seen. It’s a huge list and notes if there is code. And here’s a new survey article on advances in 3D generation.

FYI: There’s a Unity SDK Plugin for Blockade Labs for generating 3d HDRI’s in Unity.

“BlockFusion: Expandable 3D Scene Generation using Latent Tri-plane Extrapolation” has a cool video of a game character moving and platforms and walls appearing as she does.

Pix2Gestalt is a pretty cool looking segmentation, recognition, and reconstruction tool for images. They say code and a demo is coming.

I’m also kind of impressed by Moondream, a tiny vision language model that can run on lots of devices.

Depth models: These are cool because they allow you to do fun things like relighting, 3D fx and generation from a 2D image, and more. This past 2 weeks we got a nice article from Patricio Gonzalez-Vivo on depth models and then right after, a new model release, Depth Anything, with code.

Patricio also released his Prisma pipeline for image processing, which includes depth (and the new DA model, which he says is good at video but otherwise not as good). “This pipeline expands images into data that can be use for 3D reconstruction or realtime post-processing operations.” It includes flow and a segmentation model.

Video/Animation/SVG

Lumiere from Google looks amazing! Crisp output, many styles, includes masking, inpainting, art animation, etc. Getting a lot of press despite not being released for use. I’ll wait till I try it to say more!

The Sketch-to-Video SVG project I had in a previous edition now has code out: “Breathing Life to Sketches Using Text to Video Priors.”

Pretend these are animated. Sorry.

Related, this PatternPortrait plotter-based project to make “scribble” pictures with a drawbot is pretty cool. Not animated, but svg-ish.

There’s also a cool looking SVG stroke “tokenization” and generation paper, StrokeNUWA.

AI Creativity & Writing

A couple new studies of AI creativity: “Assessing and Understanding Creativity in Large Language Models,” by Zhao et al., is both deep and broad. Results suggesting that prompts, role-play, and multi-model collaboration improve creativity results related to originality. Weirdly, role playing a “scientist” scored better than role playing an engineer or an artist, which I suspect is a training data bias. In terms of personality attributes,  “levels of emotional intelligence, empathy, conscientiousness, extraversion, and neuroticism in Large Language Models have a significant positive correlation with creativity levels, while agreeableness shows a significant negative correlation.” [Bold mine.] Note that GPT4 is intimately involved in the scoring. Meanwhile, Ethan Mollick also shared a pre-print suggesting getting more diverse ideas from GPT4 is dependent on prompt engineering, especially “Chain of Thought.”

“Weaver: Foundation Models for Creative Writing,” an initial suite of Chinese LLMs trained for creative and professional writing, with English coming. “Notably, our most-capable Weaver Ultra model surpasses GPT-4, a state-of-the-art generalist LLM, on various writing scenarios, demonstrating the advantage of training specialized LLMs for writing purposes. Moreover, Weaver natively supports retrieval-augmented generation (RAG) and function calling (tool usage).” They are a company; their English site is here with a waitlist signup.

Zooniverse - Help Andrew Piper and colleagues with annotating relationships between characters in books. It’s a grand goal, and the part I most care about it the open sourcing of it. “The goal of this project is to generate knowledge about the behaviour of literary characters at large scale and make this data openly available to the public. Characters are the scaffolding of great storytelling. This Zooniverse project will allow us to crowdsource data to train AI models to better understand who characters are and what they do within diverse narrative worlds to answer one very big question: why do human beings tell stories?”

Dashtoons - an AI generation site for creating visual novels or comics using text to image (h/t hardmaru). The tutorials include how to get good hands :)

Edited to add: I meant to include u/Philipp’s Reddit Murder Mystery image post in which he buried clues to the murder (using Dalle-E, MagnificAI and Photoshop). It would require you to go see the one you can zoom in on…

A few articles with surprises in the AI art world: one announced she used AI, the other that they didn’t but claimed they did.

• “Japan literary laureate unashamed about using ChatGPT,” h/t Ted Underwood. Rie Kudan won the award and then admitted at least 5% had been written by GPT4. And that using ChatGPT had been a big help in her creative process.

• George Carlin AI special turns out to have been written by a guy not AI. Speaking of which, I linked to a good retro horse_ebooks article last issue.

Misc Arty & Procgen

This Is Not a Good Sign: an AR poetry project by JR Carpenter and Tomo Kihara. “In this project we use augmented reality to overlay the user's surroundings with signage posing questions about past and present climatic conditions. Why is the sky so blue? Has it always rained?”

Neal Agarwal’s Infinite Craft AI combinations toy is out! It takes a little while to get to cool concepts, but it’s fun! I’m not entirely sure what the node links are about though, that’s not obvious. Starting from basic elements, I got the Buddha, Titanic, Unicorns, and Atlantis.

Pong Wars (h/t luokai on Threads, who is a good tech follow). This is a dynamic animation in which each ball erodes blocks where it impacts.

I recommend the Web Game Dev newsletter. Not sent often, but packed full of tools and library updates.

Games Related News

🧭 The Square Mileage of a Virtual World, by CE Janecek. Looks at map size, play time, AI/procgen world gen, tiny maps with endless play (e.g., Animal Crossing). A good thoughtful piece.

A good article on Lethal Company’s preview sales success, given the solo dev’s history as a child Roblox builder. It sounds amazing, because it sounds collaboratively playful and funny:

“Because the creatures and hazards are so ridiculous, moments of actual fear and isolation in Lethal Company are usually followed right up with laughter from your friends in the afterlife,” Zeekerss tells me. “Gameplay-wise, it's actually a game about laughing at death.”

Watch VODs and livestreams of Lethal Company and you’ll see people laughing at death nonstop. Something about the game invites it—and not only that, but invites real play, as in pretend play.

Lethal Company isn’t just a scary game, or a funny game. It’s an interactive system that generates stories—some scary, some funny, mostly both. And when you get a great story out of a game like this, you have to tell others about it.

The NVidia NPC’s keep coming up in gaming discourse critique. They use Convai’s tools. (My last newsletter, btw, had links to 4 different code bases for interactive character AI code management.) Despite overwhelming negative response in social media, Sean Hollister admits the characters seem wooden and make mistakes, “But I’d love to see what a good writer could do with his backstory and motivations. I can now absolutely imagine games where NPCs remember what they’ve seen and react to the game’s script as it unfolds. … I just hope game developers use this to augment their games, instead of putting voice actors and writers out of work.”

Speaking of NPC’s and role-playing dialogue agents: see the DITTO system in “Large Language Models are Superpositions of All Characters: Attaining Arbitrary Role-play via Self-Alignment.”

✨ I strongly rec Mike Cook’s great article, “AI Is Already Better Than You.” His points are that quality is a poor critique of AI systems, and is not a solid ground to stand on as these systems improve very quickly. Besides, humans produce poor quality work under poor conditions, too:

“We already know, for example, that writers in the games industry are underpaid and overworked, and that the quality of writing in games often suffers because of it. … No, it's because writing is undervalued by the people funding games, in an industry that generally undervalues its employees anyway. Investors will accept putting higher pressure on writing teams because it saves money with an acceptable impact on sales. I think we can make similar arguments for many other aspects of game design and development being undervalued, too. We wouldn't make fun of hand-authored writing in a videogame for being bad, partly because it would be a shitty thing to do, but also becuase we know that it wouldn't convince anyone at the studio to invest more in the writing team. That's not how this works.

So pulling up AI technology on the basis of quality is risky, and it assumes that the people investing in this technology care about quality in the first place.

And finally, a fact about work in the game industry that is perhaps also true in many other jobs being cut right now, alluded to in the opening article:

You cannot shame this technology into disuse any more. That only works if quality is something the people with money care about. The problem with the continuing erosion of the games industry, the dehumanisation of game workers and the brutal treatment of outsourced work, is that many roles in the games industry are already treated as if they were automated. You are appealing to the better nature of money men who do not have one.

He makes many other good points, including a press for more research on how to use AI in good ways in design. Cory Doctorow said some similar vibe things last year about the anger over copyright and scraping being a displacement for the real pain over a lack of labor protection (here).

🔫 Palworld: the game labelled “Pokemon with guns,” is a smash hit in pre-release, despite being criticized for rip-off character design, derivative game play, and maybe even AI art (horrors!). I’m more disturbed by the whole goal of “capture cute Pals and make them labor for you.” This is a crafting game with cute slaves. The Steam entry for it says, “This is a completely new multiplayer open world survival craft game where you collect mysterious creatures called Pals in a vast world, and have them fight, build, farm, and work in factories.” From the Guardian article, “One of the enemy factions you face in the game is the Free Pal Alliance, an activist organisation dedicated to freeing Pals from slavery.”

Pocket Pair pic, from Guardian article.

Here’s the main designer Mizobe at PocketPair blogging (translated) 3 days before release, before it became a huge hit. It’s a fascinating ramble, a bit weird, and the part about how important guns are to the game and finding a gun obsessive animator who had never worked on games… well. It’s pretty much all like that.

“A 20-year-old junior high school graduate with no experience in the game industry and working part-time at a convenience store, was told by a small, unknown game company in Tokyo, ``We want to hire you as a full-time employee, so please come to Tokyo from Hokkaido.'' Normally people would suspect some kind of shady fraud.”

See also “Humans Have Improved at Go Since AIs Became Best in the World” by inspecting and learning new strategies.

UEVR wrapper for Unreal Engine games — So you have a VR headset and not enough games for it? Well, you can try this new tool that works on Unreal Engine games. It’s a bit tricky despite having a GUI, but there’s an interesting eco-system and tooling environment growing here. The UEVR site home page; their spreadsheet of games tried with it and results; a tool (Rai Pal) to identify which of your games were built with which engine; their “Flat2VR Discord” which is pretty useful and people are sharing their configs for trickier games. I got Abzü working in it, but controls were a bit wonky and would benefit from tinkering, I guess. (Also see CitraVR for 3DS games on a Quest.)

NLP & DataVis & Datasets

Tools for text data pipelines:

• IBM’s Unitxt: “Unitxt is a python library for getting data fired up and set for utilization. In one line of code, it preps a dataset or mixtures-of-datasets into an input-output format for training and evaluation.”

• Datatrove from Hugging Face: “DataTrove is a library to process, filter and deduplicate text data at a very large scale. It provides a set of prebuilt commonly used processing blocks with a framework to easily add custom functionality.”

• Dolma from AllenAI.

String search madness: Infini-gram: Scaling N-Gram Language Models to a Trillion Tokens: I said “holy shit” at how fast a query for documents (option 6) was on the HF demo but the API may be slammed right now.

“Learning High-Quality and General-Purpose Phrase Representations,”
for entity linking and disambiguation purposes. (I.e., relate “NY Times” and “New York Times.”) Code.

“In-Context Learning for Extreme Multi-Label Classification,” using DSPy and GPT4 (unfortunately a call is needed for each example). They require few training examples (“tens”) and achieve high quality results on 1000s of classes. Code!

Lilac just announced some cool doc clustering tooling and a waitlist for their online tool to do it fast. I need clustering during data exploration all the time!

There is now a PDF of the fab repo by Stas Bekman for Machine Learning Engineering.

Book Recs

Reminder that the recs sections are also sent to financial supporters as a separate mail you can opt-in or out-off, to make it easier to find and save them.

Engines of God, Jack McDevitt (sf). Also Deepsix, in the same series, not quite as good. Jack McDevitt writes space archaeology adventure novels. I like them because there are invariably monumental remains of a lost alien race, with us humans trying to find out who they were. A downside is that they read as a bit dated, especially in men responding to women based on their attractiveness; however, many of these women are also kicking ass, so. Engines of God would make a good movie: Exciting exploration on several different planets, trying to figure out if strange relics are related. The opening investigation features underwater tunnels in which a rosetta stone has been found but the planet is about to be terraformed. One of the other relics is a huge fake city on a moon, with weird inscriptions. Fab, nail-biting action scenes!

My Murder, Katie Williams (sf/thriller). A recent rec from Lisa Tuttle’s wonderful monthly Guardian SF books column (along with Lulabelle). This is about a group of clones post-serial killer murder, and one of them has doubts about what happened to her. (CW: post partum depression.)

Rosebud, Paul Cornell (sf). I wrote about novella Rosebud a bit in the mid-month newsletter intro on “monsters.” A small (corporate) ship staffed by prisoners, some of whom are AI personalities and others of whom are criminals, encounters a universe-bending alien entity in space. They’ve all been conditioned to love and admire their corporate employer/imprisoner. (CW: LGBTQ+ persecution.)

Thirteen Ways to Kill Lulabelle Rock, Maud Woolf (sf). 13th clone of actress and model Lulabelle Rock, who has decided to have her kill the other clones in order to deflect expected bad press for her upcoming film of Medea. It turns out that all the clones were created and sent off to perform various jobs for Lulabelle, ranging from “explore my artistic side” to “go to brunch for me today.” It’s funny and weird, with chapters based on Tarot cards. (One of the clones is a secretary and insists she is unique because “I can do Excel.”).

Three Eight One, Aliya Whiteley (fantasy). An autobiographical manuscript about a young woman going on her “quest” in a future fantasticaly world — she, like others in her small town, take off to walk “the Horned Road” with a backpack. She is followed by “the breathing man” and enounters various cute animals she knows as “cha.” The cha seem to take on different roles with different degrees of agency. It gets more and more surreal. I didn’t feel the footnotes of the far-future reader, a very light fame-story, were needed.

TV Recs

🍷 Drops of God, Apple TV+. My fav of the past month! A young woman in France and a Japanese man compete in a wine identification contest to win the collection and property of a famous (dead) wine critic. The woman is his daughter, and she has a horror of wine since her childhood under his pretty extreme tutelage. It takes places in France and Japan, in Japanese and French and English. Really excellent.

The Woman in the Wall (a BBC series). Grim but hard to look away: Women in a small Irish town who had been shut up in a Magdalene convent laundry seek reparations. A bunch of them had been pregnant girls, shut up for sins, and their children were stolen. Lorna, at best neuroatypical and at worst schizoid, is looking for her child. She sleepwalks and sees things. A cop from Dublin, who was raised in one of the convent “mother and child” homes, comes looking into the murder of a priest. Very dark, but compulsive watching. Some plot point detail issues, but on the whole, thumbs up. (CW: descriptions of abuse and violence.)

The end of Fargo was really good, quite the Greek tragedy. There is not a ton on “Sin Eaters” on Wikipedia, but some here. I also enjoyed (but found melancholy) the last season of Endeavour which I missed last year. (CW: alcoholism.)

Game Recs

Highland Song from Inkle. I like the style of the art, music, voices, and writing. It’s very atmospheric, navigating a little girl over indecently dangerous Scottish mountains with scraps of maps to “help.” I say “help” because I am bad at the mechanics of using them and navigating the 2D layered landscape. I like wandering in it and finding things, but am not “winning” in any way.

Otherwise, still failing to find Exit 8 and being trounced in Prey despite my neuromods. For which I give you:

VR

I recently upgraded to a Quest 3, to try pass-through, AR, and just get better res. Pass-through is only so-so, still a pixelated screen view albeit color, although text on UIs looks good. The res win is obviously there only if the game is made for higher res: so blocky graphics games don’t look better (e.g., Walkabout Mini Golf, Windlands 1, even NMS doesn’t benefit much—plus NMS currently has a VR bug making it unplayable).

Invisible Hours, from Tequila Works, is a VR murder mystery at Tesla’s mansion, with guests Tom Edison (a jerk), Sarah Bernhardt, other miscreants, and a blind butler. Tesla’s body lies on the floor in the front hall. You watch their scenes play out in chapters across the house, can pause and rewind time, and revisit a chapter and different scenes in other rooms. There are notes and photographs sprinkled around the house in the usual background color manner of story games. I really like it! The animation is a bit jerky and old school, but the voice acting is solid.

7th Guest, another creepy house game, looks great on the Quest 3— they updated their textures for it. Still slogging away but liking it: I need to allocate time for each room because it only saves by room and a room has multiple puzzles.

Asgard’s Wrath comes with the Quest 3. I played the start, it’s a very AAA flashy game; it’s still cartoon-ish in the headset but look and moves well on the Q3.

Asgard’s Wrath image, source Forbes review (very positive)

Abzu with UEVR (see above in the Games News section) looks very nice and is obviously more relaxing as an underwater diver than Subnautica, which as you know I quit from fear. 🐙

Poem

Art & life
drunk & sober
empty & full
guilt & grace
cabin & home
north & south
struggle & peace
after which we catch
a glimpse of stars
the white glistening pelt
of the Milky Way,
hear the startled bear crashing
through the delta swamp below me.
In these troubled times
I go inside and start a fire.

—James Harrison from Cabin Poems, via Tom Snarky

I’ve been super busy with work and work travel, sorry this is late and has no gifs in it. I will up my game (and play more games). Let me know if you had fun with this or if you want links to anything I didn’t include.

Best, Lynn (@arnicas on the sfka twitter, mastodon, and bluesky and now Threads)