TITAA #17: Crashing in Scenic Places, With Flan

(Emily Dickinson envelope poem, from The Gorgeous Nothings)

These fragments of Dickinsonian text are fascinating. The theory is that she wasn't just being frugal and using any old materials she had at hand, but that the shape and layout of her hieroglyphic text mattered as well. That we should see these as visual poems. Here is the translation of the above image, readable, if doubly cryptic due to positioning:

The book sounds very good.
 

AI & Creativity Projects

Still lots of CLIP, sorry :)

FairyTailor: Multimodel Framework for Human-in-the-Loop Visual Storytelling.  This app allows you to choose text or images to go with your typed text context to co-create an illustrated story.  As often with these things, the image search (using Unsplash data and CLIP) is much better than the GPT-2 text completions, even the "high quality" slow ones.  But it's fun and the code is there to plug in better models!

More on CLIP for image search: Vladimir Halkatov has a great project (with Colab) illustrating cool text search on the Unsplash dataset and some attempt at fancy similarity combos using an image + text (the combos didn't work wonderfully on my trials with castles).  Handily, he makes available his feature encodings for the Unsplash dataset.  A great teaching demo project, although honestly Unsplash should just use it all, because they have terrible search.

Halluzinator: I enjoyed this article on uses of Ryan Murdock's notebooks (from his patreon) to generate illustrated stories and poems using CLIP and GPT-2/3.  But I particularly liked Glenn Marshall's imagery (video link), a couple excerpt examples below:

I liked the moxy of CLIP-GLASS ("Generating Images from Captions and Vice Versa") which supports image generation via text input from a variety of StyleGAN models as well as generation of a text caption using GPT-2.  The latter is less successful than the image gen.  Here is a panel of lovely floral texture samples from an attempt to generate a Tarot card. (Okay, the card part was a failure.)

Leaving CLIP behind for scenery generation...  GANcraft looks amazing, but no code yet:  Use MineCraft (or other block editor) to design a world and then render it photo-realistically. "Our method takes a semantic block world as input, where each block is assigned a label such as dirt, grass, tree, sand, or water. We represent the world as a continuous volumetric function and train our model to render view-consistent photorealistic images from arbitrary viewpoints."

 This is another "generate infinite landscapes" paper which looks extremely cool: Aligning Latent and Image Spaces to Connect the Unconnectable. There is code; I haven't tried it yet.

Tool and Articles Lists

Admire the bibliography from the Association for Computation Creativity (who run the ICCC conference).

Everest Pipkin's enormous, tagged list of creative design tools: 

"This is a list of small, free, or experimental tools that might be useful in building your game / website / interactive project. Although I’ve included ‘standards’, this list has a focus on artful tools and toys that are as fun to use as they are functional."

There are game engines, map makers, machine learning, web design, UI design, etc... all in tag filterable combos.

Gene Kogan updated the contents of Machine Learning for the Arts (ML4A).  There are training materials, artistic model demos in Colabs, and lots of documentation.

There's an affordable book on Procedural Content Generation in Games (i.e., free online), by Shaker, Togelius and Nelson.
 

NLP-Specific

Tuomo Hiippala has concluded his online Applied Language Technology book and video series teaching Python, NLP, and modern spaCy.  It looks excellent.
 
SummVis: A toolkit for visualizing your summarization models. "Through its lexical and semantic visualizations, SummVis enables in-depth exploration across important dimensions such as factual consistency and abstractiveness."

OCTIS: Optimizing and Comparing Topic Models is Simple!  (Not just your standard LDA. Good looking docs, many models and some colabs.

Textacy is an NLP lib for working around spaCy, before and after.  Some things it will do: 

• Clean, normalize, and explore raw text before processing it with spaCy

• Extract structured information from processed documents, including n-grams, entities, acronyms, keyterms, and SVO triples

• Compare strings and sequences using a variety of similarity metrics

• Tokenize and vectorize documents then train, interpret, and visualize topic models

(Also a reminder of SpikeX for knowledge extraction with spaCy pipelines which I mentioned last time.)

Overview article: The NLP Cookbook: Modern Recipes for Transformer based Deep Learning Architectures, by Singh and Mahmood.

Excellent overview by Eugene Yan of modern text search at various companies: "Search: Query Matching via Lexical, Graph, and Embedding Methods."
 

Books

Klara and the Sun by Kazuo Ishigura: This is lovely if a bit sad.  An "artificial friend" with surprising human insight is bought to take care of a sick girl.  This isn't so much about AI as about how people treat belongings.  And many other things about sentience and humans.

The Absolute Book, by Elizabeth Knox. If you like books about faerylands and folklore that is real, this is a good one for you.  It opens with an almost off-topic murder and revenge, but there are eventually magical angelic beings, faeryland gates, legends you gradually see coming "true," ravens who talk, and demons doing deals.  Rollicking and exciting!  (If you don't know her book The Vintner's Luck, it is an amazing LGBTQ romance, one of my favorite books.)
 

TV

Shadow & Bone on Netflix was trashy fun.  I have bailed on a number of other YA Netflix fantasy series, either due to characters or writing, and this one held my attention.  I'd watch season 2.

The end of S2 of For All Mankind was very good.  S2 was more politics and personal drama and fewer space disasters than S1, but still excellent.

Tell Me Your Secrets on Amazon Prime Video was a surprisingly good thriller featuring a bunch of very troubled gray-zone characters with things to hide making (increasingly) poor decisions.
 

Games

I finally played the excellent and well-regarded Firewatch, an interactive fiction mystery set in the Wyoming wilderness.  It does a good job of getting you into the scenery and out of your confinement.  I then watched a bunch of talks about its design and construction from Game Developers Conference videos (start here).  I admit I was surprised that the idea of a "no failure" world exploration game with a narrative might not be considered a game as late as 2019.  This "game" is pretty famous, too.

In another get-out-of-the-house via software move, I got MS's Flight Simulator.  I had read some of the press about the satellite image processing and AI-involved 3D world creation by Austrian company Blackshark (e.g., this) and finally talked myself into it.

You don't just "buy this game on Steam," you need to ensure you have 160G disk space (I needed an external SSD drive with good throughput) and speedy download speeds and a good data plan.  If you want to play in VR, you have to follow a lot of online advice with debugging tips for your set up. I didn't have to overclock myself, although many do, but did have to tweak fidelity settings a little. (I have an i7 and an RTX3060 on a pretty new laptop.)

I had never used a flight simulator and was, uh, nervous.  I was afraid the plane flying stuff would take over from the scenery, and it can, but you can also arrange so it doesn't.  The game is surprisingly forgiving in some respects: there is an "AI pilot" you can turn on if you want to just watch the view out the window, and even an external view of the plane with full 360 rotation over the landscape.  However, the AI pilot has more than once casually crash-landed me, which adds a surprising amount of nervous excitement to using AI mode, so I don't know what's up with that?! 

Speaking of crashing: It's pretty good about presenting crashes in an acceptable manner.  And I'm very good at crashing. You just bounce around on the ground, no simulation of flaming death.  I haven't tried water yet, which is a recurring nightmare of mine, so we'll see.   But I've crashed in a lot of lovely places so far,  even without AI pilot help:  the coast of France near Mont St Michel (see pic), the Outer Skerries, between Donegal and Derry, Benbecula, Land's End, Alesund...   In one scenario, I watched well-articulated scrub grasses blow in the North Sea wind outside my upside down cockpit, which I thought was nice detail.

I'll tell you: flying at sunset over Mont St. Michel under a full moon rising on Monday night was something else.  And then crashing at St. Malo! Good times!!

For other flying fun, the game Cloudpunk (not VR) involves flying a delivery truck around a dystopic Bladerunner-esque city, which is relaxing if dark evening fun. You're in a pretty screwed up story universe. The main character has her dog installed as the AI personality in her flying transport, because she can't afford a real dog body.
 

Recipe

I have made a lot of crème caramel, essentially flan, this spring.  I mean, A LOT.  Since finding out I can do it in my Instant Pot knockoff, I probably make 2-3 batches a week.  Buy the caramel sauce, don't be crazy. Here it is:

1 and 7/8 cups of milk (1/3 liter), boiled with 1/3 cup of sugar and a tsp of vanilla.  Whisk (hot) into 2 beaten eggs. Put a tsp of caramel into 5 ramequins or yogurt jars, then pour in the liquid.  Put on a steamer tray inside your electric pressure cooker, cover the containers so they don't get water in them, and cook at 8 minutes on high or the level below high.  Chill.

Poem

I take out my anger 
And lay its shadow

On the stone I rolled
Over what broke me.

I plant three seeds
As a spell. One

For what will grow
Like air around us,

One for what will
Nourish and feed,

One for what will
Cling and remind me--

We are the weeds.

-- Ada Limón, On a Pink Moon

I'm still in France, still unvaccinated, and investigating residency options elsewhere. I'm also starting to think about more part-time freelance work. If you know of any freelance NLP or AI or data-related gigs, let me know? I am fine with labeling, training, tuning etc, and writing up demos or prototypes. 

I hope you are all hanging in there.  If you like my newsletter, which is free but time-intensive, let me know with a reply?  Pass it on?  Say thanks by buying me a coffee?

Best, Lynn (@arnicas)