Every night around 4:30am, my laptop wakes itself to record its latest "dream": a short dreamscape description and associated generated image. I started experimenting with generative AI in my last role at Google. Since leaving the company, I've wanted to get a sense of publicly-available models, so I hacked together this experiment. I thought it'd be fun to create an automated image generation pipeline: prompt an LLM for a description, feed that into an image generator, and see what pops up. The image and description above regenerate daily - check back tomorrow for a new dream!
Thanks to some awesome open-source work, it's possible to run both LLMs and image generation on Apple silicon with decent performance. My base model MacBook Pro (10-core M2 Pro, 16GB RAM) is too small for the latest and largest available models, but it can handle some smaller models nicely. This experiment uses Llama 2 13b and Stable Diffusion 2.1, running locally on my MBP.
I used llama.cpp to run Llama 2 13b locally. The repo has simple usage instructions that make it easy to get up and running quickly.
Apple provides a repo for running Stable Diffusion with Core ML on Apple silicon (announcement). It took some fiddling with dependencies to get an appropriate conda environment set up for this one, but it works nicely once things are in place.
I'm still experimenting with prompts to get Llama to reliably produce "dream" descriptions that might turn into interesting images. I wanted the descriptions to retain a somewhat dream-like feel (as opposed to e.g. just having it spit out a bunch of image prompt keywords), though that might be constraining the output image quality a bit. Right now it seems to like generating images of large mushrooms 🙃 I wrote some simple scripts to automate prompting Llama, extracting the image description it emits, and feeding that description into Stable Diffusion. The scripts run automatically as daily cron jobs. Fun fact: by default, cron jobs don't start if your Mac is sleeping, but you can induce scheduled runs by waking your machine with pmset. Had to learn this one on the fly!
My laptop automatically pushes daily updates to this GitHub repo; you can look through the commit history to see previous results. I've used this approach to generate a bunch of images. It's fun to see what pops up. I collected some favorites here!
