Prompting a wide range of facial expressions in Stable Diffusion

“Changing a character’s facial expression is quite easy to do!” -Shrek

Tip 1:  Try more than 1 word to describe the expression

Here’s an arsenal of words to help you think of the possibilities

grinning, cheerful, smiling, chuckling, amused, amazed, pouty, astonished, relaxed, pleased, curious, surprised, interested, attracted, bewildered, unhappy, annoyed, frustrated, angry, agitated, furious, mad, shouting, screaming, depressed, irritated, grumpy, yawning, worried, contemptuous, horrified, nasty, jealous

Tip 2: Add emphasis by making the terms more negative or positive.

Certain models may react differently to “expression” so try “face” as well, and add contrast with a negative to help guide narrow down the expression you’re looking for.

Usage:

looking ((curious face))
looking ((curious expression))  [[serious, determined]]

 

Tip 3: Move the expression closer to the front of the prompt

The words towards the start of the prompt matter the most, so try moving the expression closer or further towards the back to control how much the AI pays attention to the request.  Consider these four ideas:

  • /render portrait of a beautiful woman, looking tired
  • /render portrait of a beautiful woman, looking (tired)
  • /render a (tired looking) beautiful woman
  • /render a tired looking beautiful woman

The longer the prompt is, words in the tail end of the prompt may need (emphasis) to be taken into consideration.  But doing that too early in the prompt may result in a cartoony facial expression, too.

Tip 4: Add an expressions LoRA

We have mini-models specifically for puppeteering expressions on our models lookup page.  Try searching for the one you want, or browse the Expressions and Poses tags. An example of what you’ll find is below. Many of these models have Triggers to enable more than a single expression, so there’s hundreds of possibilities below.  Some of the anime and realistic loras are flexible enough to work in any art or photographic style, so don’t let the thumbnail images discourage you from experimentation.

The Lora method does have one limitation: they are trained on real or anime faces, meaning that they will introduce some new biases based on those images, such as adding long hair to a man with short hair if they were mostly trained on women, altering the chin structure, and so on.

Try one from a low weight of about 0.3 to 0.5 for best results.

Example:  <waking-up:0.5> yawn