Animata

Animata "is an open source real-time animation software, designed to create animations, interactive background projections for concerts, theatre and dance performances."

It is the closest thing out there to a real time version of After Effects' excellent Puppet Tool, and lets you create complex animations from 2D images. It works in much the same way, although is nowhere near as automatic. To create a puppet from a 2D image in Animata you have to manually create and rig the mesh yourself. This requires you to : 

1. Import the image you want and move/scale it to fit on the screen.

2. Draw a series of points around your model to be used as vertices for a mesh.

3. Auto-create a triangle mesh by grouping these vertices together.

4. Manually fix any errors made by the auto-triangulation

5. Add points where Joints will go

6. Create bones by connecting joints

6. Attach each bone to all the right vertices (so that the mesh moves with the bones nicely)

7. Adjust the weightings on the joints to create spring constraints

8. Move the joints around and test the meshes movement

9. Keep adjusting weightings, attachments, and skeleton structure until you get a natural looking motion

This would all be well and good, except Animata is missing a really rather crucial feature... 

AN UNDO FUNCTION!!! 

Trying to go through all these steps, where there are lots of irreversible ways to mess you model up (if you wiggle the wrong bone about by accident it can bunch up, and you have to go back to your last save) is a somewhat painful experience. You have to be extremely careful and methodical.

It took some time, but after a bit of tinkering I figured out a system that worked, and was able to create some rigged meshes for a few different toy hacks : 

A triangle mesh created for this BeaverWhale (or maybe it's a WhaleBeaver?)

And one created for this (slightly menacing) box with the arms of a baby

The baby with the skeleton in place.
In this picture, I am attaching vertices to the bone in the hand on the right

A fully rigged puppet, with the darker bones acting as springs to constrain the motion/

Once you have created these meshes, you can give the joints names, and then control them by sending them OSC messages. (OSC is a super handy messaging protocol for sending information between programs. Basically MIDI for programmers)

Now for the fun part - To manipulate them in realtime! There are a number of ways to get motion tracking data into the right format of OSC message, but my preferred tool of choice was TouchDesigner. I created a custom patch to read in information from a Leap Motion hand tracking controller, and remap it into the right ranges, and message formats to control the puppets appropriately. I designed the patch to make it extremely quick and easy to tweak the ranges and change the names of the control messages.

TouchDesigner patch to contol Animata with a Leap Motion controller

The only model I successfully got to move in a vaguely natural way

It is very much a trial and error process, but some of the results were really quite impressive, and when you get it right it is super fun to play with. Unfortunately, as it is homebrew software developed by a very small team it is just not quite stable enough to use for a serious project. It has an incredibly annoying habit of incorrectly resizing your output display to be either way too big or way too small, and there are a lot of ways to mess up your project irreversibly, forcing you to quit and reopen your last save. This is simply not workable for an installation context unfortunately, at least without heavy supervision, and frequent restarts. 

Here is a video of me destruction testing two models at once :