JACK2002

Background

From 1996 through 1998 I worked as a junior developer on an application titled "Liquid Motion". The application first saw life as an offering from a small startup named Dimension X and was actually released as "Liquid Motion Pro", but following that company's acquisition by Microsoft, and after substantially more development effort, it was rebranded as Microsoft Liquid Motion.

After the acquisition by Microsoft we introduced another runtime based on the DirectAnimation multimedia runtime plus a virtual machine interpreter, and a binary file format that targeted the virtual machine. This runtime allowed for much richer animation based on the underlying DirectDraw and Direct3D platforms, but I won't discuss that here.

The application was an animation authoring tool for web developers (loosely comparable with Macromedia Flash). It output animation files in the "JACK" format (for an early name for the product, Java Animation Construction Kit) with the JCK extension (conceptually similar to the SWF format). The runtime was an "applet" - a set of Java classes which would be downloaded to the client Web browser when visiting a page hosting an animation.

The Java-based runtime was the advantage which was promoted over the competitive product, Macromedia Flash. At the time, Java was widely supported by Web browsers (predominantly Netscape Navigator 2.0 and Microsoft Internet Explorer 3.0). When navigating to a page containing a Java applet, the browsers would download the Java class files, cache them locally (for fast access when returning to the page), and run the applet in a sandboxed virtual machine - all without prompting the user or requiring setup steps, and independently of the operating system (Windows, Macintosh, Unix). In contrast, the Flash web runtime (Shockwave for Flash, hence the SWF extension) was a binary plugin component which was not available cross-platform, and had to be downloaded and installed manually before a page containing the animation was correctly visible. Also, the Liquid Motion runtime was extensible - the JCK file could call on a custom Java class type added by a developer which would also be seamlessly downloaded. At the time, this sort of extensibility was not supported by Flash.

For completeness, I should also mention Liquid Motion's successor: Microsoft Vizact 2000. Whereas Liquid Motion (like Flash) created animations that ran "in a box" on the Web page, Vizact animated "outside the box" - it let you animate arbitrary content on a Web page. If Liquid Motion's competitor was Flash, Vizact's competitor was Dreamweaver. You could load a web page, select an image and animate it on a motion path, or select a paragraph and set it to appear when the user clicked on a title. The timing and behavior concepts were otherwise the same, although it was much more powerful, with a visual interactivity system, wizards, media browsing, and a far more powerful timeline.

The user interface was generally aligned with Microsoft Office 2000, and the product was targetted at office workers who needed to take static content (e.g. a Word document saved as HTML) and "activate it" for the Web - a not entirely crazy idea circa 1999. There's a decent review here.

Although I was very proud of working on Vizact, it really went after a niche (or non-existent) market. Web developers had tools they were happy with (Flash, Dreamweaver), and office workers asked "can I use this animation in PowerPoint?" - so after shipping Vizact, we took our expertise and added new animation capabilities to PowerPoint 2002 in Microsoft Office XP.

The web animation landscape changed rapidly along several axes. On the Windows platform, the addition of ActiveX download support enabled single-click install, letting non-technical users get a seamless experience. As Flash became a de facto standard, Web browser vendors started including it in the browser setup, eliminating the deployment hit. Flash also evolved to include a scripting language (Lingo) and richer multimedia capabilities. Finally, the legal battles between Sun and Microsoft over adherence to licensing requirements (or the right to extend the language, depending on who you ask) effectively killed client-side Java use. So Liquid Motion didn't last long in the marketplace.

That's not to say that Liquid Motion wasn't without its own faults.

Behavior-based vs. Frame-based

The fundamental animation paradigm in Liquid Motion was behavior-based, rather than Flash's frame-based model. In a behavior-based model, animations and events are produced by objects called behaviors which apply to another class of objects called actors. For example, an image is an actor, and a motion path is a behavior. A motion path applied to an image is what makes an image move over time. The motion path contains information about the actual path (typically a series of points which are interpolated) and timing (when does the behavior start and stop, does it repeat, etc).

The behavior interacts with the actor by changing the actor's properties based on the current time. For example, a motion path behavior simply changes the actor's position over time; a transition behavior changes the visual filter applied to an actor over time. Behaviors can be very complex - for example, a behavior can be implemented which sets the actor's position to the current position of the mouse pointer - which will cause the actor to follow the mouse. This behavior can then be used to implement a range of effects from custom cursors to drag-and-drop of objects within an animation.

This leads to a concept called a scene graph, where the state of the animation at any point time (e.g. for playing the animation, or interactively editing it) can be determined by evaluating the contribution of all of the behaviors and actors. This is extremely elegant, in that multiple behaviors can cooperate to create very complex interactive and non-interactive effects.

This is in contrast to frame-based animation, in which the actor's state is defined at multiple points in time, called key frames. Typically, the animation author positions an actor at a starting position at time t=0, and then sets a different position for the actor at some later time, say t=10. When the animation runs, the position is interpolated over that time range. To handle interactivity like following the mouse, user specified code is usually required.

I'll go out on a limb and hypothesize that, in most cases, frame-based animation is actually simpler to author. A frame-based animation authoring environment corresponds closely to the WYSIWYG paradigm - as an animator, you go to frame X and specify how everything will appear. Then you go to frame Y and specify how everything should appear. You let the tool figure out frames X+1, X+2 ... Y-2, Y-1 and tweak any if necessary. In contrast, a time-based animation tool typically requires the animator to select the behavior and parameters in a somewhat abstract sense. Either the tool needs to emulate the ability to tweak intermediate frames (by generating additional behaviors) or the animator must understand the runtime mechanism at a deep level.

Word 2002's Styles and Formatting task pane does an amazing job of merging the two approaches - it's one of my favorite pieces of UI in all of computer-dom and something I use heavily on a daily basis. You can be lazy and use direct formatting, then come back and "clean up" and apply structure. It allows a very ad-hoc approach to formatting without compromising the results.

Since most animators are graphical artists rather than software developers, providing an easy to use canvas for creating animations - even if it's a more tedious canvas - is a clear advantage. In the same way, direct formatting (swipe text and make it bold) is used far more commonly than named styles in Word; the latter requires a structured approach, background knowledge and planning.

Runtime Size and Capabilities

The Liquid Motion runtime (JACK) was designed in conjunction with the authoring platform - which was also a Java application. Indeed, the same object-oriented classes existed for the same objects (in slightly different flavors) at both design-time and runtime. This greatly eased the extensibility story - for example, to add a new type of multimedia element (called an actor) to Liquid Motion you simply created a new Java class, say FormButton. This FormButton would implement design-time properties and participate with design-time features (selection UI, property dialogs, serialization, etc.) as well as runtime features (rendering, animating, de-serializing, etc.).

This object-oriented nature extended into the rest of the application and runtime - for example, there were serialization and de-serialization classes, base classes for actors and behaviors as well as classes for each actor and behavior type, objects for handling vector shapes, etc. While this made the development of the application extremely rapid and simple, the runtime suffered.

The JACK runtime came in around 270k - which was comparable to the size of the Flash runtime. During the mid-1990's - pre-broadband - that was a substantial download size relative to an animated GIF. While not all of the class files were necessarily downloaded each time it was still a large hit. Much of the size was due to the approximately 50 class files - each Java class file has an approximate 2k overhead, before any of the specific code is accounted for.

One of my tasks as a junior developer on the team was to look at optimizing the size of the runtime. I probably shaved 50k off with various approaches - for example, detecting methods that were not necessary at runtime or could be eliminated, better use of inheritance, etc. 220k is better than 270k, but not by much. The size of the runtime limited adoption during the early period before Flash reached critical mass, since prospective users had to justify the substantial download for site visitors. Although I think the other factors mentioned above were an order of magnitude more important to the demise of Liquid Motion, this is still something that has bothered me throughout the years.

Fast Forward - JACK2002

Several years later (Fall 2002) after having several more years of development experience under my belt, I suddenly realized (4 years too late) that I knew how to shrink the runtime size down dramatically. I hadn't done any Java development in a few years but I dusted off my copy of VJ++ and got to work. I followed a handful of key mantras:

The end result was a runtime comparable to the JACK format in ~20k, with a few new features in some cases. This was achieved with the following high-level approaches:

Features

Feature JACK Vizact SMIL 2.0 JACK2002
File Format Custom HTML+TIME XML pseudo-XML
Timelines
Parallel t:par par par
Sequence t:seq seq seq
Exclusive excl
Actors
Scene Scene div (HTML) scene
Images ImageSequence img img bitmap
Vector Shapes VectorActor VML animation vector
Text TextSequence HTML text text
Audio AudioSequence t:audio audio sound
Particle Effects MagicActor*
Behaviors
Motion Path MotionPath an:move animateMotion path
Scale Pulsate an:scale animate** scale
Rotation Spin an:rotate animate** rotate
Color ColorCycle*** an:color animateColor color
Random Motion Jump
Interactivity
Mouse Following AvoidFollow
Interactivity Trigger onclick onclick
Hyperlink URLLink a (HTML) a onclick
Other
Transitions an:effect
Filters an:actor

* The particle effects ("Magic") were one of my first additions to Liquid Motion. The idea came from a storyboard for animated advertisements. At one point the team started working on a product called "Liquid ProMotion" which would output banner ads. The ads were Java applets specific to the animation (rather than being a generic runtime) with the class files plus media in the 10-20k range permitted at the time. The storyboard included Disney-esque sparkles; I came up with a simple particle system algorithm with mouse movements triggering particle generation. We decided to roll the effect into Liquid Motion and added several other variants, such as bubbles, clouds and smoke.

** SMIL uses generic animation primitives (animate and set) which act on arbitrary properties (another idea from Vizact). So instead of manipulating a scale property you animate both width and height.

*** ColorCycle (which begat SMIL's animateColor) was another early addition of mine to Liquid Motion inspired by color palette animation, a popular animation trick in the early 1990s.

File Format

Grammar

Pseudo-XML Syntax:

Grammar:

element ::= ws* ( (opentag element* closetag ) | emptytag )?

emptytag ::= '<' tagname (ws* attribute)* '/' '>'

opentag ::= '<' tagname (ws* attribute)* '>'

closetag ::= '<' '/' ws* '>'

attribute ::= attrname ws* '=' ws* '"' [^"]* '"'

tagname ::= alphanumeric+

attrname ::= alphanumeric+

alphanumeric ::= [a-zA-Z0-9]

ws ::= space | tab | newline | linefeed

Some of the many differences from real XML:

Common Attribute Sets

core attribute set

id = unique id

class = extension class name; if specified, an instance of the specified class is instantiated. The class must be a subclass of Anim and override onLoad(), onTick(), onPaint(), onClick() and onEvent() as appropriate. The root element cannot be an extension class.

timing attribute set

begin = start time, in seconds; double; default 0

end = end time, in seconds; double; default Infinity

dur = duration, in seconds; double; default Infinity

repeatCount = number of repetitions; double; default 1.0

fill = "remove" | "hold"; default remove

Timing Elements

par

core attribute set

timing attribute set

seq

core attribute set

timing attribute set

Children are animated in sequence. Each child must have its dur set or the default (Infinity) is assumed and the next child will never animate.

Actor Elements

event attribute set

onclick = scriptlet (see below); default none

onenter = scriptlet (see below); default none

onexit = scriptlet (see below); default none

scene

core attribute set

timing attribute set

event attribute set

x = left edge; integer; default 0

y = top edge; integer; default 0

width = width in pixels; integer; default none (no clipping)

height = width in pixels; integer; default none (no clipping)

scale = scale factor; double; default 1.0

bgcolor = background color; hexadecimal (rrggbb); default none

text

core attribute set

timing attribute set

event attribute set

text = text to display; default none; required

x = left edge of text; integer; default 0

y = top edge of text (not baseline); integer; default 0

scale = scale factor; double; default 1.0

bgcolor = background color of text; hexadecimal (rrggbb); default none

color = color of text; hexadecimal (rrggbb); default white

family = "Serif" | "SansSerif" | "Monospaced"; default "SansSerif"

size = font size, in points; double; default 10

weight = "normal" | "bold"; default "normal"

style = "normal" | "italic"; default "normal"

vector

core attribute set

timing attribute set

event attribute set

points = path-points (see below); required

x = origin; integer; default 0

y = origin; integer; default 0

scale = scale factor; double; default 1.0

angle = angle of rotation, in degrees; double; default 0.0

linecolor = RGB outline color; hexadecimal (rrggbb); default none

fillcolor = RGB fill color; hexadecimal (rrggbb); default none

bitmap

core attribute set

timing attribute set

event attribute set

src= URL of image; required

x = center position of image; integer; default 0

y = center position of image; integer; default 0

width = width in pixels; integer; default none (natural size)

height = width in pixels; integer; default none (natural size)

scale = scale factor; double; default 1.0

angle = angle of rotation, in degrees; double; default 0.0

hframes = number of frames in horizontal image strip or grid; integer; default 1

vframes = number of frames in vertical image strip or grid; integer; default 1

rate = frames per second; double; default 1.0

sound

core attribute set

timing attribute set

src= URL of audio file; required

Behavior Elements

behavior attribute set

actor = target_id ; default none (behavior operates on parent element)

path

core attribute set

timing attribute set

behavior attribute set

points = path-points (see below); required

scale

core attribute set

timing attribute set

behavior attribute set

initialScale = initial scale; double; default 1.0

finalScale = final scale; double; default 1.0

rotate

core attribute set

timing attribute set

behavior attribute set

initialAngle = initial scale, in degrees; double; default 0.0

finalAngle = final scale, in degrees; double; default 0.0

Rotate only functions on Vector, Bitmap and Text actors. Rotations on Bitmap and Text are performed in software and allocate a new image each frame and are relatively slow.

color

core attribute set

timing attribute set

behavior attribute set

colorProp = color property to animate; default "color"

initialColor = initial scale, in degrees; double; default none (use current color)

finalColor = final scale, in degrees; double; default none (use current color)

Detailed Attribute Syntax

scriptlet

scriptlet ::= statement [ ";" scriptlet ]

statement ::= object "." member [ "=" parameter ]

Where object can be "window" or an element id. For elements, this becomes a call to object.onEvent( member, parameter ), where "start" and "stop" are natively supported members. For "window", supported members are "href", and the parameter specifies a URL to navigate the page to.

path-points

path-points ::= (line | curve | end)*

line ::= ("m"|"l") x y

curve ::= ("c"|"v") x1 y1 x2 y2 x3 y3

end ::= ("x"|"e")

"m" and "l" specify a straight line to the next coordinate

"c" and "v" specify a Bezier curve with the current point and the next 3 coordinates

"x" and "e" end the path

Source

Examples

Obviously, you need a Java runtime installed to view these. From about 1996 through 2008 that was pretty much guaranteed if you were viewing this Web page.

These suck, as they were hand-coded. I suppose that's why we thought they was money in creating authoring tools. Note that the applet these reference is actually compiled with debugging output, symbols, and no optimization, so it's actually closer to 30k in size than 20k in size.

Performance

The current implementation has performance issues when rotating images and text. This is done by allocating new pixel arrays and images. In the case of images, one new pixel array and new image is allocated for each rotated frame. In the case of text, three new pixel arrays and images are allocated - the first to render text to a surface, the second to produce an image with an alpha channel, the third to produce the rotated image. This takes the time to render a test frame from 30-40ms to 100-200ms (a reduction from 25fps to 5fps).

Ways to improve performance:

Future Work / ToDo