e-Motion: a system for the development of creative animatics Santiago Negrete-Yankelevich and Nora Morales-Zaragoza División de Ciencias de la Comunicación y Diseño Universidad Autónoma Metropolitana (Cuajimalpa) Av. Constituyentes 1054, México D.F. 11950, México {snegrete/nmorales}@correo.cua.uam.mx Abstract This paper introduces e-Motion, a software system for the creation of animatics1 , which are important tools within the process of creation of animated graphics for TV. This type of animation, generated by the system from plots in plain text, allows production teams to envision how a final motion graphics piece can be developed. We argue that our system plays a creative role within the generative process. Specifically, our work is linked to a real production team, involved in the creation of animated shorts, called Imaginantes, for Mexican television. Introduction Computer systems intervene more and more in creative practices. They play different roles in teams of people working on projects that produce innovative work and whose overall process can be deemed creative by a suitably selected group of human experts. Just as in the case of creative teams formed strictly by human members, the blame for creativity can be distributed amongst the team members including computer systems (Maher,M.L. 2012). In this paper we describe e-Motion, a computer system that builds animatics for a pre-production process to create motion graphics. In order to test the system we embed it in the process of Imaginantes, a TV production of a series of animated shorts (one minute long) based on texts of different authors aimed at encouraging viewers to get involved in Music, Literature, Fine Arts and Film. The first season (12 shorts), launched in October 2006, captivated young audiences who shared and published them through different social media and the web; some people even created their own shorts. Since then, Imaginantes has won numerous awards, and it’s on the 4th season with a total of 46 shorts produced and delivered in several media (Imaginantes, 2006). 1 An animatic is a visualization tool used in the pre-production process of an animation that informs about movement, narrative structure, framing aspects and visual effects to the production team before the animation is actually done. e-Motion is part of a research project on computational creativity where we take a proven, creative process that produces a recognized, valuable product and use it as an environment to test our systems. We are interested in studying how the overall creativity is affected if computer systems take over different roles within the pre-production stage. The Imaginantes team and process are well defined as well as the work products that must be produced. We hypothesize that all stages and work products contribute to the overall creativity but we test our system’s creativity by inserting it in the human process, to see how it affects the outcome and ask human members of the team to assess the system's performance. There are two main advantages in this approach: 1. The system is assessed within a recognized real-world creative process, so we can avoid the toy-world generalization problem. 2. Our system plays a role within a human process, so it is easier for the human members of the team to assess the system's performance. They are experts in the area, they know very well what to expect. The following are the main motivations for our work: • Study computational creativity within real-world creative practices • Understand the creative process of multi-sensorial content, sound and movement of visual narratives. • Develop a computational system that works collaboratively in the creative production process of visual narratives. • Develop sound user criteria to evaluate the system as a valuable tool. • Experiment with automatically-created motion graphics to study how different frame, color and graphic element combinations transmit emotional content to an audience. We want to maximize narrative appeal while preserving the logical structure suggested by the original plot. (Malamed, 2009). Animatics constitute an intermediate step towards a full motion graphics piece and, although they depict simple Proceedings of the Fourth International Conference on Computational Creativity 2013 184 representations, they have a complex structure and most of the high level architecture and elements of the final product. They are built from storyboards,2 which in turn, are assembled from scripts and constitute an important tool within the process to develop motion graphics. They convey decisions about editing, camera framing and special effects. A production team can discuss several of these options using various animatics before embarking on the production stage, saving resources on this costly process. Hart actually describes animatics as the “future of motion control” to stress their importance (Hart, J. 2008). As a starting point of our research into the nature of the creation of animated stories, we use the output of Mexica (Pérez y Pérez and Sharples, 2001), a computer system that generates story-plots about characters, places and themes of pre-Hispanic folklore; in particular, that of the Mexicas (most commonly known as Aztecs). These stories were originally represented in codices: pictographic documents where cultures from Mesoamerica used to write their history and other important aspects of their lives (Galarza, J.1997). Mexica plots are useful for our purpose because they have very well defined and simple syntactic and narrative structures, yet they have an immense potential for expression. In fact most of the themes of classical literature can be represented by Mexica plots: betrayal, sacrifice, courtly love, deceit, loyalty conflict, etc. The basic visual elements to assemble the animatic are provided by another system: Visual Narrator (VN) (Pérez y Pérez at al. 2012). This program illustrates story-plots from Mexica by producing sequences of still images composed of characters and scenes that literally represent the input plot by following a set of rules used in some pre-hispanic codices in a pictographic fashion. The rules specify how characters are presented according to their rank in society, activity, gender, and tension (emotional links represented by facial expression). They also tell how locations must be represented as well as action conventions. For instance, the rules describe how to represent a person that has a high social rank, who is talking to the people and who is angry. All characters used by e-Motion are built by VN, within the context of the process to produce a full motion graphics piece, the sequence produced by VN can be considered as a rough storyboard. e-Motion generates animatics that follow a set of conventions for the representation of characters and locations, but also depict the dynamics of the action and emotion found in the original plot. 2 Sequential drawings adapted from the script, depicted as concept drawings that illuminate and augment the script narrative. (Hart, J. 2008) This paper is divided into four sections besides this introduction. In the first section we describe the Imaginantes project and why we think the use of computer systems can improve it. In the next section we explain how the system works. Then we propose a set of criteria to evaluate the system. Finally we present some conclusions and the current state of our project. Building Motion Graphics for Imaginantes Motion graphics are already present in everyday life. They are used in a variety of media: TV identities, film titles and credits, DVD’s, videogames, smartphones interfaces, advertising displays and multiple media. The creation of motion graphics is considered a special skill, usually handled by artists or graphic designers focused on the combination of design and television broadcast or film (Frantz, M. 2003). The term is an abbreviation of “Motion Graphic Design”. Kook refers to it as the use of graphics, video footage and animation technology to create the illusion of motion or rotation, usually combined with audio (Kook, E. 2011). The Imaginante’s team consists of 8 to 10 people including: an executive producer, an art director, a design and animation coordinator, animators, illustrators, a musician or audio designer. The total time spent on the creation of a short ranges from 10 to 12 weeks. The team starts with an original script that provides the general structure of the story. In some cases, this script has some extra indications describing shots, special effects, sound, etc. Concept creation. The team collects all kinds of reference material related to the theme and author. It’s a collaborative and exploratory work. Pre-visualization. At this stage, the team develops two main tools: first, the storyborad, whose purpose is to show the key moments of the story in a sequence, suggest framing of the scenes and inform other specifics, like lighting, camera movements and special effects. It gives the entire pre-production team, a visual sequential breakdown of the main scenes in the narrative. The other tool is the animatic, which brings the storyboard alive with motion, visual effects and a visual style for the animation. It is very effective tool to pace the narrative and timing and later add music and dialog (Hart, J. 2008). Production. After the animatic is developed, illustrations are created, digitalized, and rendered; sound and music are also added to produce the final piece. In a process like the one described above, a system like eMotion, that suggests a variety of animatics with some camera-direction decisions based on the dramatic content the director wants to pursue, would be of great value for the production team. In the regular process there is a limProceedings of the Fourth International Conference on Computational Creativity 2013 185 ited feedback the team receives from just one animatic per motion graphics project. It would also open new communication channels between the team members by expanding the discussion to new options and save time and work resources. e-Motion Plots, in Mexica, are built by selecting characters and structure from a repository of previous plots, combining them in a way that makes sense, story-wise, and trying to preserve well-known, successful emotional tensions. Emotional tensions are collected during the process in an emotional-tension profile for the story. This can be viewed as a chart where overall emotion varies against time. Emotional tension preservation in Mexica is a key factor in the guidance towards the selection and combination of elements for a successful plot. A plot is a sequence of events in the order they occur in the story. It is the skeleton, the structure that tells the main events that occur in the story in a sequence of short action descriptions. Before the story is complete and ready for a final reader, it would have to be further developed to include all aspects that fulfill a creative piece of literary work. Yet, for our purposes it constitutes a good starting point because in the Imaginantes process the starting point is a plot from a text script (similar to a plot) with a few very structured actions or events in sequential order. An example story plot can be seen below. Emotional tensions are inserted between brackets as they occur (Lc = love conflict); (Lr = life at risk), (Hr = health at risk), (Ad = actor dead): Jaguar Knight was in Texcoco Lake Enemy was in Texcoco Lake Enemy got intensely jealous of Jaguar Knight (Lc) Enemy Attacked Jaguar Knight (Lr) Jaguar Knight fought Enemy Enemy wounded Jaguar Knight (Hr) Enemy ran away Enemy went back to Texcoco Lake Enemy did not cure Jaguar Knight (Lr) Farmer prepared to sacrifice enemy Enemy ran away Jaguar Knight died by injuries (Ad) Figure 1. A plot from Mexica and its tensions. In e-Motion, a story plot with its emotional profile is taken as input as well as a set of characters generated by VN. An example character from VN can be seen in (Figure 2). It depicts the 'enemy' character from the story being angry as an emotional response to the fight it held with jaguar knight (see plot above in Figure 1). Each line in the plot is an event and these, in turn, are incorporated into scenes by e-Motion. A scene has a set of performers. A performer is a character in action. That is, a character associated to an action to be performed. Characters in the animation include anything that appears on the screen and can be animated: humans, locations, emotional tokens, etc. They are all images and can be modified by 'moods'. Figure 2. Enemy in angry mood A character can have several moods depending on the representational variations available to it. A human can be looking right or left (there are only two dimensions, so far); he/she can be normal, angry or sad, etc. A location can have rain, sunshine, etc. Actions encode the movements of the characters on screen, they have a name and are a combination of the following basic animation operations: translation, scale and rotation. Performers can realize an action from the plot, like 'fight' or enact the manifestation of an emotional-tension like 'got intensely jealous of'. Emotions in animation may be expressed in different ways: character moods, textures flying as clouds across the scene, icons depicting specific feelings —similar to the ones presented in codices—, scene elements or characters appearing as text, etc. In the latter case, the font, size and color of the text are used to manifest different emotions too. We call these: emotional tokens. e-Motion builds scenes by following cinematic rules of composition: transition, character distribution, motion trajectories, framing and color. There are several options for each and the system builds the scenes of the animatic by selecting combinations of them that reflect the emotional profile of the original plot. Emotional tensions may be of different kinds: love, hate, danger, anger, etc. As a story progresses, each event may bring new emotional tensions into consideration. Some of them may reinforce others previously introduced or may counteract them. Each new tension introduced in a plot manifest itself in the composition of a scene by affecting, in a certain amount, the dramatic quality of the scene. For every character there is an emotional profile consisting of three parameters (Table 1): affect (the level of acceptance /rejection the character feels), health (a level of well-being) and excitement (a measurement of the degree of arousal of the character). Each occurrence of a tension in the original plot contributes, by a certain, predefined, amount to some of the emotional profile parameters just mentioned. They are ranked from -3 to 3. Hence, whenever a character is to be integrated into a scene, its emotional profile defines Proceedings of the Fourth International Conference on Computational Creativity 2013 186 how it appears in it: a character's affect, health and excitement values determine its performance parameters in the scene: mood, speed, and emotional tokens (Table 1). There is also a set of global rules that determine framing transition and trajectory for the characters and scenes. Framing refers to decisions based on how close to frame an action of the story and how far to pull it back so the audience can see where the action is taking place. eMotion choses its framing from a range of 4 types, based on camera angles of photography and film (McCloud, S. 2006): First plane, Middle-shot, Middle close-up and Extreme close-up. A story will always start with a first plane view (a), making a zoom in camera movement, followed by a middle-shot (b). As the story continues with the events carrying on, the characters change their affect and health levels. e-Motion will always look for the highest tension value to change the framing. E.g. (+2 or -2) or (+3 or -3) will change to frame (c) or (d) respectively. Every time we have a middle close-up the program sets back to a middle shot view until the levels of tension of the character are increased again to +3 or -3 values. The system will always end the story with a first plane view unless the overall level of excitement shows -3 or +3. In that case the system selects and extreme close-up and tilts the object in the frame. Trajectory refers to the path that a character follows to enter and exit a scene. Each character has an entrance to the scene and a position to move to in the plane. It can also continue it towards the edges of the plane. Emotional tokens have two main trajectory paths: clouds follow a curve; stains and pictograms stay in their initial position of appearance while varying their scale according to the character's level of excitement. Transition refers to the sequence of movements from one key scene of the story to another, thus establishing the flow of the story (McCloud, S. 1993). The example plot shown in Figure 1 produces the animatic presented in Figure 3 as a sequence of images that show some of its scenes. The third frame shows character Enemy in angry mood because he is jealous of Jaguar Knight. There is a green cloud traveling across the frame from left to right showing that feeling in the scene. The eagle and the cactus compose the name of “Texcoco Lake”, the place where the scene takes place. The angry mood is due to a low level of affect (-1) and excitement (- 1). The cloud is one of many possible manifestations of jealousy; this one in particular was selected randomly. The framing of that scene is solved in a middle shot angle. In the last scene Jaguar Knight is dead. The tension refers to Actor death, health (-3), excitement (0) and affect (0). There is an emotional token (red stain) which depicts the “being hurt”action and an up-scaling black cloud covers the body, dissolving into the final scene. Assesing Animatics To evaluate an animatic we have designed a questionnaire to measure its efficiency as a valuable tool and how new and surprising are the results to the team. (Boden, M. 1992). The questions were designed following interviews with actual members of the team where they set the parameters for effectiveness. The questionnaire is aimed at members of the Imaginantes team and will be rated higher according to their experience. Each questions offers 5 levels of agreement (1 means “totally disagree”, while 5 Figure 3. Six still frames extracted from an animatic as a means of illustration. Figure 3. Six still frames extracted from an animatic as a means of illustration. Dimension -3 -2 -1 0 1 2 3 affection hate contempt envy neutral sympathy affection love health death illness hurt neutral sane welfare happiness excitement horror dread cautious neutral surprise joy bliss Table 1. Dimensions of emotional profile and their discrete values. Proceedings of the Fourth International Conference on Computational Creativity 2013 187 means “totally agree”): 1. Does the animatic show a logical selection of key moments from the text script? 2. Does the animatic give you general information on how many characters/objects/locations need to be drawn. Does it help you visualize a particular graphic style? 3. Does the animatic give you a general direction on time, special effects, movements and transitions you need to consider for the animation? 4. Does the animatic show an appropriate selection of camera angles according to the dramatic content of the script? 5. Is it likely that anyone on your team would have come up with a similar solution? We end the questionnaire with a question that asks the participant to locate in a chart the balance of intensity/clarity and creativity of the animatic they have just seen (McCloud, S. 2006). We expect that e-Motion should ideally be ranked within the upper-right quadrant. (Figure 4). All team members have access to all work products, so they can relate them to the initial plot. Clarity Transmit the logical structure narrative of the text script Intensity Effects and techniques to attract an excite readers according to the dramatic content Very Creative Suggests relevant ideas, some that I have never thought before Not so creative Suggests a number of ideas but not so good Figure 4. Efficiency and creativity chart Conclusions and Future Work e-Motion is a system that contributes to the development of motion graphics by creating animatics. By doing so, it plays an important role in the creative process since it determines a great deal of the structure and action of the final motion graphics piece. We consider our system important because it allows us to experiment with computational creativity in a proven creative process in the real world. This setting is particularly appropriate, we find, to evaluate the system's performance since the human part of the team can do so, with well defined parameters. As far as the authors of this paper are aware, there is no other work involving computational creativity and animation. At the time of writing e-Motion is in β -test for its first version. We have run it with a few plots but the evaluation process, although it has already been designed, it still hasn't been applied. It will as soon as the system is ready. In the first stage of the project we use Mexica plots as a starting point. This allows us to use a standard for plots that also contain information about their emotional tensions. In the Imaginantes project, the starting point is a script derived from an art piece. In subsequent versions we will use the experience with Mexica plots to standardize scripts taken from other sources and provide them with emotional descriptions. We are also planning to develop other systems that take over other aspects of the process and study their effects. The system currently works with a set of tension values; these contribute cumulatively to character's emotional profiles, which determine how characters are animated. The general rules about framing, transition and trajectory follow basic cinematic standards but we would also like them to be determined by emotional parameters, an aspect that needs to be further investigated References Frantz, M. 2003. Changing Over Time: The Future of Motion Graphics. Thesis and Research work. In http://www.mattfrantz.com/thesisandresearch/motiongraph ics.html. Galarza, J. 1997. Los codices mexicanos. Arqueología Mexicana. 4:6-24. Imaginates 2006. In http://www.fundaciontelevisa.org/imaginantes-2/page/3/. Malamed, C. 2009. Visual Language for Designers: Principles for creating graphics that People Understand. Massachusetts: Rockport Publishers. 2252-3018. Maher, M.L. 2012. Computational and Collective Creativity: Who’s Being Creative? In Proceedings of International Conference of Computational Creativity 2012. Pérez y Pérez, R.; Sharples, M. 2001. MEXICA: a computer model of a cognitive account of creative writing. Journal of Experimental and Theoretical Artificial Intelligence. Volume 13, number 2, pp. 119-139. Pérez y Pérez, R.; Morales, N. and Rodríguez L. 2012. In Proceedings of International Conference of Computational Creativity 2012. Hart, J. 2008. The Art of the Storyboard. A Filmmaker’s introduction. Oxford, U.K: Focal Press. Elsevier 10:175. McCloud, S. 1993. Understanding Comics The Invisible Art. N.Y.: Harper Collins Publishers. 70-89 McCloud S; 2006. Making Comics: Storytelling Secrets of Comics, Manga and Graphic Novels. Harper Collins Publishers. 19-25, 336-2900. Proceedings of the Fourth International Conference on Computational Creativity 2013 188