Creative Machine Performance: Computational Creativity and Robotic Art
Petra Gemeinboeck
College of Fine Art
University of NSW
NSW 2021 Australia
petra@unsw.edu.au
Rob Saunders
Design Lab
University of Sydney
NSW 2006 Australia
rob.saunders@sydney.edu.au
Abstract
The invention of machine performers has a long tradition
as a method of philosophically probing the nature
of creativity. Robotic art practices in the 20th Century
have continued in this tradition, playfully engaging the
public in questions of autonomy and agency. In this
position paper, we explore the potential synergies between
robotic art practice and computational creativity
research through the development of robotic performances.
This interdisciplinary approach permits the
development of significantly new modes of interaction
for robotic artworks, and potentially opens up computational
models of creativity to rich social and cultural
environments through interaction with audiences. We
present our exploration of this potential with the development
of Zwischenraume (In-between Spaces), an ¨
artwork that embeds curious robots into the walls of a
gallery. The installation extends the traditional relationship
between the audience and artwork such that visitors
to the space become performers for the machine.
Introduction
This paper looks at potential synergies between the practice
of robotic art and the study of computational creativity.
Starting from the position that creativity and embodiment
are critically linked, we argue that robotic art provides a rich
experimental ground for applying models of creative agency
within a public forum. From the robotic art perspective, a
computational creativity approach expands the performative
capacity of a robotic artwork by enhancing its potential to
interact with its ‘Umwelt’ (Von Uexkull 1957). ¨
In the 18th century, the Industrial Age brought with
it a fascination with mechanical performers: Jacques de
Vaucanson’s Flute Player automaton and Baron Wolfgang
von Kempelen’s infamous chess playing Mechanical Turk
clearly demonstrate a desire to create apparently creative automata.
Through their work, both Vaucanson and von Kempelen
engaged the public in philosophical questions about
the nature of creativity, the possibilities of automation and,
crucially, perfection.
Moving from mechanical to robotic machine performers,
artists have deployed robotics to create apparently living and
behaving creatures for over 40 years. The two dominant
motivations for this creative practice have been to question
“our premises in conceiving, building, and employing these
electronic creatures” (Kac 2001), and to develop enhanced
forms of interactions between machine actors and humans
“via open, non-determined modes” (Reichle 2009).
The pioneering cybernetic work Senster by Edward Ihnatowicz,
for example, exhibited life-like movements and was
programmed to ‘shy away’ from loud noises. In contrast
to the aforementioned automata, Ihnatowicz did not aim to
conceal the Senster’s inner workings, and yet “the public’s
response was to treat it as if it were a wild animal” (Rieser
2002). Norman White’s Helpless Robot (1987–96) was a
public sculpture, which asked for help to be moved, and
when assisted, continued to make demands and increasingly
abused its helpers (Kac 1997). Petit Mal by Simon Penny
resembled a strange kind of bicycle and reacted to and pursued
gallery visitors. With this work Penny aimed to explore
the aesthetics of machines and their interactive behaviour in
real world settings; Petit Mal was, in Penny’s words, “an actor
in social space” (Penny 2000). Ken Rinaldo’s Autopoesis
consisted of 15 robotic sculptures and evolved collective
behavior based on their capability to sense each other’s and
the audience’s presence (Huhtamo 2004). The installation
Fish-Bird by Mari Velonaki comprised two robotic actors
in the form of wheelchairs whose movements and written
notes created a sense of persona. The relationship between
the robot characters and the audience evolved based on autonomous
movement, coordinated by a central controller,
and what appeared to be personal, “handwritten” messages,
printed by the robots (Rye et al. 2005).
Our fascination with producing artefacts that appear to be
creative has created a rich history for researchers of computational
creativity to draw upon. What we learn from these
interdisciplinary artistic approaches is that, as performers,
the artificial agents are embodied and situated in ways that
can be socially accessed, shared and experienced by audiences.
Likewise, embodied artificial agents gain access to
shared social spaces with other creative agents, e.g., audience
members.
The ability of robotic performers to interact with the audience
not only relies on the robot’s behaviours and responsiveness
but also the embodiment and enactment of these
behaviours. It can be argued that the performer is most successful
if both embodiment and enactment reflect its perception
of the world, that is, if it is capable of expressing
and communicating its disposition. Looking at robotic artProceedings
of the Fourth International Conference on Computational Creativity 2013 215
works that explore notions of autonomy and artificial creativity
may thus offer starting points for thinking about social
settings that involve humans interacting and collaborating
with creative agents.
Our exploration revolves around the authors’ collaboration
to develop the robotic artwork Zwischenraume (In- ¨
between Spaces), a machine-augmented environment, for
which we developed a practice embedding embodied curious
agents into the walls of a gallery, turning them into a
playground for open-ended exploration and transformation.
Zwischenraume ¨
The installation Zwischenraume embeds autonomous robots ¨
into the architectural fabric of a gallery. The machine agents
are encapsulated in the wall, sandwiched between the existing
wall and a temporary wall that resembles it. At the
beginning of an exhibition, the gallery space appears empty,
presenting an apparently untouched familiar space. From
the start, however, the robots’ movements and persistent
knockings suggest comprehensive machinery at work inside
the wall. Over the course of the exhibition, the wall increasingly
breaks open, and configurations of cracks and hole patterns
mark the robots’ ongoing sculpting activity (Figure 1).
Figure 1: Zwischenraume: curious robots transform our fa- ¨
miliar environment.
The work uses robotics as a medium for intervention: it is
not the spectacle of the robots that we are interested in, but
rather the spectacle of the transformation of their environment.
The starting point for this interdisciplinary collaboration
was our common interest in the open-ended potential of
creative machines to autonomously act within the human environment.
From the computational creativity researcher’s
point of view, the embodied nature of the agents allowed for
situating and studying the creative process within a complex
material context. For the artist, this collaboration opened up
the affective potential to materially intervene into our familiar
environment, bringing about a strange force, seemingly
with an agenda and beyond our control.
Each machine agent is equipped with a motorised hammer,
chisel or punch, and a camera to interact and network
with the other machines by re-sculpting its environment
(Figure 2). The embodied agents are programmed to
be curious, and as such intrinsically motivated to explore
the environment. Once they have created large openings in
the wall the robots may study the audience members as part
of their environment. In the first version of this work, the
robots used their hammer to both punch holes and for communicating
amongst the collective. In a later version, we
experimented with a more formal sculptural approach that
used heuristic compositions of graffiti glyphs to perforate
walls. Using the more stealthy movements of a chisel, the
work responded to the specific urban setting of the gallery
by adapting graffiti that covered the exterior of the building
to become an inscription, pierced into the pristine interior
walls of the gallery space (Figure 3). The final version of
Zwischenraume used a punch to combine the force of the ¨
hammer and the precision of the chisel.
Figure 2: Robot gantries are attached to walls.
Similar to Jean Tinguely’s kinetic sculptures (Hulten´
1975), Zwischenraume’s performance and what it produces ¨
may easily evoke a sense of dysfunctionality. As the machines’
adaptive capability is driven by seemingly nonrational
intentions rather than optimisation, the work, in
some sense, subverts standard objectives for machine intelligence
and notions of machine agency. Rather, it opens up
the potential for imagining a machine that is ‘free’, a machine
that is creative, see (Hulten 1987). ´
Machine Creativity
This section focuses on the development of the first version
of Zwischenraume as depicted in Figures 1 and 2. Each ¨
robotic unit consisted of a carriage, mounted on a vertical
gantry, equipped with a camera mounted on an articulated
arm, a motorised hammer, and a contact microphone. The
control system for the robots combined machine vision to
detect features from the camera with audio processing to detect
the knocking of other robots and computational models
of intrinsic motivation based on unsupervised and reinforceProceedings
of the Fourth International Conference on Computational Creativity 2013 216
Figure 3: Inscription of adapted graffiti glyphs.
ment machine learning to produce an adaptive, autonomous
and self-directed agency.
The robot’s vision system was developed to construct
multiple models of the scene in front of the camera; using
colour histograms to differentiate contexts, blob detection
to detect individual shapes, and frame differencing to detect
motion. Motion detection was only used to direct the
attention of the vision system towards areas of possible interest
within the field of view. Face detection is also used
to recognise the presence of people to direct the attention
of the robots towards visitors. While limited, these perceptual
abilities provide sufficient richness for the learning algorithms
to build models of the environment to determine
what is different enough to be interesting.
Movements, shapes, sounds and colours are processed,
learned and memorised, allowing each robotic agent to develop
expectations of events in their surrounds. The machine
learning techniques used in Zwischenraume combine un- ¨
supervised and reinforcement learning techniques (Russell
and Norvig 2003): a self-organizing map (Kohonen 1984)
is used to determine the similarity between images captured
by the camera; Q-learning (Watkins 1989) is used to allow
the robots to discover strategies for moving about the wall,
using the hammer and positioning the camera.
Separate models are constructed for colours and shapes
in images. To determine the novelty of a context, sparse histograms
are constructed from captured images based on only
32 colour bins with a high threshold, so only the most significant
colours are represented and compared using a selforganising
map. Blob detection in low-resolution (32x32
pixel) images, relative to a typical model image of the wall,
is used to discover novel shapes and encoded in a selforganising
map as a binary vector. In both cases, the difference
between known prototypes in the self-organising map
provide a measure of novelty (Saunders 2001).
Reinforcement learning is used to learn the consequences
of movements within the visual field of the camera. Error
in prediction between learned models of consequences and
observed results is used as a measure of surprise. As a result
system that is able to learn a small repertoire of skills and appreciate
the novelty of their results, e.g., knocking on wood
does not produce dents. This ability is limited to immediate
consequences of actions and does not current extend to
sequences of actions.
The goal of the learning system is to maximise an internally
generated reward for capturing ‘interesting’ images
and to develop a policy for generating rewards through action.
Interest is calculated based on a computational model
that captures intuitive notions of novelty and surprise (Saunders
2001): ‘novelty’ is defined as a difference between an
image and all previous images taken by the robot, e.g., the
discovery of significant new colours or shapes; and, ‘surprise’
is defined as the unexpectedness of an image within a
known situation, e.g., relative to a learned landmark or after
having taken an action within an expected outcome (Berlyne
1960). Learning plays a critical role in both the assessment
of novelty and surprise. In novelty, the robots have to learn
suitably general prototypes for the different types of images
that they encounter. In surprise, the ‘situation’ against which
images are judged includes a learned model of the consequences
of actions (Clancey 1997).
Consequently, intrinsic motivation to learn directs both
the robot’s gaze and its actions, resulting in a feedback
process that increases the complexity of the environment
– through the robot’s knocking – relative to the perceptual
abilities of the agent. Sequences of knocking actions are
developed, such that the robots develop a repertoire of actions
that produce significant perceived changes in terms
of colour, shapes and motion. In this way, the robots explore
their creative potential in re-sculpting their environment.
Figure 4 presents a collage of images taken by a single
robot when it discovered something ‘interesting’, illustrating
how the evaluation of ‘interesting’ evolved for this robot; it
shows how the agent’s interest is affected by: (a) positioning
of the camera, e.g., the discovery of lettering on the plasterboard
wall; (b) use of the hammer, e.g., the production of
dents and holes; and, (c) interaction of visitors.
Figure 4: Robot captures, showing the evolution of interesting
changes in the environment.
Proceedings of the Fourth International Conference on Computational Creativity 2013 217
Discussion
The robots’ creative process turns the wall into a playful
environment for learning, similar to a sandpit; while from
the audiences’ point of view, the wall is turned into a performance
stage. This opens up a scenario of encounter for
studying the potential of computational creativity and the
role of embodiment. Following Pickering (2005), we argue
that creativity cannot be properly understood, or modelled,
without an account of how it emerges from the encounter
between the world and intrinsically active, exploratory and
productively playful agents.
Embodiment and Creativity
The agents’ embodiment provides opportunities to expand
their behavioural range by taking advantage of properties of
the physical environment that would be difficult or impossible
to simulate computationally (Brooks 1990). In Zwischenraume
the machines’ creative agency is not predeter- ¨
mined but evolves based on what happens in the environment
they examine and manipulate. As the agents’ embodiment
evolves based on its interaction with the environment,
the robots’ creative agency affects processes out of which it
itself is emergent.
This resonates with Barad’s argument that ‘agency is a
matter of intra-acting: it is an enactment, not something that
someone or something has’ (Barad 2007). It also evokes
Maturana and Varela’s notion of enaction, where the act
of bringing about a world occurs through the ‘structural
coupling’ between the dynamical environment and the autonomous
agents (Maturana and Varela 1987). While the
machines perturb and eventually threaten the wall’s structural
integrity, they adapt to their changing environment, the
destruction of the wall and how it changes their perception
of the world outside.
The connection to creativity is two-fold: Firstly, the
robots’ intrinsic motivation to explore, discover and constantly
produce novel changes to their environment demonstrates
a simplistic level of a creative process itself, akin to
the act of doodling, where the motivation is a reflective exploration
of possibilities rather than purposeful communication
with others. Secondly, the audiences interpret the machines’
interactions based on their own context, producing
a number of possible meaningful relations and associations.
The agents’ embodiment and situatedness becomes a portal
for entering the human world, creating meaning. The
agents’ enacted perception also provides a window on the
agents’ viewpoint, thus possibly changing the perspective of
the audience.
Furthermore, an enactive approach (Barad 2003; Clark
1998; Thompson 2005) opens up alternative ways of thinking
about creative human-machine collaborations. It makes
possible a re-thinking of human-machine creativity beyond
the polarisation of human and non-human, one that promotes
shared or distributed agency within the creative act.
Audience Participation
Autonomous, creative machine performances challenge the
most common interaction paradigm of primarily reacting to
what is sensed, often according to a pre-mapped narrative.
Zwischenraume’s curious agents proactively seek interac- ¨
tion, rather than purely responding to changes in the surrounds.
Once the robots have opened up the wall, the appearance
and behaviours of audience members are perceived
by the system as changes in their environment and become
an integral part of the agents’ intrinsic motivation system.
The agents’ behaviours adapt based on their perception
and evaluation of their environment, including the audience,
as either interesting or boring. A curious machine performer
whose behaviors are motivated by what it perceives and expects
can be thought of as an audience to the audiences performance.
Thus, in Zwischenraume it is not only the robots ¨
that perform, but also the audience that provokes, entertains
and rewards the machines’ curiosity. This notion of audience
participation expands common interaction paradigms
in interactive art and media environments (Paul 2003). The
robots don’t only respond or adapt to the audience’s presence
and behaviours, but also have the capacity to perceive
the audience with a curious disposition.
By turning around the traditional relationship between audiences
and machinic performers, the use of curious robotic
performers permits a re-examination of the machine spectacle.
Lazardig (2008) argues that spectacle, as “a performance
aimed at an audience,” was central to the conception
of the machine in the 17th century as a means of projecting
a perception of utility; allowing the machine to become “an
object of admiration and therefore guaranteed to ‘function”’.
Kinetic sculptures and robotic artworks exploit and promote
the power of the spectacle in their relationship with the audience.
This is also the case in Zwischenraume however, it is ¨
not only the machines that are the spectacle for the audience
but also the audience that becomes an ‘object of curiosity’
for the machines (Figure 5). Thus the relationship with a
curious robot extends the notion of the spectacle, and, in a
way, brings it full circle.
Figure 5: Gallery visitor captured by one of the robots’ cameras
as he performs for the robotic wall.
Proceedings of the Fourth International Conference on Computational Creativity 2013 218
Concluding Remarks
A significant aspect of Zwischenraume’s specific embodi- ¨
ment is that it embeds the creative agents in our familiar (human)
environment. This allowed us to direct both our, and
the audience’s, attention to the autonomous process and creative
agency, rather than the spectacle of the machine. The
integration of computational models of creativity into this
artwork extended the range of open-ended, non-determined
modes of interaction with the existing environment, as well
as between the artwork and the audience.
We argue that it is both, the embodied nature of the agents
and their autonomous creative capacity that allows for novel
meaningful interactions and relationships between the artwork
and the audience. The importance of embodiment for
computational creativity can also be seen in the improvising
robotic marimba player Shimon, which uses a physical gesture
framework to enhance synchronised musical improvisation
between human and nonhuman musicians (Hoffmann
and Weinberg 2011). The robot player’s movements not
only produce sounds but also play a significant role in performing
visually and communicatively with the other (human)
band members as well as the audience.
Embodying creative agents and embedding them in our
everyday or public environment is often messier and more
ambiguous than purely computational simulation. What we
gain, however, is not only a new shared embodied space for
audience experience but also a new experimentation space
for shared (human and non-human) creativity.
Acknowledgements
This research has been supported by an Australia Research
Council Grant, a New Work Grant from the Austrian Federal
Ministry for Education, Arts and Culture, and a Faculty
Research Grant from COFA (University of NSW).
<references_biblio/>
References
Barad, K. 2003. Posthumanist performativity: Toward an
understanding of how matter comes to matter. Signs: Journal
of Women in Culture and Society 23(1):801–831.
Barad, K. 2007. Meeting the Universe Halfway: Quantum
Physics and the Entanglement of Matter and Meaning.
Durham, NC: Duke University Press.
Berlyne, D. E. 1960. Conflict, Arousal, and Curiosity. New
York, NY: McGraw Hill.
Brooks, R. 1990. Elephants don’t play chess. Robotics and
Autonomous Systems 6:3–15.
Clancey, W. J. 1997. Situated Cognition: On Human Knowledge
and Computer Representations. Cambridge, England:
Cambridge University Press.
Clark, A. 1998. Where brain, body and world collide.
Daedalus: Journal of the American Academy of Arts and
Sciences 127(2):257—280.
Hoffmann, G., and Weinberg, G. 2011. Interactive improvisation
with a robotic marimba player. In Autonomous Robots
31, 133–153. Springer.
Huhtamo, E. 2004. Trouble at the interface, or the identity
crisis of interactive art. Framework: The Finnish Art Review
(2):38–41.
Hulten, P. 1975. ´ Tinguely. ‘Meta’ ´ . London, UK: Thames &
Hudson.
Hulten, P. 1987. ´ Jean Tinguely. A Magic Stronger than
Death. New York, NY: Abbeville Press.
Kac, E. 1997. Foundation and development of robotic art.
Art Journal 56(3):60–67.
Kac, E. 2001. Towards a chronology of robotic art. Convergence:
The Journal of Research into New Media Technologies
7(1).
Kohonen, T. 1984. Self-Organization and Associative Memory.
Berlin: Springer.
Lazardig, J. 2008. The machine as spectacle: Function
and admiration in seventeenth-century perspectives on machines.
In de Gryter, W., ed., Instruments in art and science:
on the architectonics of cultural boundaries in the 17th century.
152–175.
Maturana, H., and Varela, F. 1987. The Tree of Knowledge:
The biological roots of human understanding. Boston, MA:
Shambhala Publications.
Paul, C. 2003. Digital Art. London, UK: Thames & Hudson.
Penny, S. 2000. Agents as artworks and agent design as
artistic practice. In Dautenhahn, K., ed., Human Cognition
and Social Agent Technology. John Benjamins Publishing
Co. 395–414.
Pickering, J. 2005. Embodiment, constraint and the creative
use of technology. In Freedom and Constraint in the
Creative Process in Digital Fine Art.
Reichle, I. 2009. Art in the Age of Technoscience: Genetic
Engineering, Robotics, and Artificial Life in Contemporary
Art. Wien: Springer.
Rieser, M. 2002. The art of interactivity: from gallery to
street. In Mealing, S., ed., Computers and Art. Bristol, UK:
Intellect. 81–96.
Russell, S. J., and Norvig, P. 2003. Artificial Intelligence: A
Modern Approach. New Jersey, NY: Prentice Hall.
Rye, D.; Velonaki, M.; Williams, S.; and Scheding, S. 2005.
Fish-bird: Human-robot interaction in a contemporary arts
setting. In Proceedings of the 2005 Australasian Conference
on Robotics and Automation.
Saunders, R. 2001. Curious Design Agents and Artificial
Creativity. Ph.d. thesis, Faculty of Architecture, The University
of Sydney.
Thompson, E. 2005. Sensorimotor subjectivity and the enactive
approach to experience. Phenomenology and the Cognitive
Sciences 4(4):407–427.
Von Uexkull, J. 1957. A stroll through the worlds of animals ¨
and men: A picture book of invisible worlds. In Schiller,
C., ed., Instinctive Behavior: The Development of a Modern
Concept. New York, NY: Int’l Universities Press. 5–80.
Watkins, C. 1989. Learning from Delayed Rewards. Phd
thesis, Cambridge University, Cambridge, England.
Proceedings of the Fourth International Conference on Computational Creativity 2013 219