The application underlying this case study is at the core of an award-winning concept of a distributed multi-player performance called Music for Gongs and Satellites. In short, the vision aims at turning commercially available mobile phones into devices for coordinating a group of performers who are acting as a distributed mass. While the application itself may not be of much practical use to most people, it serves perfectly as archetype of an Android app that makes decisive use of real-time features such as system clock and GPS sensor modern portable devices are equipped with. In this paper, I shall explain how to design, organize and implement such an app in Java on the basis of the Android Studio IDE.
The Project
“Music for Gongs and Satellites” is a participatory work for a random public audience that can be performed in any given public space. The basic idea behind the work is to create a situation in which participants can “run wild” within a system while undertaking performative actions but remain coordinated by some higher power regardless of where or how they “run” within the system. Each participant is given a paired set of tuned gongs and asked to download the needed app to his or her mobile phone. Participants are readied in about 5 minutes and performing is accomplished through a process of learning-by-doing. Performances of the work have lasted several hours, but participants can play for a few minutes or a few hours at their will.
The Application
The application uses satellites and the mobile phone´s system clock to control an intelligent algorithm that in turn coordinates and instructs the participants via optical marks, based on where they stand, how, when and where they move, as well as how quickly and often they do so. Using a tool based on satellites, the application makes it possible to perform the work in any open space without constraining the dimensions of the area in which the work takes place. In principle, the group of participants could be distributed around the globe and still perform as a whole to create ever changing patterns of beautiful gong sounds spread out across an open space, paired with choreographic movements and highlighted by visual impact of the performers.
A simple version of the work will be discussed in this paper in more technical detail. Our sample performance takes place in a circular playground area that is divided up into three concentric “zones” around a common center point called “ground zero”. Depending on the zone and the location within the zone that a performer is standing at the time, the controlling app changes the timing of the signaling mark by adding a slight amount of delay or by skipping it entirely.
Design and Implementation of the Application in Java on Top of Android Studio
The development of the GongPong.SAT app in Java on top of the Android Studio IDE platform provided for some specific challenges. Among these are the need to operate in real-time in a precise rhythm and in total synchronicity with all companion players, the need to know the exact location of a player at any time, and the desire to give useful optical feedback.
Receiving and processing GPS data from satellites is at the core of this app. With the goal of providing this in a simple as possible application with a minimum of code, we decided in favor of using Google´s Fused Location Provider API, where “fused” refers to an implicit handling of the source of the location data from GPS satellites or earthly network, whichever is available and more appropriate at the place at which the location data is needed.
Moving forward, we will explain our use of the API in detail: First we observe that any location service code running on top of Android’s operating system must be guarded by a statement ensuring compatibility with the application’s permissions, as they are specified in the application’s manifest file AndroidManifest.xml:
Basically, starting the Fused Location Provider service consists of calling two constituents of the API: a) Getting an initial location fix and b) Launching a task listening for location updates. The following two sections will give the details on both calls.
Get an Initial Location Fix
The following code for getting an initial location fix reveals the asynchronicity of the model: Two listeners are supposed to call (back) a method in case of success and of failure respectively:
Starting a Listening Task
In the case of having received a valid location fix by the onSuccess method, a listening-and-updating task is launched with the following code:
Obviously, the decisive part of this piece of code for task-launching is the installation of the function LocationCallback, which periodically calls (back) the onLocationResult method with a list of newly computed locations, governed by the priority and frequency specifications in the LocationRequest object req. The callback itself first causes a conversion of the list of newly received locations into an array and subsequently sends a call to the UpdateWindow function whose name refers to a sliding window of distances between the performer´s position and the initial “ground zero” position, computed from location data received via recent callbacks:
In technical terms, the sliding window serves as a basic data structure connecting the two major tasks of this application, a) delivering (“producing”) new location data and b) their processing (“consuming”) that we shall discuss in the following sections.
However, we should not lose sight of the bigger picture and the final purpose of our app: Turning a mobile phone device into a tool for performing a work for gongs and satellites, which is largely a user interface topic. After some initial experiments, we decided in favor of a simple, but efficient, graphical user interface, which is created by switching a circular spot through different sizes (from small to large) and various colors (from yellow to orange and red) at predefined time intervals. The changes in color and size alert the performers when it is time to play i.e. to strike the gong. As the following helper Java class called Signal shows, the user-interface is controlled by another runnable (called Notify) whose task is it to force a redraw-event for the spot by invalidating its current representation:
The formal picture of the user-interface is revealed by the app’s activity_main.xml file: a linear layout specifying three elements in vertical order: a text field (for logging), a button (for restarting the app interactively) and the circular signal just explained (for alerting):
It is now time to come back to the consumer task that defines the operating rhythm of the app by periodically processing the data provided by the sliding window and implementing the actual performance logic. This task is implemented by a scheduler (of type Handler) posting a runnable piece of code called Check that is reposting itself to be run again after each second, according to the device´s clock.
Here is the framework code:
The performance logic still to be filled into the body of the run method roughly comprises two functionalities: a) computing the current distance from ground zero and b) deciding about possible actions to be performed depending on the time and the performer’s current position within the playground.
Fig. 1. A diagram showing the design for the GongPong.SAT app.
Computing the Current Distance from Ground Zero
Early experiments showed that computing distances from individual location fixes unavoidably leads to distorting jumps caused by noisy GPS data. Consequently, we added a simple algorithm for computing a smoothed-out approximation of the current distance from ground zero. At this point our sliding window comes in handy as it supports, for example, a solution based on dropping a few outlier values (in comparison with the previous smoothed-out distance value) and averaging across the remaining recent distance values:
Deciding about the Next Action to be Performed
Instructing players via graphical feedback about the next action to be performed is at the core of the performance logic. As can be seen from the piece of code below, the decision if or not to alert a player (via posting sig.Notify after a slight time shift depending on the player´s relative position within the playground) is based on the player´s zone (circle of radius r0, r1 or r around ground zero) and the multiplicity of ticks (seconds since the start of the performance):
The one final challenge that remains is provided by the request of precisely synchronizing the devices of a (potentially large) number of GongPong.SAT participants. As it is demonstrated by the following code excerpt, we have overcome this challenge quite elegantly by making use of the universal GPS time when launching the checker task of each device at the next full satellite second:
In concluding our presentation, we mention again the built-in option of interactively restarting a performance, which allows the app to be recalibrated at ground zero. This is achieved simply by pressing the restart button of our user-interface:
Summary
By taking a media art work as a case study, we demonstrate the feasibility of developing custom-tailored software for off-the-shelf commercial mobile phones, including the use of their high-precision sensors like GPS module and system clock, and for the use as a real-time controller. From a software-engineering point of view the multi-threaded modeling approach with the benefit of elegantly separating different concerns is particularly noteworthy. All this is a highly promising result, in particular as it is by no means limited just to creating applications for the arts.
Acknowledgment
I would like to thank Art Clay for inspiring me to dive into this adventure at all and my brother Ernst for numerous extremely valuable comments and hints regarding the use of the Android Studio IDE. Further thanks go to the Swiss Arts Council for providing the needed funding in support of the work and to the PHONOS Foundation, Barcelona for hosting the premiere of the work during their Fall Festival.
– Jürg Gutknecht, Swiss informatics Society