Introduction

Ninety percent of all digital data was created in the past few years, and the global datasphere is predicted to grow from 44 zettabytes in 2020 to 175 zettabytes by 2025. Reinsel et al. [1] observe that “Mankind is on a quest to digitize the world. The data-driven world will be always on, always tracking, always monitoring, always listening, and always watching.” Data is sought by corporations for marketing and predictive analytics and by governments for surveillance. By tracking, monitoring, listening and watching individuals, marketers can target them with ongoing streams of media and advertisements [2]. The possibility that this process, the creation and manipulation of docile consumers, could lead to the impoverishment of human experience was anticipated by the scholars of the Frankfurt School. In the Dialectic of Enlightenment, published in 1947, [3] stated that “Reason itself has become the mere instrument of the all-inclusive economic apparatus... the primordial fear of losing one’s own name has been realized.”

Mood of the Planet is an art installation that deconstructs and reassembles this vision of humanity and Big Data. It is an interactive physical-digital sculpture that has as its center-piece a large “arch” or “doorway” that emits colored light and sound as a form of visualization and sonification of the changing, live emotions expressed by people all around the Earth. Streams of data are not manipulated to create hidden models for marketing and surveillance. Instead from moment to moment everyone’s feelings are combined and transformed into colors and music, which envelope us as we enter a portal. For the Frankfurt School, the Dialectic of Enlightenment meant the recovery of our humanity, a spiritual journey from the everyday preoccupation with the self to our universality, our place in the world. Mood of the Planet creates that experience.

Using sentic computing [4] to analyze Twitter data, the sculpture incorporates an embedded real-time, networked multimedia system that visualizes microblogging posts expressing emotional content. The detection of such content, in particular, is implemented by using the 200,000 concepts of SenticNet [5], a commonsense knowledge base for sentiment analysis, as keywords for the Twitter search. These concepts are linked to emotions using the Hourglass of Emotions [6], an emotion categorization model featuring 24 basic emotions associated with 24 different colors. Through this model, collected concepts are analyzed and mapped into digital animation and music associated with the respective emotions. Variations in color, shape, texture, pattern, speed of animation as well as musical elements of pitch, harmony, and rhythm are correlated with each emotion and displayed on the sculpture.

The remainder of this paper is organized as follows: “Related Work” discusses related work in the context of emotion visualization in the arts; “Sentic Computing: Pairing Social Media Data with Emotions” presents the framework employed for pairing social media data with emotions; “Hourglass Model: Pairing Emotions with Colors and Intensities” illustrates the model used for pairing emotions with colors and intensities; “The Arch: Pairing Colors with Tangible Experiences” describes in detail the structure designed for pairing colors with tangible experiences; “Discussion and Future Work” proposes a discussion and ideas for future work; finally, “Conclusion” provides concluding remarks.

Related Work

One of the major uses of Big Data is market research, for businesses that want to know what people are feeling as it influences their buying and consumption [7]. Consumers are motivated by desire, fear, happiness, and other feelings, which are fleeting emotional states [8]. In today’s global markets, and global communication, these emotional states transcend traditional physical or political borders. A population’s anxiety or happiness affects markets dramatically and, hence, is watched and reacted to daily by investors [9]. In this way, business intelligence becomes entangled with surveillance and commercial exploitation.

We envisioned another possibility, which became the genesis of Mood of the Planet. We asked ourselves whether we can use Big Data to discover the shifting emotional states of the world population which, like the weather, changes constantly. Unlike rain, shine, snow and ice, however, we do not have direct evidence for emotional states. Instead they are expressed in words, images, gestures, sound, music, and other aesthetic forms. Our goal is to use Big Data and networks to detect the emotions and feelings of people around the world, creating a compelling alternate vision to market research, and providing an aesthetic experience that re-positions Big Data as a form of personal and public art. The goal is to provoke awareness of the datasphere that surrounds us and the Earth, in a form that is human in scale, meaningful and engaging. It further seeks to bring abstract data to tangible experience by translating it to a physical form, an architectural structure filled with light and music that invites contemplation, reflection, and meditation.

Earlier works in interactive architectural installation that explore these themes include Morocco Memory [10], Sanctuary [11], Sensity [12], Res Sapiens [13], Illuminations [14], Emotions in Space [15], and the dome work Vishwaroop [16]. Sanctuary (Fig. 1, left) is an interactive architectural installation based on cross cultural interpretations of “safe haven,” focusing on natural and spiritual sanctuaries, and the common need for safety by all creatures of the world. Emotions in Space (Fig. 1, right) seeks to re-imagine ways in which global real-time datasets can provide a unique perspective on the understanding of world motion through emotional currents emanating from 3200 different cities. In this interactive work, clouds of the world’s emotions rise from the cities, forming a spiral nebula around the Earth.

Fig. 1
figure 1

Sanctuary (left) and Emotions in Space (right)

Full size image

Sensity (Fig. 2, left) uses environmental sensors measuring, light, noise, sound, humidity, and temperature that are scattered all over the museum and in the surrounding city to make visualizations and sonifications in the gallery and demonstrate the current emotional state of the city. Res Sapiens (Fig. 2, right) is an installation that uses multiple mechanical desklamps connected to the Internet that react to positive and negative tweets by moving and by turning on and off.

Fig. 2
figure 2

Sensity (left) and Res Sapiens (right)

Full size image

Illuminations (Fig. 3, left) was a 30-m-long glowing, illuminated folding screen that combined the Asian panorama with computer animation rear-projection onto screens. Sensors were placed under 3 Tibetan Singing Bowls that detected viewers playing, and also in cushions that were activated when people sat on them. There were also live plants with sensors that measured their CO2 output. All were interfaced to the real-time system and altered the animation and music. Vishwaroop (Fig. 3, right) means the appearance of God or Brahma in forms that incorporate the creation of worlds and the Universe within them. This piece is both a real-time dome installation and a fixed animation with music by sitar virtuoso Kartik Seshadri.

Fig. 3
figure 3

Illuminations (left) and Vishwaroop (right)

Full size image

Besides interactive architectural installations, the literature of emotion visualization also include online works such as AffectiveSpace [17], a navigable multidimensional vector space where words and multiword expressions, displayed in different color codes, are slowly rotating around different emotion axes like clusters of stars in a galaxy (Fig. 4, left), Emotion Forecast [18], an emotional prediction service based on statistical foundations that anticipates the emotional state of the planet (Fig. 4, right), and We Feel Fine [19], an interactive website that searches the Internet every 10 m for expressions of human emotion on blogs and then displays the results in several visually rich dynamic representations.

Fig. 4
figure 4

AffectiveSpace (left) and Emotion Forecast (right)

Full size image

Mood of the Planet stems from some of these works with the goal of giving the audience a more accurate, personal and tangible experience about the data-emotions dichotomy.

Sentic Computing: Pairing Social Media Data with Emotions

With the recent development of deep learning, research in artificial intelligence (AI) has gained new vigor and prominence. Machine learning, however, suffers from three big issues, namely:

  1. 1.

    Dependency: it requires (a lot of) training data and is domain-dependent;

  2. 2.

    Consistency: different training or tweaking leads to different results;

  3. 3.

    Transparency: the reasoning process is uninterpretable (black-box algorithms).

Sentic computing addresses such issues in the context of natural language processing (NLP) through a multi-disciplinary approach that aims to bridge the gap between statistical NLP and many other disciplines that are necessary for understanding human language, such as linguistics, commonsense reasoning, and affective computing. Sentic computing, whose term derives from the Latin sensus (as in commonsense) and sentire (root of words such as sentiment and sentience), enables the analysis of text not only at document, page or paragraph level, but also at sentence, clause, and concept level.

This is possible thanks to an approach to NLP that is both top-down and bottom-up: top-down for the fact that sentic computing leverages symbolic models such as semantic networks and conceptual dependency representations to encode meaning; bottom-up because it uses sub-symbolic methods such as deep neural networks and multiple kernel learning to infer syntactic patterns from data. Coupling symbolic and sub-symbolic AI is key for stepping forward in the path from NLP to natural language understanding. Relying solely on machine learning, in fact, is simply useful to make a “good guess” based on past experience, because sub-symbolic methods only encode correlation and their decision-making process is merely probabilistic. Natural language understanding, however, requires much more than that. To use Noam Chomsky’s words, “you do not get discoveries in the sciences by taking huge amounts of data, throwing them into a computer and doing statistical analysis of them: that’s not the way you understand things, you have to have theoretical insights.”

In this paper, we use sentic computing to deconstruct natural language data into numerical data that can be later processed and visualized. In particular, we use the 200,000 concepts of SenticNet (Fig. 5) to mine emotion-bearing microblogging posts in real time via the Twitter API. SenticNet encodes the denotative and connotative information commonly associated with real-world objects, actions, events, and people. It steps away from blindly using keywords and word co-occurrence counts, and instead relies on the implicit meaning associated with commonsense concepts. Superior to purely syntactic techniques, SenticNet can detect subtly expressed sentiments by enabling the analysis of multiword expressions that do not explicitly convey emotion, but are instead related to concepts that do so. After natural language concepts are collected and extracted, we then use the Hourglass of Emotions to connect them to emotion labels and intensities.

Fig. 5
figure 5

SenticNet graph

Full size image

Hourglass Model: Pairing Emotions with Colors and Intensities

The Hourglass of Emotions is a brain-inspired and psychologically motivated emotion categorization model that goes beyond mere categorical and dimensional approaches (Fig. 6). Such model represents affective states both through labels and through four independent but concomitant affective dimensions, namely Introspection (the joy-versus-sadness dimension), Temper (the calmness-versus-anger dimension), Attitude (the pleasantness-versus-disgust dimension), and Sensitivity (the eagerness-versus-fear dimension).

Fig. 6
figure 6

The Hourglass of Emotions

Full size image

Each affective dimension is characterized by six levels of activation measuring the strength of an emotion. Such levels are also labeled as a set of 24 primary emotions (Fig. 7) in a way that allows the model to specify the affective information associated with text both in a dimensional and in a discrete form. For each dimension, thus, each level of activation represents the intensity thresholds of the perceived emotion both in positive terms (e.g., ecstasy> joy> contentment) and negative ones (melancholy < sadness< grief ). The dimensional form, instead, is a four-dimensional float vector, which can potentially describe the full range of emotional experiences that are rooted in any of us. In the model, the vertical dimension represents the intensity of the different affective dimensions, while the radial dimension models the activation of different emotional configurations, resembling Minsky’s k-lines [20].

Fig. 7
figure 7

Emotion classification with five sample emotion words for each category

Full size image

The model, in fact, is based on the idea that the mind is made of different independent resources and that emotional states result from turning some set of these resources on and turning another set of them off [21]. Each such selection changes how we think by changing our brain’s activities: the state of anger, for example, appears to select a set of resources that help us react with more speed and strength while also suppressing some other resources that usually make us act prudently. Evidence of this theory is also given by several fMRI experiments showing that there is a distinct pattern of brain activity that occurs when people are experiencing different emotions.

The model follows the pattern used in color theory and research [22] in order to obtain judgements about combinations, i.e., the emotions that result when two or more fundamental emotions are combined, in the same way as primary colors can be combined to form new ones (Fig. 8). Such combinations can be bidimensional, e.g., love (joy+pleasantness); tridimensional, e.g., bittersweetness (sadness+anger+pleasantness); and four-dimensional, e.g., jealousy (anger+fear+sadness+disgust).

Fig. 8
figure 8

Examples of compound emotions

Full size image

In this paper, we use the Hourglass model to connect the emotion-bearing concepts extracted from social media to specific colors and emotion intensities that can be later used for creating tangible experiences through visualization and sonification. Other works that have used the Hourglass model for the arts include [23,24,25].

The Arch: Pairing Colors with Tangible Experiences

In Mood of the Planet, the visual correlative for emotion is non-objective or abstract imaging, employing pure color and geometric shapes that change shape, or animate, in time. This is because emotions are experienced internally rather than externally, and so do not have a shape or form that would conform to an external object or real-world photographic representation. Emotions are also dynamic, fluctuating in response to unfolding events in the world. Thus, the images are not objective representations of the external world, like pictures of cars and trees, rather they are non-objective representations or evocations of internal states that more often look like colorful swirls of lines and shifting rhythmic patterns. Physical computing and installation were selected as media as they are highly immersive, and analogous to the invisible data carried through waves in the atmosphere that surround us, like the constantly changing weather sweeping endlessly across our planet, or like the Chinese concept of Chi, where energy is thought to be constantly swirling around and through us in spirals.

Conceptual Design

Emotional experience arises from many complex, interrelated personal and interpersonal factors (such as health and security, personal events such as births, deaths, marriage, divorce, loss of jobs, or making new friends, or achieving new professional heights), that broadly speaking are related to “understanding our place in the world and in the Universe” during a lifetime. Data, and Big Data in particular, can provide a new understanding in this regard. Instead of simply being a harbinger of the next stock market rise or fall, or a way to target products via mass media to individuals, it can be used to explore patterns that connect people across time and space.

Historically, ancient cultures such as the Ancient Egyptians, Inca, Greeks and Khmer used their building technologies to express their understanding of cosmological phenomena. Tiwanaku, an archeological site in Bolivia, symbolized the Incan understanding of the passage of time and the connections between people, living things, and the heavens (Fig. 9, left). In the current era, social and digital media show us that people seek connections with other people, and also deeper understanding of the environment, especially with regard to the climate crisis today as it impacts survival. The COVID-19 pandemic has shown us that we are not separate from nature. It has also shown the developed world that it can live with much less consumption. Are people happier or more anxious in this “new normal”? What are the best color-emotion relationships to express this? While color-emotion associations are culturally variable, emotions themselves are universal. For Mood of the Planet, we used the Hourglass of Emotions for pairing emotions with color/texture combinations. Using this model, animation and graphical layout of the sculpture were designed, produced and programmed using a combination of software including Adobe Aftereffects and Pure Data/GEM. The sound and music were composed using Pure Data, and employed harmonic, dissonant, rhythmic changes correlated with the 24 different emotions. Influenced by multicultural musical traditions, the music incorporated polyrhythms, speed, and timbre changes, and dynamics. For each display, both the animation and music varied according to the stream of aggregated emotional states.

Fig. 9
figure 9

Stone arch of Kalasasaya in Tiwanaku, Bolivia (left), and Mood of the Planet installation at NTU ADM, Singapore (right)

Full size image

Content Design

Twitter messages were analyzed constantly, but because it takes roughly a minute to capture, sort and analyze the data, the entire analytical process could not run in real time [26]. This resulted in a time lag in the playback of animation and music reflecting the previous minute’s analysis. Each minute was divided into proportions reflecting the 24 emotions detected. For example, if in the minute of data analyzed, one third of the messages reflected emotions belonging to the positive spectrum of Introversion (i.e., ecstasy, joy, and contentment), then 20 s (33.3% of 1 minute) would be devoted to different shades of yellow. If another third of the data collected was anger, then 20 s would play in red. If the remaining 33.3 % were equally split between “Fear” and “Disgust,” then 10 s each would play in purple and green, respectively. Corresponding music would play in the same proportions and synchronized with the changing colors. Each minute the display would start over, reflecting a new set of texts analyzed, and with the new proportions of time, color, and sound.

Physical Design

Physically, the overall installation design was inspired by the Inca archeological site of Tiwanaku, mentioned above. The central portal at Tiwanaku is an arch shape, an upside down U. The sun can be seen entering it at sunrise on the Winter Solstice, June 21. Every year there is an annual ritual that takes place with Incan descendants still living in the region high in the Andes mountains of Bolivia. A human figure made of rock is seen through the portal, facing it as one enters. Similar to Tiwanaku, the central sculpture in Mood of the Planet is also an arch shape and both light and a person can be seen through it, looking at the portal. In this case the person that one sees is a reflection of oneself.

The arch is an upside down extruded U with the front and back sides of the U separated by a space to house electronics (Fig. 10). The front and back of the arch are entirely covered with LED “digi-tiles” that emit light (Fig. 11). The in-between surfaces are covered with black metal sheeting that can be removed to allow access to the electronics. There are 15 “digi-tiles” on each side of the sculpture, for a total of 30. Each consists of 4 LED arrays linked together into a square measuring 44.6 cm × 44.6 cm, and covered with diffusion film to eliminate hot spots produced by individual LEDs. A layer of crushed recycled glass is placed over this, and then this “sandwich” is placed in a clear Plexiglas box attached to a custom steel frame.

Fig. 10
figure 10

Diagram of the flat panel arrays and steel frame for the arch construction

Full size image
Fig. 11
figure 11

Crushed glass digi-tiles

Full size image

The crushed glass refracts the light emitted from the LED array, which produces the effect of a glowing cube of gem stones, with colored highlights that shimmer with changing rainbow colors. Behind each LED panel are chips and wires that control the display, and network them to a master computer running custom software that is housed inside the sculpture. A proximity sensor is also embedded, and inside the main sculpture is a sound system with speakers interfaced to the system so that the sculpture emits sound as well as light in real time. The sculpture, weighing about 500 kg given the glass, steel, and electronics, is placed in the middle of a raised wooden walkway (hiding power and other cables underneath) that traverses the middle of the arch, so that people can walk through it and see the arch from both sides. The arch is positioned inside a large room with 2 parallel walls covered with mirrors, facing both sides of the arch (Fig. 12). The two glowing sides, when viewed from the walkway of the arch, produce an endless tunnel of light.

Fig. 12
figure 12

Installation design and layout with measurements

Full size image

The entry “hall” to the room housing the arch is based on traditional temple designs, which often include more than one gate. Each gate represents a passage to the next stage of life. In Mood of the Planet, the entry is designed so that visitors enter through a passage that brings them directly into in the middle of the sculpture on the platform, and from there they walk to the inside of the infinite tunnel of light. As the visitor enters, a proximity sensor activates the sculpture with rainbow colors that represent the full spectrum of emotions existing across all populations, and as a way to welcome the visitor. When the person passes through the arch, the process of text analysis and display of the mood of the planet begins.

Discussion and Future Work

Following Mood of the Planet’s premiere in 2015, Mood of New York was created as an interactive painting, at Experiments in Art, Design and Technology (EADT) as part of New York Tech Week 2016, in Manhattan, NY, USA. In 2017, Mood of Singapore was installed at the Experimental Medicine Building of Nanyang Technological University (NTU), where it is now part of the NTU Campus Art Trail. Due to legal limitations accessing and analyzing Twitter feeds in Singapore, initially Mood of Singapore was not allowed to run in real time by NTU. Hence, a simulation was substituted. In New York City, the Twitter feeds were analyzed, and guests to the exhibition were found to be spending up to 30 min with Mood of New York.

Both of these works were considered to be interactive, dynamic paintings rather than sculptures, as they were housed in large rectangular frames and hung on walls. Both had sound systems and multimedia computers placed behind the frames and LED arrays, as well as sensors in front to detect visitor interaction. In 2019, Mood of the Planet was re-installed at NTU School of Art, Design and Media (ADM), this time in the building’s entrance courtyard (Fig. 9, right). It also inspired the cover of NTU’s Pushing Frontiers magazine, where it was discussed in a feature article about art and technology.Footnote 1

It was expected that the audience engaging with Mood of the Planet would find the abstraction confusing and would require more direct representation of the moods displayed. Reactions from the audience were probed both anecdotally in situ and in follow-up conversations between the artist, visitors, and gallerists who observed and spoke with visitors. Contrary to expectations, the visitors easily followed the color and sound, even if some of the guests associated color and emotion differently. Visitors experienced varying associations between emotions and colors. These alternative color-emotion models were investigated and the results used to explore their integration in a real-time interactive installation for a multicultural audience, anticipating the installation of the work in different cultures and locations, both physically and online (Fig. 13). Additional considerations were the limitations of Twitter for accessing populations unreachable by social media networks, and especially those whose everyday languages are not English [27]. This led to further research correlating emotion with multimodal data, including facial expression, body temperature, touch, and other inputs to the system from local and remote locations [28]. Some guests, in fact, thought that their own observation of the work affected the Twitter feed (even though the guest had not sent any Twitter message).

Fig. 13
figure 13

Different configurations of Mood of the Planet

Full size image

As future work, we plan to enable guests to provide input to the system. One solution could be to ask the audience to connect their Twitter account to the installation in order to visualize the emotions contained in the guest’s tweets. Another solution could be to display the current emotions of the audience in real time by applying multimodal fusion techniques [29, 30] on different inputs, e.g., facial expression, voice and tactile input. Finally, we also plan to allow Mood of the Planet to perform finer-grained real-time visualizations of the world’s emotions about a specific topic, event or issue upon request.

Conclusion

As a global artwork, Mood of the Planet rethinks the term “public art” in the context of social and interactive multimedia, creating a spiritual journey from the everyday preoccupation with the self to our universality, our place in the world. The “arch” or “doorway” is inspired by the ancient Incan archeological site, Tiwanaku, an iconic shape in many world cultures. Arches often symbolize developmental transformation, a metaphoric passage between states. The arch also invokes the human transformation of the environment, today both physical and digital. In the Mood of the Planet installation, the “doorway” is reflected within a wall-mirrored room where it is repeated into a tunnel-like shape that exists as an endless cycle, or echo, of past and future in space and time, that collapses into the eternal present. A wooden pathway traverses the room housing the arch and passes through the doorway, connecting the two mirrored walls. It thus creates a seemingly infinite pathway for the audience to walk through and reflect upon.

The arch sculpture is made of 30 building blocks of “digi-tiles.” Consisting of crushed recycled glass and custom electronics, they emit colored light. The colors and shapes change in real time based on a continuous analysis of keywords and multiword expressions representing human emotions, emanating from Twitter and from all across our planet. The current mood of the people of the Earth through color and motion thus become an immersive presence, a dynamic rainbow that bathes us in light.