Process
The project was initially focused on how we could visualise sound, and determining a concept to connect with this goal involved a somewhat intricate process. Among the potential concepts considered were ASMR mukbang, Childhood Memory/Cultural Diversity, and The Nostalgic Emotion.
The idea for ASMR mukbang involved visualising sounds related to food (chewing, packaging, etc.) to obtain intuitive results. To further develop this concept, we combined Childhood Memory and Cultural Diversity, ultimately collecting sounds of eating various childhood snacks and visualising them.
The plan was to extract the Hertz values present in each audio and input them into Touch Designer to create unique visuals for each sound. However, when attempting this, several flaws were identified.
Firstly, audio does not have a single Hertz value but multiple, leading to a broad data collection that reduces accuracy. Additionally, the data itself, being numeric, gave a somewhat monotonous feel when visualised, potentially causing issues in creating artefacts during the visualisation process. Overall, the concept was complex and intertwined, resulting in internal confusion during the execution. Consequently, the concept was narrowed down to 'Soundscapes of Memory.'
Proposal, background
The project initially aimed to visualize sound, considering concepts like ASMR mukbang, Childhood Memory/Cultural Diversity, and The Nostalgic Emotion. ASMR mukbang focused on visualizing food-related sounds for intuitive outcomes. Combining Childhood Memory and Cultural Diversity, we collected sounds of eating childhood snacks for further development. The plan involved extracting Hertz values from audio and using Touch Designer to create unique visuals for each sound. However, flaws emerged, such as the presence of multiple Hertz values in audio data, leading to broad and less accurate data collection. The numeric nature of the data resulted in a monotonous visual representation, posing challenges in creating artefacts. The overall concept complexity led to internal confusion during execution, prompting the decision to narrow down the concept to 'Soundscapes of Memory.'
Proposed objective
To develop and implement a multimedia sensory approach that emphasises the auditory and visual senses by utilising audio recordings of nostalgic things/sounds and synchronising the sounds visuals for an enriched sensory experience.
Proposed approach
Focusing on imaginative data aligned with our concept, "Soundscape of Memory," we conducted surveys to identify sounds triggering childhood nostalgia. The initial survey explored personal resonances with sounds, yielding responses like mom’s cooking, school bell, and slime. To formalize this data, we designed another survey, prompting participants to record or download nostalgic sounds and draw the images evoked. This survey aimed to capture people's memories through emotional triggers, transforming nostalgic sounds into visual representations. The depicted photo showcases participants listening to a nostalgic sound and immediately drawing associated images, serving as the final data for our concept.
Data collected
With the intention of collecting data that is both interesting and closely aligned with our concept, we decided to focus on imaginative data.
Our revised concept, "Soundscape of Memory," revolves around the sounds that evoke childhood nostalgia. To determine the most suitable data to describe this concept, we pondered on what sounds trigger a sense of nostalgia for individuals. Consequently, we conducted a survey before initiating the project to explore the sounds that personally resonate with people. Some interesting responses included sounds like mom’s cooking, school bell, and slime.
To formalise this data and transform it into proper data, we designed another survey specifically for imaginative data. In this survey, participants directly recorded or downloaded various nostalgic sounds acquired from the earlier survey. Since certain sounds might be exclusive to specific regions, we aimed to play these sounds for participants and prompt them to draw the images that came to mind.
In essence, we utilised nostalgic sounds as emotional triggers to visualise people's memories and the associated sounds. As depicted in the photo, participants listened to a nostalgic sound, immediately drawing the images that came to mind. These drawings serve as the final data to convey our concept.
Artefact
With the data ready, we sought a way to digitise it and display it interactively. We decided to revisit TouchDesigner, previously used for generating artefacts. To elaborate, we converted the prepared drawings into a point cloud format (pixelating the images) using TouchDesigner. We synchronised the movement and speed of these visuals to correspond with the audio, creating a motion that mimicked memories coming to life.
After infusing vitality into the predetermined images, we aimed to make them interactive. We connected them to the keyboard, allowing the audience to change between various visuals using keyboard and button functionalities.
Once the audio-reactive visuals responding to the keyboard were ready, the next consideration was how to exhibit them. We opted to directly connect the TouchDesigner file to a projector for display. Using the full-screen presentation feature in TouchDesigner, we projected the audio-reactive visuals onto a wall, enabling the audience to freely change visuals using the keyboard. However, a dilemma arose during this process. To use the projector effectively, an enclosed area was needed, and conducting this in a simple dark room did not align with the virtues of an exhibition. Consequently, we contemplated constructing cardboard huts to provide a defined space for the audience to sit and view a clear screen.
Due to time constraints, this plan could not be executed. Instead, as an alternative, we created a space for the audience to sit and view the physical drawings and digitised audio-reactive visuals. To demarcate the exhibition space, we used a large blackboard to divide the area. We hung black paper on the wall where the visuals were displayed to enhance visibility.
Ultimately, this approach, integrating physical drawings with TouchDesigner-generated visuals responsive to nostalgic sounds, added vibrancy to the exhibition. The setup, situated in an enclosed space with child-sized chairs and tables encourages the audience to reminisce their childhood memories in a more lively and vibrant manner.
Conclusion
Overall, the visuals communicate with the audience and reflect the audio quite well. The audience were able to evoke a sense of curiosity and a feel like they are walking down the memory lane so this part was effective and impactful. The setting could be a little different to enhance the childhood part. Our initial idea of building a children’s cardboard house could have been implemented for this project. The metal stands used for the white screen could be covered up better to make the video look better.
The interplay between the imaginative and digitalized elements in our project was intriguing, particularly in the initial phase where participants engaged in painting while listening to sounds. Although this method effectively provided us with a clearer direction, it became a little disappointing that we veered away from this hands-on approach and leaned towards a more digitalized aspect.
So, the shift was not as progressive as anticipated. Balancing the imaginative and digital aspects was a notable challenge, and that we should have a more focused approach. Our emphasis should be on maintaining that tactile connection while incorporating digital elements to enhance rather than overshadow the project.