Psychoacoustic Sonification

Sound for Data Presentation, Sonic-Interaction and Augmented Reality

Visualization is the presentation of data by means of graphics. Sonification is the presentation of data by means of sound. It can this be considered as the auditory counterpart of visualization. Sonifying low amounts of data is straightforward, just like visualizing time series as plots. But in order to sonify high amounts of data, the sonification design needs to take the complicated, nonlinear processing of the auditory system into account. The goal of this project is the development and experimental evaluation of a psychoacoustic sonification that

is clearly interpretable,
can present mutidimensional and multivatiate data in a perceptually orthognal way
exhibits a high resolution
can present absolute and relative magnitudes.

Psychoacoustic Sonification Goals — Interpretable sonifications need linear, orthogonal dimensions with a high resolution, both polarities and a clear coordinate origin.

The current state of the project is the successful implementation and evaluation of a sonification for three-dimensional data. It can serve for navigation in three-dimensional space, like video games, image-guided surgery, and virtual reality applications.

Sonification Apps

Three sonification apps are freely available:

1. The CURAT Sonification Game

The CURAT Sonification game has been developed in order to provide us with research data. The game is available for Android, and for Linux, Windows and MacOS. Play the game, experience sonification and improve your hearing skills. If you give us the permission, we will track your game stats to evaluate and improve the psychoacoustic sonification. Its main aim is a navigation aid for image-guided surgery. You can find the CURAT teaser video and short gaming videos on YouTube.

2. Tiltification

TIltification is a spirit level app. The sonification helps you align your smartphone horizontally. This enables you to level your table, hang a painting or take perfectly horizontal pictures. Tiltification is available for Android and for iOS. With over 30,000 Downloads, Tiltification may be the best-known sonification app worldwide. Our YouTube-Playlist introduces you to the app and its main functions.

Tiltification Sonification App — Tiltification: Level your phone, guided by sound

3. Sonic Tilt

Tiltification utilizes our psychoacoustic sonification for two-dimensional data. The sonification has a huge information-density and allows for very precise navigation. But the sonification is not very pleasant for inexperienced listeners. This is why we made the Tiltification source code freely available under the name Sonic Tilt. You can download and compile the Android app and replace our sonification with your own sound design. Our Video Tutorial helps you download, modify and compile the app.

Graphic: Navid Mirzayousef Jadid

Sonic Tilt: Rebuild Tiltification using your own sound design

Events

Several events took place in the course of the Psychoacoustic Sonification project:

The Sonic Tilt Competition 2023

The Sonic Tilt Competition was a Sonification Design competition endoresed by ICAD. Participants reinvented the Tiltification sound using the Sonic Tilt source code. A hands on workshop was provided at the Interactive Sonification Workshop (ISon 2022). In addition, a written manual and a YouTube tutorial were provided. The award ceremony took place at the International Conference on Auditory Display (ICAD 2023). Documentations of all submissions were included in the ICAD 2023 proceedings together with a short editorial on the Sonic Tilt Competition.

Interactive Sonification Workshop (ISon) 2022

The Interactive Sonification Workshop (ISon) 2022 had the theme “Psychoacoustics in the Loop”. It was organized by Tim Ziemer, Thomas Hermann and Roberto Bresin and the paper chairs Sara Lenzi and Niklas Rönnberg. The Proceedings of the Interactive Sonification Workshop (ISon 2022) [PDF] are provided here. Another result from this workshop is the Special Issue on Design and Perception of Interactive Sonification in Springer's Journal on Multimodal User Interfaces (JMUI).

The Tiltification project

The Tiltification project was a master's project for computer science and digital media students at the University of Bremen and the University of Arts Bremen. The outcomes are the Sonic Tilt Software, the Sonic Tilt Competition, and the famous Tiltification app.

The CURAT Project

The CURAT project started as a bachelor's project and continued as a master's project in computer science at the University of Bremen. Students developed software to train cognitive abilities that are necessary to carry out laparoscopic procedures, like minimally-invasive surgery. One software is focused on hand-eye coordination and visuo-spatial orientation. Another software aims at avoiding the high visuo-spatial load by sonification-guidance. Once learned during surgical training, mistakes resulting from reference frame conflicts can be avoided if clinicians navigate using sonification.

Teaching

Ziemer, Tim: Seminar „Sonifikation und Musik”: Implementierung einer Sonifikation als Handy-App unter Berücksichtigung von Prinzipien der Psychoakustik, Multimodalität und Human-Computer Interaction [For Bachelor and Master Students in Systematic Musicology at the University of Hamburg, summer semester 2023, 3 semester hours]

Ziemer, Tim & Kleinert, Steffen: Hands-On Workshop “ICAD2023: Sonic Tilt Competition – Implementing your own Sonification in an Android Spirit Level App”: Sound Design and Programming [For researchers and PhD students, 4 hours at the Interactive Sonification Workshop (ISon) 2022]

Ziemer, Tim: Online Masters’ Project “Sonification Apps”: Conceptualization, implementation and release of Tiltification [For Master students in Computer Science and Digital Media at the University of Bremen and the University of Arts Bremen, winter semester 2020, 8 semester hours]

Ziemer, Tim: Seminar “Sonification”: Definitions, use cases, cognitive and psychoacoustic principles of auditory display and practical implementation For Bachelor and Master students in Computer Science or Musicology at the University of Bremen, winter semester 2019, 2 semester hours)

Ziemer, Tim: Practical course “Psychoacoustical Audio Programming”: Definitions, use cases, cognitive and psychoacoustic principles of auditory display and practical implementation [For Bachelor and Master students in Computer Science or Musicology at the University of Bremen, summer semester 2019, 2 semester hours)

Ziemer, Tim: Seminar “Applied Psychoacoustics”: Theoretic foundations and practical implementation of psychoacoustics in spatial audio, sonification and music production [For Bachelor’s and Master’s students in Systematic Musicology at the University of Hamburg, winter semester 2018, 2 semester hours]

Ziemer, Tim: Seminar „Psychoakustische Sonifikation”: Praktische Anwendung der Psychoakustik in Audioprogrammierung [For Bachelor and Master students in Systematic Musicology at the University of Hamburg, summer semester 2018, 2 semester hours]

Publications

Feel free to contact me via ResearchGate ([RG] links) to get a private copy of a paper.

Journal papers:

Ziemer, Tim, “Three-Dimensional Sonification as a Surgical Guidance Tool”, in: Journal on Multimodal User Interfaces 17(4), 2023, https://doi.org/10.1007/s12193-023-00422-9 [RG] [BIB]

Ziemer, Tim & Schultheis, Holger, “PAMPAS: A PsychoAcoustical Method for the Perceptual Analysis of multidimensional Sonification”, in: Frontiers in Neuroscience 16, 2022, paper number: 16:930944, https://doi.org/10.3389/fnins.2022.930944 [Open Access] [BIB]

Vajsbaher, Tina, Ziemer, Tim & Schultheis, Holger, “A multi-modal approach to cognitive training and assistance in minimally invasive surgery”, in: Cognitive Systems Research 64, 2020, pp. 57–72, https://doi.org/10.1016/j.cogsys.2020.07.005 [RG] [BIB]

Ziemer, Tim, Nuchprayoon, Nuttawut & Schultheis, Holger, “Psychoacoustic Sonification as User Interface for Human-Machine Interaction” in: International Journal of Informatics Society 12(1), 2020, pp. 3–16, http://www.infsoc.org/journal/vol12/12-1 [RG] [BIB]

Ziemer, Tim & Schultheis, Holger, “Psychoacoustic Auditory Display for Navigation. An Auditory Assistance System for Spatial Orientation Tasks”, in: Journal on Multi-Modal User Interfaces 13(3), 2019, pp. 205–218, http://doi.org/10.1007/s12193-018-0282-2 [RG] [BIB]

Ziemer, Tim, Schultheis, Holger, Black, David & Kikinis, Ron, “Psychoacoustical Interactive Sonification for Short-Range Navigation”, in: Acta Acustica United With Acustica 104(6), 2018, pp. 1075–1093, http://doi.org/10.3813/AAA.919273 [RG] [BIB]

Conference Papers:

Ziemer, Tim, “Three-Dimensional Sonification for Image-Guided Surgery”, in: 28th International Conference on Auditory Display (ICAD 2023), https://doi.org/10.21785/icad2023.2324, Norrköping, Sweden, June 2023, pp. 8–14 [Open Access] [BIB]

Ziemer, Tim, “The Sonic Tilt Competition”, in: 28th International Conference on Auditory Display (ICAD 2023), part of https://doi.org/10.21785/icad2023.000, Norrköping, Sweden, June 2023, pp. 203–206 [DL] [RG] [BIB]

Ziemer, Tim, “Mixing Levels – A Rock Music Spirit Level App”, in: 28th International Conference on Auditory Display (ICAD 2023), part of https://doi.org/10.21785/icad2023.000, Norrköping, Sweden, June 2023, pp. 233–235 [DL] [RG] [BIB]

Ziemer, Tim, “Visualization vs. Sonification to Level a Table”, in: Interactive Sonification Workshop (ISon) 2022, part of https://doi.org/10.5281/zenodo.7552217, Delmenhorst, Germany, September 2022, pp. 31–36 [RG] [BIB]

Kleinert, Steffen & Ziemer, Tim, “Sonic Tilt Manual: Create your Own Version of Tiltification”, in: Interactive Sonification Workshop (ISon) 2022, Delmenhorst, Germany, September 2022 [DL] [RG] [BIB]

Ziemer, Tim & Schultheis, Holger, “Both Rudimentary Visualization and Prototypical Sonification can Serve as a Benchmark to Evaluate New Sonification Designs”, in: Proceedings of the 16th International Audio Mostly Conference, https://doi.org/10.1145/3561212.3561228, St. Pölten, Austria, September 2022 [RG] [BIB]

Ziemer, Tim & Jadid, Navid Mirzayousef, “Recommendations to Develop, Distribute and Market Sonification Apps”, in: 27th International Conference on Auditory Display (ICAD 2022), Virtual Conference, June 2022 https://doi.org/10.21785/icad2022.003, pp. 7–15 [Open Access] [BIB]

Ziemer, Tim & Schultheis, Holger, “The CURAT Sonification Game: Gamification for Remote Sonification Evaluation”, in: 26th International Conference on Auditory Display (ICAD 2021), https://doi.org/10.21785/icad2021.026, Virtual Conference, June 2021, pp. 233–240 [OPen Access] [BIB]

Asendorf, Malte, Kienzle, Moritz, Ringe, Rachel,Ahmadi, Fida, Bhowmik, Debaditya, Chen, Jiumeng, Huynh, Kelly, Kleinert, Steffen, Kruesilp, Jatawan, Lee, Ying Ying, Wang, Xin, Luo, Wei, Jadid, Navid Mirzayousef, Awadin, Ahmed, Raval, Varun, Schade, Eve Emily Sophie, Jamam, Hasanur, Sharma, Kashish, Weber, Colin, Winkler, Helena & Ziemer, Tim, “Tiltification – An Accessible App to Popularize Sonification”, in: 26th International Conference on Auditory Display (ICAD 2021), https://doi.org/10.21785/icad2021.025, Virtual Conference, June 2021, pp. 184–191 [Open Access] [BIB]

Ziemer, Tim & Schultheis, Holger, “Psychoakustische Sonifikation zur Navigation in Bildgeführter Chirurgie”, in: Rebecca Grotjahn; Nina Jaeschke (Eds.): Freie Beiträge zur Jahrestagung der Gesellschaft für Musikforschung 2019 – 1, https://doi.org/10.25366/2020.76, Paderborn/Detmold, Germany, September 2019, pp. 347–358 [Open Access] [BIB]

Ziemer, Tim, Höring, Thomas, Meirose, Lukas & Schultheis, Holger, “Monophonic Sonification for Spatial Navigation”, International Workshop on Informatics (IWIN) 2019, part of http://www.infsoc.org/conference/iwin2019/download/IWIN2019-Proceedings.pdf#page=95, Hamburg, Germany, September 2019, pp. 83--87 [RG] [BIB]

Ismailogullari, Abdullah & Ziemer, Tim, “Soundscape Clock: Soundscape Compositions That Display the Time of the Day”, in: 25th International Conference on Auditory Display (ICAD2019), http://doi.org/10.21785/icad2019.034, Newcastleupon Tyne, UK, June 2019 [Open Access][BIB]

Schwarz, Sebastian & Ziemer, Tim, “A Psychoacoustic Sound Design for Pulse Oximetry”, in: 25th International Conference on Auditory Display (ICAD2019), https://doi.org/10.21785/icad2019.024, Newcastle upon Tyne, UK, June 2019 [Open Access] [BIB]

Ziemer, Tim & Schultheis, Holger, “Psychoacoustical Signal Processing for Three-Dimensional Sonification”, in: 25th International Conference on Auditory Display (ICAD2019), https://doi.org/10.21785/icad2019.018, Newcastleupon Tyne, UK, June 2019 [Open Access] [BIB]

Ziemer, Tim & Schultheis, Holger, “A Psychoacoustic Auditory Display for Navigation”, in: 24th International Conference on Auditory Display (ICAD2018), http://doi.org/10.21785/icad2018.007, Houghton (MI), USA: Jun 2018, pp. 136–144 [Open Access] [BIB]

Ziemer, Tim, Black, David & Schultheis, Holger, “Psychoacoustic Sonification Design for Navigation in Surgical Interventions”, in: Proceedings of Meetings on Acoustics (POMA) 30(1), https://doi.org/10.1121/2.0000557, 2017, Paper-Number: 050005 [Open Access][BIB]

Abstracts and Précis:

Ziemer, Tim, Lenzi, Sara, Rönnberg, Niklas, Hermann, Thomas & Bresin, Roberto, “Introduction to the Special Issue on Design and Perception of Interactive Sonification”, in: Journal on Multimodal User Interfaces 17(4), https://doi.org/10.1007/s12193-023-00425-6, [Open Access] [BIB]

Schultheis, Holger, Ziemer, Tim & Vajsbaher, Tina, “Cognitive Surgical Assistance in Minimally-Invasive Interventions”, 8. Interdisziplinären Workshops zu Kognitiven Systemen, https://www.researchgate.net/publication/332901260_Cognitive_Surgical_Assistance_in_Minimally-Invasive_Interventions, Duisburg, Germany, March 2019

Ziemer, Tim & Schultheis, Holger, “Perceptual Auditory Display for Two-Dimensional Short-Range Navigation”, in: Fortschritte der Akustik – DAGA 2018, https://www.dega-akustik.de/publikationen/online-proceedings/, Munich, Germany 2018, pp. 1094–1096 [RG] [BIB]

Ziemer, Tim & Black, David, “Psychoacoustically motivated sonification for surgeons”, in: International Journal of Computer Assisted Radiology and Surgery (JCARS) 12(Suppl 1:1), http://doi.org/10.1007/s11548-017-1588-3, 2017, pp. 265–266 [RG]

Ziemer, Tim, “Two-dimensional psychoacoustic sonification”, in: Franziska Olbertz (Ed.): 33. Jahrestagung der deutschen Gesellschaft für Musikpsychologie (DGM), https://www.researchgate.net/publication/319778727_Two-dimensional_psychoacoustic_sonification, Hamburg, Germany, September 2017, pp. 60–61.

Ziemer, Tim, Black, David & Schultheis, Holger, “A Psychoacoustic Review of Auditory Displays in the OR”, 16. Jahrestagung der Deutschen Gesellschaft für Computer- und Roboterassistierte Chirurgie e.V. (CURAC), https://www.researchgate.net/publication/320958966_A_Psychoacoustic_Review_of_Auditory_Displays_in_the_OR_Precis, Hannover, Germany, October 2017

Other Publications:

Ziemer, Tim, “Numbers, Facts and Methods, State-of-the-Art, Current Trends and Evaluation of Arguable Claims and Statements — A 2024 Overview”, Preprints 2023, https://doi.org/10.20944/preprints202312.0138.v1

Rönnberg, Niklas, Lenzi, Sara, Ziemer, Tim, Hermann, Thomas & Bresin, Roberto: Psychoacoustics in the Loop, Proceedings of the 7th Interactive Sonification Workshop, Delmenhorst, Germany, September 2022, https://doi.org/10.5281/zenodo.7552268

Ziemer, Tim & Schultheis, Holger, “Three Orthogonal Dimensions for Psychoacoustic Sonification”, arXiv Preprint 2019, https://doi.org/10.48550/arXiv.1912.00766 and https://doi.org/10.13140/RG.2.2.36037.19683

Ahmadi, Fida Mohammad, Asendorf, Malte, Bhowmik, Debaditya, Chen, Jiumeng, Huynh, Kelly, Kienzle, Moritz, Kleinert, Steffen, Kruesilp, Endy Jatawan, Lee, Ying Ying, Luo, Wei, Jadid, Navid Mirzayousef, Awadin, Ahmed, Raval, Varun, Ringe, Rachel, Schade, Eve Emily Sophie, Seam, Hasanur Jaman, Sharma, Kashish, Wang, Xin, Weber, Colin, Winkler, Helena, Ziemer, Tim, “Tiltification”, Multi-Platform App, 2021, available on Google Play Store: https://play.google.com/store/apps/details?id=de.uni_bremen.informatik.sonification_apps, on Apple App Store: https://apps.apple.com/de/app/tiltification/id1557133147 and as open source project “Sonic Tilt” on GitHub: https://github.com/Tiltification/sonic-tilt

Press Mentions

2023

Stephen Barrass, The Sonic Tilt Competition, sonification.com, 2023

Tim Ziemer, Sonic Tilt Wettbewerb — Ihr Sound in der Wasserwaagen-App, SystMus

Tim Ziemer, Sonifikation — Daten hörend verstehen, SystMus

Markus Ertl und Hellmuth Platz, Apps für Android – zugänglich für blinde Nutzer mit Sprachausgabe bzw. sehbehinderte Nutzer, Berufsförderungswerk Würzburg, Feb. 2023

2022

Rudolf Ring, Tiltification: Mit dieser Gratis-App stehen keine Module mehr schief, Modellbahntechnik Aktuell 78, S. 17–19, Nov. 2022

Lisa, 4 Wasserwaagen-Apps im Überblick, Letsbecrazy.de, Jun. 2022

Lennart Möller, Dr. Tim Ziemer aus Buchholz forscht auf dem Gebiet der Sonifikation, kreiszeitung-wochenblatt.de, Feb. 2022

Kai Uwe Bohn, 20,000 Downloads: The Successful "Tiltification” App, up2date, Mar. 2022

Lennart Möller, Eine Wasserwaage zum Hören, Kreiszeitung Wochenblatt, 19. Feb 2022

Michael Derbort: Die besten neuen Apps, Smartphone – Das große Handytest- und Kaufberatungsmagazin, Jan. 2022

2021

Lena Steinmann, The DSC supports the organization of the hybrid “Interactive Sonification Workshop” (ISon 2022), Data Science Center, Dec. 2021

Tina Bauer: Nachgefragt: Was führt eine akustische Wasserwaage „vor Ohren”?, DUZ 12/21, Dec. 2021

Michael Derbort: Frei nach Gehör: Tiltification, Smartphone – Das große Handytest- und Kaufberatungsmagazin, Nov. 2021

Helmut Hackl: Tiltification macht aus dem iPhone eine akustische Wasserwaage, pocket.at – lässige Produkte, leiwande Apps für iOS & Watch, Nov. 2021

Benedikt Bucher: Intelligente Werkzeug-App: Studenten machen es besser als Apple, CHIP Online, Oct. 2021

Andreas Wilkens: Tiltification: Akustische Wasserwaage fürs Smartphone, Heise Online, Sep. 2021

Patrick Klapetz: Das Bild hängt schief: Nun kann man es auch hören, mdr Wissen, Sep. 2021

Ferdinand Pönisch: Wasserwaage-App: So verwenden Sie Ihr Handy als Wasserwaage, FOCUS Online, Sep. 2021

Benedikt Bucher: Studenten entwickeln App und übertrumpfen sogar Apple: Darum ist diese Smartphone-App deutlich besser, CHIP Online, Sep. 2021

SOA: Anheben, bis es piept, Weser Kurier, Sep. 2021

Nadja Podbregar: Handy-App als akustische Wasserwaage, scinexx – Das Wissensmagazin, Sep. 2021

Stefan Asche: Mit der Wasserwaage hören, ob alles im Lot ist, vdi nachrichten, Sep. 2021

Florian Fügemann: "Tiltification" macht Handy zur Wasserwaage, Pressetext.com, z.B. Financial Times / Remonews.com, Sep. 2021

문광주 기자 (Reporter Kwang-Joo Moon): 기울기 측정하는 휴대폰 어플, The SCIENCE plus, Sep. 2021

TeDo Verlag: Hätten Sie gewusst…, Automation Newsletter 38 2021, Sep. 2021.

iPhone Ticker: Tiltification: App fungiert als akustische Wasserwaage, iphone-ticker.com, Sep. 2021

Nicole Lücke: Neue App: Handy bekommt ungeahnte Funktion und erweitert unsere Wahrnehmung, INGENIEUR.de, Sep. 2021

Market Research Telecast: Tiltification: Acoustic spirit level for the smartphone, Market Research Telecast, Sep. 2021

Rappelsnut: Tiltification: Die akustische Wasserwaage, rappelsnut – Wandern, Punkrock und der ganze Rest, Sep. 2021

Heinz Schmitz: Tiltification macht Handy zur Wasserwaage, Heinz-Schmitz.org – Interessantes und aktuelles aus der IT-Szene, Sep. 2021.

Dr. Torsten Beyer: Nützliche Android-Tools für den Chemiealltag, Analytik-News: Das Online-Labormagazin, Sep. 2021

HanseValley: Studenten der Uni Bremen bringen akustische Wasserwaage aufs Handy, HanseValley – Der digitale Norden, Sep. 2021,

Kai Uwe Bohn: Let’s Hear if the Painting Hangs Crooked/So hört (!) man, ob das Bild schief hängt, Press Release Uni Bremen, Sep. 2021

International Community for Auditory Display: Community Spotlight: Tim Ziemer, ICAD Community Spotlight, Mar. 2021

pn-Verlag: Vom Gamer zum Lebensretter? Mit psychoakustischer Sonifikation die Chirurgie und das Patientenmonitoring unterstützen, KTM – Krankenhaus Technik + Management, Mar. 2021

2020

Kai Uwe Bohn: Playing Video Games to Save Lives? / Mit „Daddeln“ Leben retten?, up2date, Nov. 2020

Arne Grävemeyer: Operieren nach Gehör, c’t 21/20, Heise Medien, S. 58

Rico-Thore Kauert: Mit Computerspielen besser operieren, StartupMag 2020

N/A: Wissenschaftlichen Nachwuchs weiter fördern, Nordwest-Zeitung (NWZ) Online, Sep. 2020

Frederik Grabbe: HWK in Delmenhorst will Nachwuchs fördern, Delmenhorster Kreisblatt, Sep. 2020

Hanse-Wissenschaftskolleg: Wissenschaftliche Nachwuchsförderung am Hanse-Wissenschaftskolleg: Sieben neue Associate Junior Fellows im Postdoc-Programm, Pressemitteilung Hanse-Wissenschaftskolleg, Sep. 2020

Lieselotte Scheewe: Durch Töne Wege finden, Bremen Zwei, September 17, 2020

RBO: Akustischer Wegweiser, ARS MEDICI 20, 2020, S. 607

Quintessence-Publishing: Neue psychoakustische Trainingsmethode für Chirurgen?, www.quintessence-publishing.com, Sep. 2020.

ck/pm: So helfen Töne beim Operieren, Zahnärztliche Mitteilungen, Sep. 2020

Kai Uwe Bohn: Playing Games and Saving Lives by Doing So? / Spiele spielen – und dadurch später Leben retten?, Press Release Uni Bremen, Aug. 2020

2019

Eddie Gandelman: Hearing Is The New Seeing, prioritydesigns 2019

Dauer: Seit 2019
Projektleitung: Dr. Tim Ziemer
Drittmittelgeber: Central Research Development Fund (CRDF), Hanse Institute for Advanced Study, Data Science Center