HEAR-ME: A Multisensory Investigation into Enhanced Music Perception of Cochlear Implant Users
Who
Dr Lauren Hayes (PI), assistant professor, Arts, Media & Engineering, ASU
Dr Xin Luo (PI), assistant professor, Department of Speech & Hearing Science, ASU
Gabriella Isaac, graduate student, MA Digital Culture, AME, ASU
Dominic Bonelli, student, BS Electrical Engineering, ASU
Slater Olsen, student, BA Digital Culture, AME, ASU
Kiana Gerard, ARES student, SHS, ASU
Firas Haddadin, student, SHS, ASU
Katie Pulling, research specialist, SHS, ASU
Marije Baalman, artist & researcher, ex-STEIM, Amsterdam
Why
- Cochlear Implant (CI) is a medical device that partially restores hearing sensation to profoundly deaf people by electrically stimulating the surviving auditory neurons.
- CI users have trouble with speech recognition in noisy environments and various
aspects of music perception
- F0 is not faithfully encoded by CIs which presents problems in music perception
- Some CI users have residual low frequency acoustic hearing up to ~1000 Hz
- When a hearing aid used to provide low frequency acoustic cues for CI users, their speech recognition in noise and music perception greatly improve
- With no residual hearing, a tactile aid could help
- Evelyn Glennie:
Hearing is basically a specialized form of touch. Sound is simply vibrating air which the ear picks up and converts to electrical signals, which are then interpreted by the brain. The sense of hearing is not the only sense that can do this, touch can do this too.
Background
- Vibration and vibrotactile feedback to accompany music and sonic cues can enhance the experience for performers and listeners alike
- performer-instrument relationship for augmented/digital instruments
Isaac, G, Hayes, L and Ingalls, T. 2017. Cross-Modal Terrains: Navigating Sonic Space through Haptic Feedback. In Proceedings of the International Conference on New Interfaces for Musical Expression. Copenhagen
Hayes, L. 2012. Performing Articulation and Expression through a Haptic Interface. In Proceedings of the 2012 International Computer Music Conference (pp. 400-403).
​
Hayes, L. 2011. Vibrotactile Feedback-Assisted Performance. In Proceedings of the International Conference on New Interfaces for Musical Expression (pp. 72-75). Oslo, Norway.
Hayes, L. 2013. Haptic Augmentation of the Hybrid Piano. Contemporary Music Review, 32(5). 499-509. London: Taylor & Francis.
- networked communication between musicians; cueing; conduction
Hayes, L and Michalakos, C. 2012. Imposing a Networked Vibrotactile Communication System for Improvisational Suggestion. Organised Sound, 17(1). 36-44. Cambridge University Press.
- audio-haptic compositions
Hayes, L. 2015. Skin Music (2012): an Audio-Haptic Composition for Ears and Body. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition (pp. 359-360). ACM.
​
Approach
- Vibrotactile thresholds tested at different sites
- Vibrotactile perception adaptation
- Frequency interval detection
- Melodic contour detection, improved with training
- worse than normal hearing, comparable to CI hearing
- approx. 7% improvement on melodic contour with haptic feedback and CI simulation
