Existing binaural approaches to speech segregation place an exclusive burden on localization information. These approaches can achieve excellent performance in anechoic conditions but degrade rapidly in realistic environments where room reverberation corrupts binaural cues. In this project we propose integrating monaural grouping and binaural localization to achieve sequential organization and segregation of speech in reverberant environments.
In this project we look at methods for analyzing the direction of arrival and the level of directionality of sound energy using a binaural hearing aid setup. The goal of this work is to avoid any assumptions on the direction of arrival of the target source and to enhance any sound that is identified as being directional. This is naturally a more conservative approach than traditional spatial filtering with the idea that designs that facilitate listening in difficult environments without selecting an individual source or direction may be of interest to some hearing impaired listeners.
The ideal binary mask (IBM) has been established as a valid computational goal for CASA systems and has been shown to produce dramatic improvements in speech intelligibility for both normal hearing and hearing impaired listeners. A definition of the IBM in reverberant environments has not yet been established and with this project we seek to provide an appropriate definition by comparing subjective intelligibility improvements with alternative IBM definitions.
In this project we explore a novel method of estimating sinusoidal parameters (amplitude and phase) of instrument harmonics that overlap with eachother in time-frequency space. We utilize the principle that an instrument's harmonics tend to have correlated amplitude envelopes and that changes in phase can be predicted from a signal's fundamental frequency. With these two observations we are able to determine the parameters of overlapped harmonics using a least-squares estimation framework. Our results indicate that this approach achieves significant performance improvements over state of the art monaural music separation systems.
Audio Examples
J. Woodruff and D. L. Wang. "Binural speech segregation based on pitch and azimuth tracking," accepted to ICASSP 2012.
J. Woodruff and D. L. Wang. "Binaural localization of multiple sources in reverberant and noisy environments," accepted for publication in IEEE Trans. Audio, Speech and Lang. Proc.
N. Roman and J. Woodruff. "Intelligibility of reverberant noisy speech with ideal binary masking," in J. Acoust. Soc. Amer., vol. 130 (4), pp. 2153-2161, 2011. [.pdf , abstract]
J. Woodruff and D. L. Wang. "Directionality-based speech enhancement for hearing aids," in Proc. ICASSP, 2011. [.pdf]
J. Woodruff, R. Prabhavalkar, E. Fosler-Lussier and D. L. Wang. "Combining monaural and binaural evidence for reverberant speech segregation," in Proc. INTERSPEECH, 2010. [.pdf]
J. Woodruff and D. L. Wang. "Sequential organization of speech in reverberant environments by integrating monaural grouping and binaural localization," in IEEE Trans. Audio, Speech and Lang. Proc., vol. 18, no. 7, pp. 1856-1866, 2010. [.pdf]
J. Woodruff and D. L. Wang. "Integrating monaural and binaural analysis for localizing multiple reverberant sound sources," in Proc. ICASSP, 2010. [.pdf]
Y. Li, J. Woodruff and D. L. Wang. "Monaural musical sound separation using pitch and common amplitude modulation," in IEEE Trans. Audio, Speech and Lang. Proc., vol. 17, no. 7, pp. 1361-1371, 2009. [.pdf]
J. Woodruff and D. L. Wang. "On the role of localization cues in binaural segregation of reverberant speech," in Proc. ICASSP, 2009. [.pdf , poster]
J. Woodruff, Y. Li and D. L. Wang. "Resolving overlapping harmonics for monaural musical sound separation using pitch and common amplitude modulation," in Proc. ISMIR, 2008. [.pdf , poster]
J. Woodruff and B. Pardo. "Using pitch, amplitude modulation and spatial cues for separation of harmonic instruments from stereo music recordings," in EURASIP J. Adv. Signal Proc., vol. 2007. [.pdf]
J. Woodruff, B. Pardo and R. Dannenberg. "Remixing stereo music with score-informed source separation," in Proc. ISMIR, 2006. [.pdf]
A. D. Shamma, B. Pardo and J. Woodruff. "MusicStory: an autonomous, personalized music video creator," in Intelligent Music Information Systems: Tools and Methodologies, J. Shen, J. Shepherd, B. Cui, L. Liu, Eds., 2007. [link]
J. Woodruff and B. Pardo. "Active source estimation for improved source separation," Technical Report, Northwestern University, Dept. of Electrical Engineering and Computer Science, NWU-EECS-06-01, 2006. [.pdf]