MIXED REALITY SYSTEM WITH SPATIALIZED AUDIO

• US 20190327574A1
• Filed: 07/03/2019
• Published: 10/24/2019
• Est. Priority Date: 08/01/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: ROACH, David Thomas, MILLER, Samuel A., Sanger, George Alistair, SCHMIDT, Brian, O'GARA, Terry Micheal, TAJIK, Anastasia Andreyevna, TAJIK, Jehangir
• Granted Patent: US 10,856,095 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation

Abstract

A dual source spatialized audio system includes a general audio system and a personal audio system. The general system includes a general audio processor and general speakers to produce general sound. The personal system includes a frame to be worn on a head of a user and a plurality of personal speakers attached to the frame. The personal system also includes a head pose sensor to collect head pose data of the user. The personal system further includes a head pose processor to determine a head pose of the user from the head pose data. Moreover, the personal system includes a personal audio processor to generate personal audio data based on the head pose of the user. The personal speakers generate personal sound corresponding to the personal audio data. The personal processor receives timing information/metadata from the general audio processor to synchronize the personal sound with the general sound.

Claims

1. A spatialized audio system, comprising:

a frame to be worn on a head of a user;

a plurality of speakers attached to the frame such that, when the frame is worn by the user, each of the plurality of speakers are disposed at a respective non-zero distance from the user'"'"'s head, such that each of the plurality of speakers does not contact any surface of the user'"'"'s head, including the user'"'"'s ears;

a head pose sensor to collect head pose data of the user;

a head pose processor to determine a head pose of the user from the head pose data;

a room sensor to measure an acoustic property of a room in which the user is disposed; and

a spatialized audio processor to generate spatialized audio data based on the determined head pose of the user and the measured acoustic property of the room.

2. The spatialized audio system of claim 1, wherein the room sensor is a microphone to collect room sounds to measure the acoustic property of the room.

3. The spatialized audio system of claim 2, wherein the acoustic property of the room comprises a room sound.

4. The spatialized audio system of claim 1, wherein the room sensor is a camera to collect a room image to measure the acoustic property of the room.

5. The spatialized audio system of claim 4, wherein the acoustic property of the room comprises a room image.

6. A spatialized audio system, comprising:

a frame to be worn on a head of a user;

a plurality of speakers attached to the frame such that, when the frame is worn by the user, each of the plurality of speakers are disposed at a respective non-zero distance from the user'"'"'s head, such that each of the plurality of speakers does not contact any surface of the user'"'"'s head, including the user'"'"'s ears;

a head pose sensor to collect head pose data of the user;

a head pose processor to determine a head pose of the user from the head pose data; and

a spatialized audio processor to generate spatialized audio data based on the determined head pose of the user,

wherein the speakers generate first sounds corresponding to the generated spatialized audio data, and

wherein the spatialized audio processor receives first timing information at a first time to synchronize the first sounds with second sounds at the first time.

7. The system of claim 6, wherein the spatialized audio processor receives second timing information at a second time later than the first time to synchronize the first sound with the second sounds at the second time.

8. The system of claim 6, further comprising:

a cinema audio processor operatively coupled to the spatialized audio processor, wherein the cinema audio processor generates cinema audio data; and

a plurality of cinema speakers to generate the second sounds corresponding to the generated cinema audio data,

wherein the spatialized audio processor receives the first timing information from the cinema audio processor.

9. The system of claim 8, wherein the cinema audio processor transmits cinema information to the spatialized audio processor, and

wherein the cinema information comprises one of timing information, virtual object sound information, position information, orientation information, a hyperlink, or acoustic modeling information.

10. The system of claim 8, wherein the spatialized audio processor compensates for a latency difference between the spatialized audio processor and the cinema audio processor.

11. The system of claim 8, further comprising a user sensor to collect user information for transmission to the cinema audio processor.

12. The system of claim 11, wherein the cinema audio processor modifies the cinema audio data based on the user information such that the second sounds correspond to the modified cinema audio data.

13. The system of claim 12,

wherein the plurality of cinema speakers includes a subwoofer configured to generate low frequency sound,

wherein the user information includes low frequency audio data,

wherein the cinema audio processor modifies the cinema audio data based on the low frequency audio data, and

wherein the subwoofer generates the low frequency sound corresponding to the modified cinema audio data.

14. The system of claim 13, wherein the subwoofer is disposed adjacent a seat for the user or the subwoofer is worn by the user.

15. The system of claim 12, wherein the cinema audio data is organized in a branch structure, and wherein the modified cinema audio data has a modified branch structure smaller than the branch structure of the cinema audio data.

16. The system of claim 11, wherein the user information includes a sampled sound from the user, and wherein the modified cinema audio data incorporates sampled sound.

17. The system of claim 8, further comprising an application processor operatively coupled to the spatialized audio processor to provide source data from which the spatialized audio processor generates the spatialized audio data, and

wherein the application processor synchronizes the spatialized audio data with the cinema audio data.

18. The system of claim 6, wherein the frame focuses the first sounds to the user'"'"'s ears.

19. The system of claim 6, wherein the generated spatialized audio data includes position, orientation and volume data for each of a plurality of virtual audio sources.

20. The system of claim 6, wherein the plurality of speakers are removably attached to the frame with respective magnets.