System and method for providing simulated environment

• US 10,896,544 B2
• Filed: 09/27/2017
• Issued: 01/19/2021
• Est. Priority Date: 10/07/2016
• Status: Active Grant

• Current Assignee: HTC Corporation
• Original Assignee: HTC Corporation
• Inventors: Yang, Hung-Yi, Choi, Iok-Kan
• Primary Examiner(s): Yang, Andrew G
• Application Number:
  US15/717,927
• Publication Number:
  US 20180101990A1
• Time in Patent Office: 1,210 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: H04R 2227/003 Digital PA systems using, e.g. LAN or internet
• CPC Class Codes: H04R 27/00 Public address systems circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; monitoring or testing arrangements for public address systems H04R29/007; amplifiers H03F
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 7/303 Tracking of listener position or orientation
• CPC Class Codes: H04S 7/304 For headphones

Abstract

Present disclosure relates to a system for providing a simulated environment and a method thereof. The system comprises a first wearable device, a second wearable device and a computing device. The first wearable device is configured to output a scenario of the simulated environment and to output a first audio. The second wearable device is configured to collect an environmental sound around the second wearable device and send out the sound. The computing device is configured to merge the sound into the first audio according to an index and send the merged first audio to the first wearable device, wherein the index is determined by a relative distance and a relative direction between coordinates being assigned to the first wearable device and the second wearable device in the simulated environment.

Claims

1. A virtual reality system for providing a virtual reality simulated environment, comprising:

a computing device, configured to generate a scenario of the VR simulated environment and to connect to both a first head-mounted display (HMD) device and a second HMD device;

the first HMD device in a first space, configured to output the scenario of the VR simulated environment, to output a first audio, to collect a first environmental sound comprising a first user voice, to retrieve a first spatial vector with respect to the first space of the first HMD device, and to transmit the first environmental sound and the first spatial vector to the computing device;

the second HMD device in a second space that is different from the first space, configured to output the scenario of the VR simulated environment, to output a second audio, to collect a second environmental sound comprising a second user voice, to retrieve a second spatial vector with respect to the second space of the second HMD device, to transmit the second environmental sound and the second spatial vector to the computing device, wherein the first HMD device is spaced apart from the second HMD device by a real distance; and

the computing device, further configured to receive the first environmental sound comprising the first user voice, the first spatial vector with respect to the first space, the second environmental sound comprising the second user voice and the second spatial vector with respect to the second space to automatically assign coordinates of the first HMD device and the second HMD device in the VR simulated environment according to the received first spatial vector with respect to the first space and the received second spatial vector with respect to the second space,

to determine a relative distance and a relative direction between the coordinate assigned to the first HMD device in the VR simulated environment and the coordinate assigned to the second HMD device in the VR simulated environment, wherein the first user and the second user communicate to each other in the VR simulated environment through the first HMD device and the second HMD device,

to merge the second environmental sound into the first audio according to the relative distance and the relative direction in the VR simulated environment and send the merged first audio to the first HMD device, and to merge the first environmental sound into the second audio according to the relative distance and the relative direction in the VR simulated environment and send the merged second audio to the second HMD device, and

wherein; and

the real distance between the first HMD device in the first space and the second HMD device in the second space is inconsistent with the relative distance between the coordinates assigned to the first HMD device and the second HMD device in the VR simulated environment.

2. The virtual reality system of claim 1, wherein the second HMD device is further configured to retrieve a face characteristic and send the face characteristic to the computing device, and wherein the face characteristic is provided to illustrate a face pattern of an avatar in the scenario.

3. The virtual reality system of claim 1, wherein the second HMD device is further configured to scan an object to generate scan information and send the scan information to the computing device, wherein the object is selected by pointing the object through a targeting window in a display of the second HMD device, and

wherein the scan information is provided to illustrate a virtual object having a contour corresponding to the object in the scenario of the VR simulated environment.

4. The virtual reality system of claim 1, wherein the first HMD device further comprises a detector, which is configured to track a spatial vector of the second HMD device with respect to the first HMD device, wherein the spatial vector is sent to the computing device for determining the relative distance and the relative direction.

5. The virtual reality system of claim 4, wherein the detector comprises an optical detector, a resonance detector, a camera and a motion detector.

6. The virtual reality system of claim 1, further comprising:

a first detector, placed in the first space, provided to track the first spatial vector of the first HMD device with respect to the first space; and

a second detector, placed in the second space, provided to track the second spatial vector of the second HMD device with respect to the first space;

wherein the first spatial vector and the second spatial vector are sent to the computing device for determining the relative distance and the relative direction.

7. The virtual reality system of claim 6, wherein the first detector and the second detector comprise an optical detector, a resonance detector, a camera and a motion detector.

8. The virtual reality system of claim 6, wherein:

the first detector continuously tracks the first spatial vector of the first HMD device with respect to the first space, and sends the continuously tracked first spatial vector to the computing device;

the second detector continuously tracks the second spatial vector of the first HMD device with respect to the second space, and sends the continuously tracked second spatial vector to the computing device;

the computing device varies the coordinates of the first HMD device and the second HMD device in the VR simulated environment according to the received and continuously tracked first spatial vector and the second spatial vector, respectively;

the computing device determines variations in the relative distance and the relative direction between the coordinates in the VR simulated environment being assigned to the first HMD device and the second HMD device;

the computing device varies the merging of the second environmental sound into the first audio according to the variations in the relative distance and the relative direction, and sends the merged first audio to the first HMD device; and

the computer device varies the merging of the first environmental sound into the second audio according to the variations in the relative distance and the relative direction, and sends the merged second audio to the second HMD device.

9. The virtual reality system of claim 1, wherein the sound is merged into the first audio based on HRTF (Head Related Transfer Function).

10. A method for providing a virtual reality (VR) simulated environment, comprising:

generating, by a computing device, a scenario of the VR simulated environment and to connect to both a first head-mounted display (HMD) device in a first space and the second HMD device in a second space that is different from the first space wherein the first HMD device is spaced apart from the second HMD device by a real distance;

outputting, by the first HMD device, the scenario of the VR simulated environment and a first audio;

collecting, by the first HMD device, a first environmental sound comprising a first user voice;

retrieving, by the first HMD device, a first spatial vector with respect to the first space of the first HMD device;

transmitting, by the first HMD device, the first environmental sound and the first spatial vector to the computing device;

outputting, by the second HMD device, the scenario of the VR simulated environment, and a second audio;

collecting, by the second HMD device, a second environmental sound comprising a second user voice;

retrieving, by the second HMD device, a second spatial vector with respect to the second space of the second HMD device;

transmitting, by the second HMD device, the second environmental sound and the second spatial vector to the computing device;

receiving, by the computing device, the first environmental sound comprising the first user voice, the first spatial vector with respect to the first space, the second environmental sound comprising the second user voice and the second spatial vector with respect to the second space to automatically assign coordinates of the first HMD device and the second HMD device in the VR simulated environment according to the received first spatial vector with respect to the first space and the received second spatial vector with respect to the second space;

determining, by the computing device, a relative distance and a relative direction between the coordinates assigned to the first HMD device in the VR simulated environment and the coordinate assigned to the second HMD device in the VR simulated environment wherein the first user and the second user communicate to each other in the VR simulated environment through the first HMD device and the second HMD device;

merging, by the computing device, the second environmental sound into the first audio according to the relative distance and the relative direction in the VR simulated environment and sending the merged first audio to the first HMD device; and

merging, by the computing device, the first environmental sound into the second audio according to the relative distance and the relative direction in the VR simulated environment and send the merged second audio to the second HMD device;

wherein the real distance between the first HMD device in the first space and the second HMD device in the second space is inconsistent with the relative distance between the coordinates assigned to the first HMD device and the second HMD device in the VR simulated environment.

11. The method of claim 10, further comprising:

retrieving, by the second HMD device, a face characteristic;

sending, by the second HMD device, the face characteristic to the computing device; and

illustrating, by the computing device, a face pattern of an avatar in the VR simulated environment according to the face characteristic.

12. The method of claim 10, further comprising:

pointing, by the second HMD device, a physical object through a targeting window in a display of the second HMD device,

scanning, by the second HMD device, the physical object to generate scan information;

sending, by the second HMD device, scan information to the computing device; and

illustrating, by the computing device, a virtual object having a contour corresponding to the physical object in the scenario of the VR simulated environment according to the scan information, wherein the objected is selected by pointing the object through a targeting window in a display of the second HMD device.

13. The method of claim 10, further comprising:

tracking, by a detector configured in the first HMD device, a spatial vector of the second HMD device with respect to the first HMD device; and

sending, by the first HMD device, the spatial vector to the computing device for determining the relative distance and the relative direction.

14. The method of claim 13, wherein the detector comprises an optical detector, a resonance detector, a camera and a motion detector.

15. The method of claim 10, further comprising:

tracking, by the first detector placed in the first space, the first spatial vector of the first HMD device with respect to the first space;

tracking, by the second detector placed in the second space, the second spatial vector of the second HMD device with respect to the second space;

sending, by the first detector, the first spatial vector to the computing device;

sending, by the second detector, the second spatial vector to the computing device; and

determining, by the computing device, the relative distance and the relative direction according to the first spatial vector and the second spatial vector.

16. The method of claim 15, wherein the first detector and the second detector comprise an optical detector, a resonance detector, a camera and a motion detector.

17. The method of claim 10, wherein the sound is merged into the first audio based on HRTF (Head Related Transfer Function).