SYSTEM AND METHOD FOR GENERATING A MIXED REALITY ENVIRONMENT

• US 20100103196A1
• Filed: 10/27/2009
• Published: 04/29/2010
• Est. Priority Date: 10/27/2008
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Oskiper, Targay, Kim, Janet, Samarasekera, Supun, Kumar, Rakesh, Zhu, Zhiwei, Naroditsky, Oleg
• Granted Patent: US 9,600,067 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A system and method for generating a mixed-reality environment is provided. The system and method provides a user-worn sub-system communicatively connected to a synthetic object computer module. The user-worn sub-system may utilize a plurality of user-worn sensors to capture and process data regarding a user'"'"'s pose and location. The synthetic object computer module may generate and provide to the user-worn sub-system synthetic objects based information defining a user'"'"'s real world life scene or environment indicating a user'"'"'s pose and location. The synthetic objects may then be rendered on a user-worn display, thereby inserting the synthetic objects into a user'"'"'s field of view. Rendering the synthetic objects on the user-worn display creates the virtual effect for the user that the synthetic objects are present in the real world.

Claims

1. A method of rendering a synthetic object onto a user-worn display showing the user'"'"'s view of a real world scene, the method comprising the steps of:

capturing real world scene information using one or more user-worn sensors;

producing a pose estimation data set and a depth data set based on at least a portion of the captured real world scene information;

receiving the synthetic object generated in accordance with the pose estimation data set and the depth data set; and

rendering the synthetic object onto the user-worn display in accordance with the pose estimation data set and the depth data set, thereby integrating the synthetic object into the user'"'"'s view of the real world scene.

2. The method of claim 1, wherein the real world scene information is captured only by the one or more user-worn sensors.

3. The method of claim 1, wherein the one or more user-worn sensors comprises a first video camera and a second video camera in stereo configuration.

4. The method of claim 1, wherein one of the one or more user-worn sensors comprises a LIDAR sensor.

5. The method of claim 1, wherein one of the one or more user-worn sensors comprises an inertial measurement unit.

6. The method of claim 1, wherein one of the one or more user-worn sensors comprises a global positioning system sensor.

7. The method of claim 1, wherein the one or more sensors comprises an inertial measurement unit and a video sensor.

8. The method of claim 7, further comprising:

processing a portion of the pose estimation data set based on at least a portion of the real world scene information captured by the inertial measurement unit and the video sensor.

9. The method of claim 8, further comprising:

receiving a first video frame from the video sensor;

receiving the inertial measurement unit dataset from the inertial measurement unit; and

predicting the portion of the pose estimation dataset based on the inertial measurement unit dataset and the first video frame.

10. The method of claim 1, further comprising calculating a user-controlled device pose as part of the pose estimation data set.

11. The method of claim 10, wherein calculating the user-controlled device pose comprises:

receiving a user-controlled device data set from one of the one or more user-worn sensors located on a user-controlled device; and

calculating the user-controlled device pose based on the user-controlled device data set and data received from a communicatively connected landmark database.

12. The method of claim 3, wherein a user-controlled device includes a marker and the user-controlled device pose comprises:

receiving a first video frame of the marker from a first video camera;

receiving a second video frame of the marker from a second video camera; and

utilizing the first video frame and the second video frame to triangulate the location of the marker, thereby calculating the user-controlled device pose.

13. The method of claim 10, wherein calculating the user-controlled device pose comprises:

estimating a relative orientation between the user and the user-controlled device through the use of at least one user-worn sensor; and

calculating the user-controlled device pose based on the relative orientation and information received from a landmark database.

14. The method of claim 1, wherein the step of producing the pose estimation data set utilizes a landmark database and the real world scene information.

15. The method of claim 1, wherein the captured real world scene information is used to update a landmark database.

16. A method of rendering a synthetic object onto a first user-worn display showing the first user'"'"'s view of a real world scene and the synthetic object onto a second user-worn showing the second user'"'"'s view of the real world scene, the method comprising the steps of:

capturing a first set of real world scene information using one or more sensors worn by the first user;

capturing a second set of real world scene information using one or more sensors worn by the second user;

producing a first pose estimation data set and a first depth data set based on at least a portion of the first set of real world scene information;

producing a second pose estimation data set and a second depth data set based on at least a portion of the second set of real world scene information;

receiving the synthetic object generated in accordance with the first pose estimation data set and the first depth data set;

receiving the synthetic object generated in accordance with the second pose estimation data set and the second depth data set;

rendering the synthetic object onto the first user-worn display in accordance with the first pose estimation data set and the first depth data set, thereby integrating the synthetic object into the first user'"'"'s perception of the real world scene; and

rendering the synthetic object onto the second user-worn display in accordance with the second pose estimation data set and the second depth data set, thereby integrating the synthetic object into the second user'"'"'s perception of the real world scene, wherein the synthetic object appears consistent within the first user'"'"'s perception of the real world scene and the second user'"'"'s perception of the real world scene

17. A system for rendering a synthetic object onto a user-worn display showing the user'"'"'s view of a real world scene, comprising:

a user-worn computer module configured to;

capture real world scene information using at least one user-worn sensor;

producing a pose estimation data set and depth data set based on at least a portion of the real world scene information;

receive the synthetic object generated in accordance with the pose estimation data set and the depth data set; and

render the synthetic object onto the user-worn display, in accordance with the pose estimation data set and the depth data set, thereby integrating the synthetic object into the user'"'"'s view of the real world scene;

a synthetic object computer module configured to;

retrieve the synthetic object from a database in accordance with the pose estimation data set and the depth data set; and

transmit the synthetic object to the user-worn computer module.

18. The system of claim 17, wherein the real world information is captured only be the one or more user-worn sensors.

19. The system of claim 17, wherein the one or more user-worn sensors comprises a first video camera and a second video camera in stereo configuration.

20. The system of claim 17, wherein one of the one or more user-worn sensors comprises a LIDAR sensor.

21. The system of claim 17, wherein one of the one or more user-worn sensors comprises an inertial measurement unit.

22. The system of claim 17, wherein one of the one or more user-worn sensors comprises a global positioning system sensor.

23. The system of claim 17, wherein the one or more sensors comprises an inertial measurement unit and a video sensor.

24. The system of claim 23, wherein the user-worn computer module is further configured to:

produce a portion of the pose estimation data set based on at least a portion of the real world scene information captured by the inertial measurement unit and the video sensor.

25. The system of claim 24, wherein the user-worn computer module is further configured to:

receive a first video frame from the video sensor;

receive the inertial measurement unit dataset from the inertial measurement unit; and

predict the portion of the pose estimation dataset base on the inertial measurement unit dataset and the first video frame.

26. The system of claim 17, wherein the user-worn computer module is further configured to calculate a user-controlled device pose as part of the pose estimation data set.

27. The system of claim 26, wherein the user-worn computer module is further configured to:

receive a user-controlled device data set from one of the one or more user-worn sensors located on a user-controlled device, and

calculate a user-controlled device pose based on the user-controlled device data set and data received from a communicatively connected landmark database.

28. The system of claim 18, further comprising:

a user-controlled device which includes a marker; and

the user-worn computer module is further configured to;

receive a first video frame of the marker from a first video camera,

receive a second video frame of the marker from a second video camera, and

utilize the first video frame and the second video frame to triangulate the location of the marker, thereby calculating the user-controlled device pose.

29. The system of claim 28, wherein the user-worn computer module is further configured to:

estimate a relative orientation between the user and the user-controlled device through the use of at least one user-worn sensor, and

calculate the user-controlled device pose based on the relative orientation and information received from a landmark database.

30. The system of claim 17, wherein the user-worn computer module is further configured to produce the pose estimation dataset utilizes a landmark database and the real world scene information.

31. The system of claim 17, wherein the captured real world scene information is used to update a landmark database.

32. A system for rendering a synthetic object onto a first user-worn display showing a first user'"'"'s view of a real world scene and the synthetic object onto a second user-worn display showing a second user'"'"'s view of a real world scene, comprising:

a first user-worn computer module configured to;

capture a first set of real world scene information using at least one user-worn sensor;

producing a first pose estimation data set and a first depth data set based on at least a portion of the first set of real world scene information;

receive the synthetic object generated in accordance with the first pose estimation data set and the first depth data set; and

render the synthetic object onto the first user-worn display, in accordance with the first pose estimation data set and the first depth data set, thereby integrating the synthetic object into the first user'"'"'s view of the real world scene;

a second user-worn computer module configured to;

capture a second set of real world scene information using at least one user-worn sensor;

producing a second pose estimation data set and a second depth data set based on at least a portion of the second set of real world scene information;

receive the synthetic object generated in accordance with the second pose estimation data set and the second depth data set; and

render the synthetic object onto the second user-worn display, in accordance with the second pose estimation data set and the second depth data set, thereby integrating the second embodiment of the synthetic object into the second user'"'"'s view of the real world scene, wherein the synthetic object appears consistent within the first user'"'"'s perception of the real world scene and the second user'"'"'s perception of the real world scene;

a synthetic object computer module configured to;

retrieve first embodiment of the synthetic object and the second embodiment of the synthetic object from a database in accordance with the pose estimation data set and the depth data set; and

transmit the first embodiment of the synthetic object to the first user-worn computer module and the second embodiment of the synthetic object to the second user-worm computer module.