Augmented Reality System Using a Portable Device

• US 20130215230A1
• Filed: 02/22/2013
• Published: 08/22/2013
• Est. Priority Date: 02/22/2012
• Status: Abandoned Application

• Current Assignee: Dekko, Inc.
• Original Assignee: Dekko, Inc.
• Inventors: Miesnieks, Silka, Sietsma, John, Miesnieks, Matt, Baillot, Yohan, Gardeya, Marc, Naimark, Leonid, Maes, Anthony
• Application Number:
  US13/774,762
• Publication Number:
  US 20130215230A1
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/20 Image signal generators

Abstract

A system and a method are disclosed for capturing real world objects and reconstructing a three-dimensional representation of real world objects. The position of the viewing system relative to the three-dimensional representation is calculated using information from a camera and an inertial motion unit. The position of the viewing system and the three-dimensional representation allow the viewing system to move relative to the real objects and enables virtual content to be shown with collision and occlusion with real world objects.

Claims

1. A computer-implemented method for augmenting real-world objects with virtual content, comprising:

receiving a video feed including real-world objects;

constructing, from the video feed, a three-dimensional model including real-world objects;

determining, a position of a camera relative to the three-dimensional model;

placing a virtual object in the three-dimensional model of the real-world objects;

rendering for display, from a view of the position, unoccluded portions of the virtual object in the three-dimensional model;

overlaying the unoccluded portions of the virtual object on the video feed; and

displaying the video feed with the overlaid virtual object.

2. The computer-implemented method of claim 1, further comprising:

detecting motion of the camera; and

responsive to the detected motion of the camera, updating the camera position relative to the three-dimensional model and re-rendering the unoccluded portions of the virtual object from a view of the updated camera position.

3. The computer-implemented method of claim 2, wherein the updated rendered virtual objects remain fixed relative to the real-world objects.

4. The computer-implemented method of claim 1, further comprising:

moving the virtual object at least partially behind the real-world object from the perspective of the camera in the three-dimensional model;

re-rendering the unoccluded portions of the virtual object to exclude the portions of the virtual object that are at least partially behind the real-world object.

5. The computer-implemented method of claim 1, wherein determining the position of the camera is based on data from an inertial motion unit.

6. The computer-implemented method of claim 1, wherein determining the position of the camera is based on a simultaneous localization and mapping engine.

7. The computer-implemented method of claim 1, wherein determining the position of the camera is based on data from an inertial motion unit and a simultaneous localization and mapping engine.

8. The computer-implemented method of claim 7, wherein the position of the camera is determined by a combination of the position from the inertial motion unit and the simultaneous localization mapping engine.

9. The computer-implemented method of claim 1, wherein constructing the three-dimensional model includes identifying features from at least one frame of the video feed, wherein the features include at least one of edges, flat surfaces, and corners.

10. The computer-implemented method of claim 1, further comprising

applying a physics engine to determine collisions of the real-world objects with the virtual object.

11. A system for augmenting real-world objects with virtual content, comprising:

a processor configured to execute instructions;

a memory including instructions when executed by the processor cause the processor to;

receive a video feed including real-world objects;

construct, from the video feed, a three-dimensional model including real-world objects;

determine, a position of a camera relative to the three-dimensional model;

place a virtual object in the three-dimensional model of the real-world objects;

render for display, from a view of the position, unoccluded portions of the virtual object in the three-dimensional model;

overlay the unoccluded portions of the virtual object on the video feed; and

display the video feed with the overlaid virtual object.

12. The system of claim 11, wherein the instructions further cause the processor to:

detect motion of the camera; and

responsive to the detected motion of the camera, update the position relative to the three-dimensional model and re-render the unoccluded portions of the virtual object from a view of the updated camera position.

13. The system of claim 12, wherein the updated rendered virtual objects remain fixed relative to the real-world objects.

14. The system of claim 11, wherein the instructions further cause the processor to:

move the virtual object at least partially behind the real-world object from the perspective of the camera in the three-dimensional model;

re-render the unoccluded portions of the virtual object to exclude the portions of the virtual object that are at least partially behind the real-world object.

15. The system of claim 11, wherein determining the position of the camera is based on data from an inertial motion unit.

16. A computer-readable medium for augmenting real-world objects with virtual content, comprising instructions causing a processor to:

receive a video feed including real-world objects;

construct, from the video feed, a three-dimensional model including real-world objects;

determine, a position of a camera relative to the three-dimensional model;

place a virtual object in the three-dimensional model of the real-world objects;

render for display, from a view of the position, unoccluded portions of the virtual object in the three-dimensional model;

overlay the unoccluded portions of the virtual object on the video feed; and

display the video feed with the overlaid virtual object.

17. The computer-readable medium of claim 16, wherein the instructions further cause the processor to:

detect motion of the camera; and

responsive to the detected motion of the camera, update the position relative to the three-dimensional model and re-render the unoccluded portions of the virtual object from a view of the updated camera position.

18. The computer-readable medium of claim 17, wherein the updated rendered virtual objects remain fixed relative to the real-world objects.

19. The computer-readable medium of claim 16, wherein the instructions further cause the processor to:

move the virtual object at least partially behind the real-world object from the perspective of the camera in the three-dimensional model;

re-render the unoccluded portions of the virtual object to exclude the portions of the virtual object that are at least partially behind the real-world object.

20. The computer-readable medium of claim 16, wherein determining the position of the camera is based on data from an inertial motion unit.