Visually Convincing Depiction of Object Interactions in Augmented Reality Images

• US 20150130790A1
• Filed: 11/14/2013
• Published: 05/14/2015
• Est. Priority Date: 11/14/2013
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo of America, Inc. (Nintendo Company Limited)
• Inventors: RABIN, Steve, VASQUEZ, Joseph Gregory II
• Granted Patent: US 9,286,725 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2210/21 Collision detection, intersection

Abstract

The disclosure provide for improved depiction of interactions between virtual objects and physical objects in augmented reality (AR) images, and may include operations for: receiving a captured image of the real world physical structure, the captured image provided by a camera disposed in a housing; identifying, by image processing performed on the captured image using at least one processor arrangement, the at least one marker disposed at a location of the real world physical structure; determining aspects of user viewpoint of the housing relative to the physical structure at least in part in response to the identified at least one marker; using an at least partial three dimensional model of the real world physical structure to determine a view of interaction between the real world physical structure and the provided virtual object from the determined user viewpoint; and displaying, on a display, an augmented reality view of the captured image including the determined view of the interaction.

Claims

9. A computer-implemented method for providing a virtual reality experience interacting with a real world physical structure having at least one marker, the method comprising:

receiving a captured image of the real world physical structure, the captured image provided by a camera disposed in a housing;

identifying, by image processing performed on the captured image using at least one processor arrangement, the at least one marker disposed at a location of the real world physical structure;

determining aspects of user viewpoint of the housing relative to the physical structure at least in part in response to the identified at least one marker;

using an at least partial three dimensional model of the real world physical structure to determine a view of interaction between the real world physical structure and a virtual object from the determined user viewpoint; and

displaying, on a display, an augmented reality view of the captured image including the determined view of the interaction.

24. An apparatus for providing a virtual reality experience interacting with a real world physical structure having at least one marker, apparatus comprising:

an image interface configured to receive, from a physical camera disposed at a location of a housing, a captured image of the real world physical structure;

a memory storing data of at least a partial three dimensional model of the real world physical structure; and

at least one processing arrangement in communication with the memory and the image interface, the processing arrangement configured to provide a virtual object, to determine aspects of user viewpoint of the housing relative to the real world physical structure at least in part in response to said at least one marker, and to display an augmented reality view of the captured image on a display, the processing arrangement using the at least partial three dimensional model of the real world physical structure to display the provided virtual object interacting with the real world physical structure from the determined user viewpoint.

25. A non-transitory computer-readable storage medium having stored instructions which, when executed by at least one processor of a processing system, causes the processing system to perform operations for providing a virtual reality experience interacting with a real world physical structure having at least one marker, the operations comprising:

receiving a captured image of the real world physical structure, the captured image provided by a camera disposed in a housing;

identifying, by image processing performed on the captured image using at least one processor arrangement, the at least one marker disposed at a location of the real world physical structure;

determining aspects of user viewpoint of the housing relative to the physical structure at least in part in response to the identified at least one marker;

using an at least partial three dimensional model of the real world physical structure to determine a view of interaction between the real world physical structure and a virtual object from the determined user viewpoint; and

displaying an augmented reality view of the captured image including the determined view.

26. The non-transitory computer-readable storage medium according to claim 25, wherein the operations further comprise:

forming a virtual model object corresponding to the real world physical structure using data of the at least partial three dimensional model stored in a memory of the processing system, the forming including placing the virtual model object in a virtual space in accordance with a position and the orientation of the at least one marker in a real space in which the real world physical structure is located;

detecting said interaction between the virtual model object and the virtual object in the virtual space; and

generating the augmented reality view of the captured image, the generated augmented reality view showing one or more of occluding or colliding between the virtual object and the real world physical structure.

27. The non-transitory computer-readable storage medium according to claim 26, wherein the operations further comprise:

comparing dimensions of the marker as detected in the captured image with dimensions of the marker as previously recorded in the three dimensional model, wherein the three dimensional model includes relative dimensions of the marker and the real world physical structure; and

transforming the at least partial three dimensional model in accordance with the comparing, wherein the virtual model object is based upon the transformed at least partial three dimensional model.

28. The non-transitory computer-readable storage medium according to claim 26, wherein the operations further comprise:

identifying a plurality of markers in the captured image, wherein each of the plurality of markers is associated in the three dimensional model with a respective one of removable or adjustable elements of the real world physical structure;

choosing at least a subset of the plurality of removable or adjustable elements based upon the identified plurality of markers;

generating corresponding virtual elements and configurations for respective ones of the removable or adjustable elements in the subset; and

incorporating the generated corresponding virtual elements and configurations in the virtual model object.