Method and apparatus for visualizing 2D product images integrated in a real-world environment

• US 20120120113A1
• Filed: 11/15/2010
• Published: 05/17/2012
• Est. Priority Date: 11/15/2010
• Status: Abandoned Application

• Current Assignee: Eduardo Hueso
• Original Assignee: Eduardo Hueso
• Inventors: Hueso, Eduardo
• Application Number:
  US12/927,401
• Publication Number:
  US 20120120113A1
• Time in Patent Office: Days
• US Class Current: 345/672
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A software application, which uses a portable device and augmented reality techniques to reconstruct a 2D image of the user'"'"'s environment augmented with a 2D element representing an object or product which looks like part of the environment image.

Claims

1. A method comprising:

receiving in a hand-held augmented reality device a set of product images from an online database;

isolating the consumer product from the background in a product image;

capturing an environment image using a camera of the hand-held augmented reality device;

selecting the product image that better matches a desired perspective;

synthesizing an augmented image with product images embedded in the environment image using the processing unit of the hand-held augmented reality device;

displaying the augmented image in real-time on a display of the hand-held augmented reality device;

allowing a user to manually position the product image within the augmented image;

allowing the user to manually re-size the product image within the augmented image; and

allowing the user to manually orient the product image about a normal axis a plane of the image within the augmented image.

2. The method in claim 1 further comprising:

rendering the augmented image by projecting a product billboard onto an environment image;

3. The method in claim 2 further comprising:

allowing the user to manually orient a product billboard by specifying the product billboard'"'"'s rotation in 3 cartesian axes;

allowing the user to manually position a product billboard by specifying the product billboard'"'"'s position in 3 cartesian axes; and

allowing the user to manually re-size a product billboard by specifying the product billboard'"'"'s dimensions in cartesian axes.

4. The method in claim 1 further comprising:

allowing the user to specify a set of three-dimensional features that are used to construct a three-dimensional environment model;

allowing the user to align the three-dimensional features of the three-dimensional environment model with an environment image;

employing sensor data to determine position and or altitude of the device'"'"'s camera in respect to the device'"'"'s camera'"'"'s environment;

registering the three-dimensional environment model with the environment image

5. The method in claim 4 further comprising:

receiving a consumer product'"'"'s description;

constructing an approximate three-dimensional product model from the description;

extracting the camera position and altitude of the camera used to photograph a consumer product from a product image;

automatically select a product image that best matches a desired perspective;

rendering the augmented image by projecting a product billboard created from the selected product image onto an environment image;

allowing the user to specify the location and orientation of a three-dimensional product model within a three-dimensional environment model;

automatically determining the position and orientation of the product billboard so that the product billboard'"'"'s best represents visually the three-dimensional product model;

automatically determining the scale of the product billboard so that the product billboard'"'"'s scale in respect to the environment image reflects the absolute scale of the real-world product and the user defined placement of the product model within the three-dimensional environment model;

6. The method in claim 2 further comprising:

allowing the user to specify an initial scale and orientation of the product billboard employing sensor data to determine altitude changes on the device'"'"'s camera;

automatically adjusting the placement and orientation of the product billboard in order to keep the product billboard registered with a changing environment image

7. A hand-held computing device comprising:

a data interface receiving a set of product images from an online database;

an imaging manipulation module isolating the consumer product from the background in a product image;

a camera capturing an environment image;

the user interface for selecting a product image that better matches a desired perspective;

a processing unit for synthesizing an augmented image with the product image embedded in the environment image using a processing unit;

a display to show the augmented image in real-time; and

wherein the user interface for allowing the user to manually position the product image within the augmented image, to re-size the product image within the augmented image, and to manually orient the product image about the image plane'"'"'s normal axis within the augmented image.

8. The hand-held computing device of claim 7 wherein the display for displaying the augmented image by projecting a product billboard onto an environment image;

9. The hand-held computing device of claim 8 wherein the user interface allows the user to manually orient a product billboard by specifying the product billboard'"'"'s rotation in 3 cartesian axes, manually position a product billboard by specifying the product billboard'"'"'s position in 3 cartesian axes;

and manually re-size a product billboard by specifying the product billboard'"'"'s dimensions in cartesian axes.

10. The hand-held computing device of claim 7 further comprising:

sensors to determine position and or altitude of the device'"'"'s camera in respect to the device'"'"'s camera'"'"'s environment.

11. The hand-held computing device of claim 10, wherein

the user interface allows the user to specify a set of three-dimensional features that are used to construct a three-dimensional environment model and to align the three-dimensional features of the three-dimensional environment model with an environment image;

and

the processing unit to register the three-dimensional environment model with the environment image.

12. The hand-held computing device of claim 10, wherein

the data interface receives a consumer product'"'"'s description;

the processing unit constructs an approximate three-dimensional product model from the description and automatically selects a product image that best matches a desired perspective;

the display renders the augmented image by projecting a product billboard created from the selected product image onto an environment image;

the user interface allows the user to specify the location and orientation of a three-dimensional product model within a three-dimensional environment model; and

the processing unit automatically determines the position and orientation of the product billboard so that the product billboard best represents visually the three-dimensional product model and determines the scale of the product billboard so that the product billboard'"'"'s scale in respect to the environment image reflects the absolute scale of the real-world product and the user defined placement of the product model within the three-dimensional environment model.

13. The hand-held computing device of claim 10, wherein:

the user interface allows the user to specify an initial scale and orientation of the product billboard; and

the processing unit automatically adjusts the placement and orientation of the product billboard in order to keep the product billboard registered with a changing environment image

14. A tangible machine-readable medium having a set of instructions detailing a method stored thereon that when executed by one or more processors cause the one or more processors to perform the method, the method comprising:

receiving in a hand-held augmented reality device a set of product images from an online database;

isolating the consumer product from the background in a product image;

capturing an environment image using a camera of the hand-held augmented reality device;

selecting a product image that better matches a desired perspective;

synthesizing an augmented image with product images embedded in the environment image using the processing unit of the hand-held augmented reality device;

displaying the augmented image in real-time on a display of the hand-held augmented reality device;

allowing a user to manually position the product image within the augmented image;

allowing the user to manually re-size the product image within the augmented image; and

allowing the user to manually orient the product image about a normal axis a plane of the image within the augmented image.

15. The tangible machine-readable medium of claim 14, further comprising:

rendering the augmented image by projecting a product billboard onto an environment image;

16. The tangible machine-readable medium of claim 15, further comprising:

allowing the user to manually orient a product billboard by specifying the product billboard'"'"'s rotation in 3 cartesian axes;

allowing the user to manually position a product billboard by specifying the product billboard'"'"'s position in 3 cartesian axes; and

allowing the user to manually re-size a product billboard by specifying the product billboard'"'"'s dimensions in cartesian axes.

17. The tangible machine-readable medium of claim 14, further comprising:

allowing the user to specify a set of three-dimensional features that are used to construct a three-dimensional environment model;

allowing the user to align the three-dimensional features of the three-dimensional environment model with an environment image;

employing sensor data to determine position and or altitude of the device'"'"'s camera in respect to the device'"'"'s camera'"'"'s environment;

registering the three-dimensional environment model with the environment image

18. The tangible machine-readable medium of claim 17, further comprising:

receiving a consumer product'"'"'s description;

constructing an approximate three-dimensional product model from the description;

extracting the camera position and altitude of the camera used to photograph a consumer product from a product image;

automatically select a product image that best matches a desired perspective;

rendering the augmented image by projecting a product billboard created from the selected product image onto an environment image;

allowing the user to specify the location and orientation of a three-dimensional product model within a three-dimensional environment model;

automatically determining the position and orientation of the product billboard so that the product billboard'"'"'s best represents visually the three-dimensional product model;

automatically determining the scale of the product billboard so that the product billboard'"'"'s scale in respect to the environment image reflects the absolute scale of the real-world product and the user defined placement of the product model within the three-dimensional environment model;

19. The tangible machine-readable medium of claim 15, further comprising:

allowing the user to specify an initial scale and orientation of the product billboard employing sensor data to determine altitude changes on the device'"'"'s camera;

automatically adjusting the placement and orientation of the product billboard in order to keep the product billboard registered with a changing environment image