Methods and systems for capturing and moving 3D models and true-scale metadata of real world objects

• US 9,443,353 B2
• Filed: 07/27/2012
• Issued: 09/13/2016
• Est. Priority Date: 12/01/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Dereck, Chelsea, Vaddadi, Sundeep, Chimalamarri, Krishnakanth S., Gandhi, Ketal V., Jayaswal, Anubha, Gupta, Prince, dos Santos, Jose Ricardo Leal
• Primary Examiner(s): Tseng, Charles
• Application Number:
  US13/560,678
• Publication Number:
  US 20130187905A1
• Time in Patent Office: 1,509 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2210/04 Architectural design, interior design
• CPC Class Codes: G06T 2219/2021 Shape modification

Abstract

In some embodiments, methods and systems are provided for assisting a user in visualizing how a modified real-world setting would appear. An imaging device may capture a plurality of images of one or more objects or settings. A three-dimensional model of each object or setting may be created based on the images. These models may then be used to create a realistic image of a modified setting. For example, an image may display a setting (e.g., a living room) with an additional object (e.g., a couch) in the setting. The image may be realistic, in that it may accurately represent dimensions of the object relative to dimensions in the setting. Because three-dimensional models were created for both the setting and object, a user may be able to manipulate the image to, e.g., re-position and/or re-orient the object within the setting and view the setting from different perspectives.

Claims

1. A method for presenting a representation of an augmented real-world setting on a mobile device, the method comprising:

accessing a plurality of images, wherein the plurality of images comprises a first set of images depicting one or more objects and a second set of images depicting a real-world setting;

detecting at least one real-world object within the first set of images, wherein the detecting is based on estimating a shape of the real-world object based on one or more image correspondences identified among the plurality of images;

detecting a real-world setting within the second set of images;

automatically creating an object model of the detected real-world object based on the first set of images, the object model having dimensions based at least in part on one or more estimated intrinsic parameters associated with an imaging-device used to capture the plurality of images;

creating a setting model of the setting based on the second set of images;

receiving information indicative of user input identifying a request that the setting be augmented;

creating a modified setting model comprising the object model and the setting model; and

presenting the modified setting model on the mobile device.

2. The method of claim 1, further comprising:

automatically extracting frames from a video of the real-world object, wherein the extracted frames comprise the first set of images of the real-world object.

3. The method of claim 1, wherein the object model comprises an estimated mass or density of the object.

4. The method of claim 3, further comprising:

receiving the estimated mass or density of the object from a user of the mobile device.

5. The method of claim 1, wherein a degree of detail included in the object model depends on how many images of the real-world object were accessed.

6. The method of claim 1, wherein a degree of detail included in the object model is determined at least partly based on information indicative of input by a user of the mobile device, the input by the user of the mobile device indicating a balance between the degree of detail and a processing time associated with creation of the object model of the object.

7. The method of claim 1, further comprising:

capturing the first set of images of the real-world object; and

capturing the second set of images of the real-world setting.

8. The method of claim 1, wherein the method is performed in its entirety on the mobile device.

9. The method of claim 8, wherein the modified setting model is created upon receiving model-modification input from a user via the mobile device.

10. The method of claim 8, wherein the object model is created upon receiving first model-creation information indicative of first model-creation user input, and the setting model is created upon receiving second model-creation information indicative of second model-creation user input.

11. A method for sharing models of real-world objects, the method comprising:

accessing a plurality of images, wherein the plurality of images comprises a first set of images depicting one or more objects and a second set of images depicting a real-world setting;

detecting at least one real-world object within the first set of images, wherein the detecting is based on estimating a shape of the real-world object based on one or more image correspondences identified among the plurality of images;

automatically creating a three-dimensional model of the detected real-world object based on the first set of images, the three-dimensional model having dimensions based at least in part on one or more estimated intrinsic parameters associated with an imaging-device used to capture the plurality of images;

creating a setting model of the setting based on the second set of images;

receiving information indicative of user input identifying a request that the three-dimensional model of the object be shared with another user;

creating a modified setting model comprising the three-dimensional model and the setting model;

transmitting the three-dimensional model over a network to another mobile device; and

presenting the modified setting model on the another mobile device.

12. The method of claim 11, wherein the three-dimensional model is transmitted over the network via a social networking website.

13. The method of claim 11, wherein the method for sharing models of real-world objects is performed in its entirety on a cellular phone.

14. A system for presenting a representation of an augmented real-world setting on a mobile device, the system comprising:

an imaging device configured to capture a first set of images of a real-world setting and a second set of images of a real-world object;

a feature identifier configured to detect at least one real-world object within the first set of images and a real-world setting within the second set of images, wherein the detecting is based on estimating a shape of the real-world object based on one or more image correspondences identified among the first set of images;

a model generator configured to automatically generate an object model of the detected real-world object based on the first set of images to generate a setting model of the real-world setting based on the second set of images, the object model having dimensions based at least in part on one or more estimated intrinsic parameters associated with the imaging device;

a user input component configured to receive information indicative of user input identifying a request that the setting be augmented;

a model interactor configured to create a modified setting mode comprising the object model and the setting model; and

a display configured to present the modified setting model.

15. The system of claim 14, wherein the model interactor comprises a setting-model augmentor configured to augment the setting model with the object.

16. The system of claim 14, wherein the setting model comprises models of a plurality of objects, one of the plurality of objects being the real-world object.

17. The system of claim 14, wherein the object model includes real-world dimensions of the object, and wherein the setting model includes real-world dimensions of the setting.

18. A system for presenting a representation of an augmented real-world setting on a mobile device, the system comprising:

means for accessing plurality of images, wherein the plurality of images comprises a first set of images depicting one or more objects and a second set of images depicting a real-world setting;

means for detecting at least one real-world object within the first set of images, wherein the detecting is based on estimating a shape of the real-world object based on one or more image correspondences identified among the plurality of images;

means for detecting a real-world setting within the second set of images;

means for automatically creating an object model of the detected real-world object based on the first set of images, the object model having dimensions based at least in part on one or more estimated intrinsic parameters associated with an imaging-device used to capture the plurality of images;

means for creating a setting model of the setting based on the second set of images;

means for receiving information indicative of user input identifying a request that the setting be augmented;

means for creating a modified setting model comprising the object model and the setting model; and

means for presenting the modified setting model.

19. The system of claim 18, wherein the modified setting model accurately reflects a ratio of dimensions of the real-world object compared to dimensions of the real-world setting.

20. The system of claim 18, wherein the setting model and the object model each include metadata indicating globally common-scale dimensions associated with the respective model.

21. The system of claim 18, further comprising:

means for matching a region from an image of the first set of images depicting the object to a region from an image of the second set of images depicting the object,

wherein the matched regions are predicted to correspond to a same real-world feature.

22. A non-transitory computer-readable medium containing a program for execution by a processor which executes steps of:

accessing a plurality of images, wherein the plurality of images comprises a first set of images depicting one or more objects and a second set of images depicting a real-world setting;

detecting at least one real-world object within the first set of images, wherein the detecting is based on estimating a shape of the real-world object based on one or more image correspondences identified among the plurality of images;

detecting a real-world setting within the second set of images;

automatically creating an object model of the detected real-world object based on the first-set of images, the object model having dimensions based at least in part on one or more estimated intrinsic parameters associated with an imaging-device used to capture the plurality of images;

creating a setting model of the setting based on the second set of images;

receiving information indicative of user input identifying a request that the setting be augmented;

creating a modified setting model comprising the object model and the setting model; and

presenting the modified setting model.

23. The non-transitory computer-readable medium of claim 22, wherein the modified setting model is created by modifying the setting model to replace an initial object present in the real-world setting with the object.

24. The non-transitory computer-readable medium of claim 22, wherein the real-world object is an object in the real-world setting, and wherein creating the modified setting model comprises moving the object within the setting.

25. The non-transitory computer-readable medium of claim 22, wherein the real-world object is an object in the real-world setting, and wherein creating the modified setting model comprises modifying the setting model to remove the object from the setting.

26. A method for presenting a representation of an augmented real-world setting on a mobile device, the method comprising:

accessing an object model of a detected real-world object based on a first set of images depicting the object, the object model having been automatically created based on the first set of images of the object, the object model having dimensions based at least in part on one or more estimated intrinsic parameters associated with an imaging-device used to capture the first set of images, and wherein detection of the real-world object is based on estimating a shape of the real-world object based on one or more image correspondences identified among the first set of images;

accessing a setting model of a real-world setting based on a second set of images depicting the setting, the setting model having been created based on the second set of images of the setting;

receiving information indicative of user input identifying a request that the setting be augmented;

creating a modified setting model based on the setting model and the object model; and

presenting the modified setting model on the mobile device.

27. The method of claim 26, wherein the first set of images depicting the object were captured by a user of another mobile device.

28. The method of claim 26, wherein the object model is accessed via a website.

29. The method of claim 26, wherein the object model comprises a model shared by a user of another device with the user of the mobile device.