Real time texture mapping for augmented reality system

• US 9,659,381 B2
• Filed: 01/26/2015
• Issued: 05/23/2017
• Est. Priority Date: 01/26/2015
• Status: Active Grant

• Current Assignee: META Platforms Technologies LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: DAQRI, LLC
• Inventors: Mullins, Brian, Kammerait, Matthew, Sararu, Mark Anthony, Krage, Andrew Thomas, Khachaturyan, Gregory
• Primary Examiner(s): Chu, David H
• Application Number:
  US14/605,249
• Publication Number:
  US 20160217590A1
• Time in Patent Office: 848 Days
• Field of Search: None
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06V 10/54 relating to texture
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A system and method for real-time texture mapping for an augmented reality system are described. A viewing device includes an optical sensor to capture an image of a real-world object. A texture extraction module extracts a texture of the image of the real-world object. A recognition module identifies the real-world object based on the captured image. A texture mapping module retrieves a virtual object corresponding to the identified real-world object, maps the texture to the virtual object, dynamically updates the texture to the virtual object in real time, and generates a visualization of the virtual object in a display of the viewing device.

Claims

1. A method comprising:

capturing, by a viewing device, an image of a real-world object, the image captured using an optical sensor of the viewing device;

identifying, from the image of the real-world object, a first predefined area of the real-world object captured in the image;

identifying the real-world object based on the captured image;

retrieving a virtual object corresponding to the real-world object;

generating a visualization of the virtual object on a display of the viewing device;

after generating the visualization of the virtual object on the display of the viewing device, capturing a subsequent image of the real-world object;

determining, from the subsequent image, that a texture has been added within the first predefined area of the real-world object;

identifying, based on the texture within the first predefined area of the real-world object, a virtual object behavior mapped to the texture being within the first predefined area of the real-world object, the virtual object behavior identifying a predetermined behavior to be exhibited by the virtual object; and

updating the visualization of the virtual object on the display to cause the virtual object to exhibit the predetermined behavior identified by the virtual object behavior.

2. The method of claim 1, further comprising:

extracting the texture from the first predefined area in the image of the real-world object.

3. The method of claim 2, further comprising:

mapping the texture of the first predefined area in the image to a corresponding area of the virtual object.

4. The method of claim 3, further comprising:

mapping the texture of the first predefined area to a corresponding surface in a three-dimensional model of the virtual object.

5. The method of claim 1, further comprising:

mapping the texture to a portion of the virtual content corresponding to the first predetermined area.

6. The method of claim 5, further comprising:

dynamically updating in real time the visualization of a portion of the virtual content corresponding to the first predetermined area to reflect the texture added to the first predetermined area.

7. The method of claim 1, further comprising:

extracting the texture on a periodic basis; and

updating a mapping of the texture to the virtual object on the periodic basis.

8. The method of claim 1, wherein

the texture comprises a color and a visual pattern in the image of the real-world object.

9. The method of claim 1, further comprising rendering the visualization of the virtual content in a transparent display of the viewing device, the virtual content visually perceived as an overlay to the real-world object.

10. The method of claim 1, wherein the predetermined behavior is an animated movement to be exhibited by the virtual object.

11. The method of claim 10, wherein the predetermined behavior further includes sound to be presented while the virtual object is exhibiting the animated movement.

12. A non-transitory machine-readable medium comprising instructions that, when executed by one or more computer processors of a viewing device, cause the viewing device to perform operations comprising:

capturing an image of a real-world object, the image captured using an optical sensor of the viewing device;

identifying, from the image of the real-world object, a first predefined area of the real-world object captured in the image;

identifying the real-world object based on the captured image;

retrieving a virtual object corresponding to the real-world object;

generating a visualization of the virtual object on a display of the viewing device;

after generating the visualization of the virtual object on the display of the viewing device, capturing a subsequent image of the real-world object;

determining, from the subsequent image, that a texture has been added within the first predefined area of the real-world object;

identifying, based on the texture within the first predefined area of the real-world object, a virtual object behavior mapped to the texture being within the first predefined area of the real-world object, the virtual object behavior identifying a predetermined behavior to be exhibited by the virtual object; and

updating the visualization of the virtual object on the display to cause the virtual object to exhibit the predetermined behavior identified by the virtual object behavior.

13. A viewing device comprising:

one or more computer processors; and

one or more computer-readable mediums storing instructions that, when executed by the one or more computer processors, cause the viewing device to perform operations comprising;

capturing an image of a real-world object, the image captured using an optical sensor of the viewing device;

identifying, from the image of the real-world object, a first predefined area of the real-world object captured in the image;

identifying the real-world object based on the captured image;

retrieving a virtual object corresponding to the real-world object;

generating a visualization of the virtual object on a display of the viewing device;

after generating the visualization of the virtual object on the display of the viewing device, capturing a subsequent image of the real-world object;

determining, from the subsequent image, that a texture has been added within the first predefined area of the real-world object;

identifying, based on the texture within the first predefined area of the real-world object, a virtual object behavior mapped to the texture being within the first predefined area of the real-world object, the virtual object behavior identifying a predetermined behavior to be exhibited by the virtual object; and

updating the visualization of the virtual object on the display to cause the virtual object to exhibit the predetermined behavior identified by the virtual object behavior.

14. The viewing device of claim 13, the operations further comprising:

extracting the texture from the first predefined area in the image of the real-world object.

15. The viewing device of claim 14, the operations further comprising:

mapping the texture of the first predefined area in the image to a corresponding area of the virtual object.

16. The viewing device of claim 15, the operations further comprising:

mapping the texture of the first predefined area to a corresponding surface in a three-dimensional model of the virtual object.

17. The viewing device of claim 13, the operations further comprising:

mapping the texture to a portion of the virtual content corresponding to the first predetermined area.

18. The viewing device of claim 17, the operations further comprising:

dynamically updating in real time the visualization of a portion of the virtual content corresponding to the first predetermined area to reflect the texture added to the first predetermined area.

19. The viewing device of claim 13, the operations further comprising:

extracting the texture on a periodic basis; and

updating a mapping of the texture to the virtual object on the periodic basis.

20. The viewing device of claim 13, wherein the texture comprises a color and a visual pattern in the image of the real-world object.

21. The viewing device of claim 13, further comprising rendering the visualization of the virtual content in a transparent display of the viewing device, the virtual content visually perceived as an overlay to the real-world object.

22. The viewing device of claim 13, wherein the predetermined behavior is an animated movement to be exhibited by the virtual object.

23. The viewing device of claim 22, wherein the predetermined behavior further includes sound to be presented while the virtual object is exhibiting the animated movement.