Augmented reality surface painting

• US 10,068,547 B2
• Filed: 06/29/2012
• Issued: 09/04/2018
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Sobeski, David, Leake, Brian, Mitchell, Kenny
• Primary Examiner(s): Wang, Samantha (Yuehan)
• Application Number:
  US13/538,644
• Publication Number:
  US 20140002472A1
• Time in Patent Office: 2,258 Days
• Field of Search: 345589, 345633, 345582
• CPC Class Codes: G09G 5/026 Control of mixing and/or overlay of colours in general G09G5/022 and G09G5/024 take precedence

Abstract

Techniques for simulating interactions using an augmented reality device are described. Embodiments receive a request to paint over portions of a visual scene. Here, the visual scene is captured using one or more camera devices of the augmented reality device and is presented on a display of the augmented reality device. A first object in the visual scene corresponding to the first location is identified. Additionally, embodiments paint at least a portion of the first object as specified by the received request, and render a series of frames depicting the first object, such that the painted at least a portion of the first object is shown as painted when viewed from different perspectives using the augmented reality device.

Claims

1. A computer-implemented method to maintain conformity of a virtual painting onto a physical object captured and portrayed via an augmented reality device, even as the augmented reality device physically moves relative to the physical object in a physical environment over time, the computer-implemented method comprising:

presenting, via a display of the augmented reality device, the physical object within a visual scene captured in the physical environment using one or more camera devices of the augmented reality device, the physical object having an area of a first color;

receiving a request to at least partially fill the area using virtual paint of a second color different from the first color;

responsive to the request, identifying the physical object depicted by the visual scene, based on at least one of;

(i) one or more edges determined for the physical object from analyzing the visual scene; and

(ii) predefined geometrical data specifying geometrical characteristics of a specified type of the physical object;

determining one or more illumination characteristics of the physical environment;

generating, based on an initial image of the identified physical object in the visual scene and based further on the one or more illumination characteristics of the physical environment, an initial virtual painting projected onto the area of the physical object in the initial image and as viewed from a first physical position of the augmented reality device relative to the physical object in the physical environment; and

responsive to the augmented reality device being moved from the first physical position to a second physical position relative to the physical object in the physical environment, the second physical position being distinct from the first physical position, generating, based on an updated image of the identified physical object in the visual scene and based further on the one or more illumination characteristics of the physical environment, an updated virtual painting projected onto the area of the physical object in the updated image and as viewed from the second physical position of the augmented reality device, wherein the updated virtual painting is visually distinct from the initial virtual painting in terms of at least one visual aspect selected from size, shape, orientation, pattern, and color;

wherein the updated virtual painting is generated in order to maintain conformity of the virtual painting onto the physical object even as the augmented reality device physically moves relative to the physical object in the physical environment over time, wherein the initial virtual painting and the updated virtual painting are output.

2. The computer-implemented method of claim 1, wherein identifying the physical object within the visual scene comprises:

analyzing the visual scene to determine the one or more edges of the physical object within the visual scene, wherein the physical object is detected based on the determined one or more edges.

3. The computer-implemented method of claim 1, wherein identifying the physical object within the visual scene comprises:

retrieving the predefined geometrical data, which specifies geometrical characteristics of the specified type of the physical object; and

analyzing the visual scene using the predefined geometrical data in order to determine that the physical object is an instance of one of the specified type of physical object within the visual scene.

4. The computer-implemented method of claim 1, wherein the area is of a specific shape.

5. The computer-implemented method of claim 1,

wherein the one or more illumination characteristics include at least one of a position of a light source within the physical environment, an angle of the light source, an indication of whether the light source is omnidirectional, a color of the light source, an intensity of the light source, and a reflectivity value of the physical object.

6. The computer-implemented method of claim 1, wherein the request specifies to remove content at a first location within the visual scene, wherein adjusting the visual scene by painting at least a portion of the identified first object comprises:

removing the portion of the identified physical object from the visual scene as portrayed by the augmented reality device; and

replacing the removed portion with replacement content that is displayed by the augmented reality device, wherein the replacement content is determined based at least in part on content within the visual scene surrounding the removed portion of the identified physical object in the physical environment.

7. The computer-implemented method of claim 1, further comprising:

upon detecting that at least a predefined threshold of the physical object has been painted, rendering a series of frames depicting the physical object as an animated virtual object within the visual scene.

8. The computer-implemented method of claim 1, wherein the area is at least partially filled using the virtual paint of the second color as viewed from the augmented reality device, wherein the area is of a specific shape on one or more surfaces of the physical object, wherein the computer-implemented method further comprises:

generating a plurality of virtual objects corresponding to the physical object and portrayed at distinct locations separate from a physical location of the physical object in the physical environment;

wherein each of the plurality of virtual objects is visually distinct from the physical object in terms of at least one visual aspect selected from size, shape, orientation, pattern, and color, wherein each of the plurality of virtual objects is visually distinct from one another in terms of at least one visual aspect selected from size, shape, orientation, pattern, and color;

wherein each of the plurality of virtual objects is updated in order to maintain conformity of the respective virtual object to the physical environment as portrayed by the augmented reality device, in order to simulate actual presence of the respective virtual object in the physical environment;

wherein the presence of the respective virtual object is simulated at a respective physical location, in the physical environment, that is not occupied by any physical object;

wherein the conformity of the respective virtual object to the physical environment is maintained even as the augmented reality device physically moves relative to the respective physical location over time, wherein the respective updated virtual object is output via the augmented reality device.

9. The computer-implemented method of claim 8, wherein the area is at least substantially filled using the virtual paint of the second color as viewed from the augmented reality device, wherein the computer-implemented method further comprises:

rendering a series of frames depicting the physical object, wherein the area remains painted in the rendered series of frames when the physical object is viewed from different perspectives as a physical position of the augmented reality device changes relative to the physical object, wherein rendering the series of frames comprises;

inserting the initial virtual painting into a first frame of the series of frames, when the physical object is viewed from the first physical position of the augmented reality device relative to the physical object; and

inserting the updated virtual painting into a second frame of the series of frames, when the physical object is viewed from the second physical position of the augmented reality device relative to the physical object.

10. The computer-implemented method of claim 9, wherein the area is completely filled using the virtual paint of the second color as viewed from the augmented reality device, wherein the updated virtual painting is visually distinct from the initial virtual painting in terms of at least two visual aspects selected from size, shape, orientation, pattern, and color, wherein the specified type comprises a first of a plurality of specified types of physical objects, wherein the augmented reality device is configured to identify the physical object within the visual scene by:

in a first instance, analyzing the visual scene based on the one or more edges determined for the physical object, wherein the physical object is detected based on the determined one or more edges; and

in a second instance;

(i) retrieving predefined geometrical data specifying geometrical characteristics of the plurality of specified types of physical objects, including the predefined geometrical data specifying geometrical characteristics of the specified type of the physical object; and

(ii) analyzing the visual scene based on the retrieved, predefined geometrical data in order to determine that the physical object corresponds to the first specified type.

11. The computer-implemented method of claim 10, wherein the updated virtual painting is visually distinct from the initial virtual painting in terms of at least three visual aspects selected from size, shape, orientation, pattern, and color.

12. The computer-implemented method of claim 11, wherein the updated virtual painting is visually distinct from the initial virtual painting in terms of at least four visual aspects selected from size, shape, orientation, pattern, and color, wherein the received request further specifies:

(i) a desired texture for the specific shape and (ii) a desired pattern for the specific shape, wherein each of the initial virtual painting and the updated virtual painting is generated based further on the desired texture and the desired pattern,

wherein the one or more illumination characteristics are determined based on;

(i) the visual scene captured using the one or more camera devices of the augmented reality device and (ii) accelerometer data captured by an accelerometer component of the augmented reality device;

wherein the one or more illumination characteristics include;

(i) a position of a light source within the physical environment;

(ii) an angle of the light source;

(iii) an indication of whether the light source is omnidirectional;

(iv) a color of the light source;

(v) an intensity of the light source; and

(vi) a reflectivity value of the physical object.

13. The computer-implemented method of claim 12, wherein the updated virtual painting is visually distinct from the initial virtual painting in terms of each visual aspect selected from size, shape, orientation, pattern, and color, wherein the request specifies to remove content at the first location within the visual scene, wherein the physical object corresponds to the first of specified type of physical object, wherein adjusting the visual scene by painting at least a portion of the identified first object further comprises:

removing the at least a portion of the identified physical object from the visual scene as portrayed by the augmented reality device;

replacing the removed portion with replacement content that is displayed by the augmented reality device, wherein the replacement content is determined based at least in part on content within the visual scene surrounding the removed portion of the identified physical object in the physical environment;

wherein the replacement content is updated in order to maintain absence of the removed portion from the physical environment even as the augmented reality device physically moves relative to the removed portion in the physical environment over time, wherein the updated replacement content is output via the augmented reality device; and

only upon detecting that at least a predefined threshold of the physical object has been painted, rendering a series of frames depicting the physical object as an animated virtual object within the visual scene as portrayed by the augmented reality device.

14. An augmented reality device to maintain conformity of a virtual painting onto a physical object captured and portrayed via the augmented reality device, even as the augmented reality device physically moves relative to the physical object in a physical environment over time, the augmented reality device comprising:

a processor; and

a memory containing a program that, when executed by the processor, performs an operation comprising;

presenting, via a display of the augmented reality device, the physical object within a visual scene captured in the physical environment using one or more camera devices of the augmented reality device, the physical object having an area of a first color;

receiving a request to at least partially fill the area using virtual paint of a second color different from the first color;

responsive to the request, identifying the physical object depicted by the visual scene, based on at least one of;

(i) one or more edges determined for the physical object from analyzing the visual scene; and

(ii) predefined geometrical data specifying geometrical characteristics of a specified type of the physical object;

determining one or more illumination characteristics of the physical environment;

generating, based on an initial image of the identified physical object in the visual scene and based further on the one or more illumination characteristics of the physical environment, an initial virtual painting projected onto the area of the first physical object in the initial image and as viewed from a first physical position of the augmented reality device relative to the physical object in the physical environment;

responsive to the augmented reality device being moved from the first physical position to a second physical position relative to the physical object in the physical environment, the second physical position being distinct from the first physical position, generating, based on an updated image of the identified physical object in the visual scene and based further on the one or more illumination characteristics of the physical environment, an updated virtual painting projected onto the area of the physical object in the updated image and as viewed from the second physical position of the augmented reality device, wherein the updated virtual painting is visually distinct from the initial virtual painting in terms of at least one visual aspect selected from size, shape, orientation, pattern, and color;

wherein the updated virtual painting is generated in order to maintain conformity of the virtual painting onto the physical object even as the augmented reality device physically moves relative to the physical object in the physical environment over time, wherein the initial virtual painting and the updated virtual painting are output.

15. The augmented reality device of claim 14, wherein identifying the physical object within the visual scene comprises:

analyzing the visual scene to determine the one or more edges of the physical object within the visual scene, wherein the physical object is detected based on the determined one or more edges.

16. The augmented reality device of claim 14, wherein identifying the physical object within the visual scene comprises:

retrieving the predefined geometrical data, which specifies geometrical characteristics of the specified type of the physical object; and

analyzing the visual scene using the predefined geometrical data in order to determine that the physical object is an instance of one of the specified type of physical object within the visual scene.

17. The augmented reality device of claim 14, wherein the area is of a specific shape.

18. The augmented reality device of claim 14,

wherein the one or more illumination characteristics include at least one of a position of a light source within the physical environment, an angle of the light source, an indication of whether the light source is omnidirectional, a color of the light source, an intensity of the light source, and a reflectivity value of the physical object.

19. The augmented reality device of claim 14, wherein the request specifies to remove content at a first location within the visual scene, wherein adjusting the visual scene by painting at least a portion of the identified first object comprises:

removing the portion of the identified physical object from the visual scene as portrayed by the augmented reality device; and

replacing the removed portion with replacement content that is displayed by the augmented reality device, wherein the replacement content is determined based at least in part on content within the visual scene surrounding the removed portion of the identified physical object in the physical environment.

20. The augmented reality device of claim 14, the operation further comprising:

upon detecting that at least a predefined threshold of the physical object has been painted, rendering a series of frames depicting the physical object as an animated virtual object within the visual scene.

21. A non-transitory computer-readable medium containing a program that, when executed, performs an to maintain conformity of a virtual painting onto a physical object captured and portrayed via an augmented reality device, even as the augmented reality device physically moves relative to the physical object in a physical environment over time, the operation comprising:

presenting, via a display of the augmented reality device, the physical object within a visual scene captured in the physical environment using one or more camera devices of the augmented reality device, the physical object having an area of a first color;

receiving a request to at least partially fill the area using virtual paint of a second color different from the first color;

responsive to the request, identifying the physical object depicted by the visual scene, based on at least one of;

(i) one or more edges determined for the physical object from analyzing the visual scene; and

(ii) predefined geometrical data specifying geometrical characteristics of a predefined type of the physical object;

determining one or more illumination characteristics of the physical environment;

generating, based on an initial image of the identified physical object in the visual scene and based further on the one or more illumination characteristics of the physical environment, an initial virtual painting projected onto the area of the physical object in the initial image and as viewed from a first physical position of the augmented reality device;

responsive to the augmented reality device being moved from the first physical position to a second physical position relative to the physical object in the physical environment, the second physical position being distinct from the first physical position, generating, based on an updated image of the identified physical object in the visual scene and based further on the one or more illumination characteristics of the physical environment, an updated virtual painting projected onto the area the physical object in the updated image and as viewed from the second physical position of the augmented reality device, wherein the updated virtual painting is visually distinct from the initial virtual painting in terms of at least one visual aspect selected from size, shape, orientation, pattern, and color;

wherein the updated virtual painting is generated in order to maintain conformity of the virtual painting onto the physical object even as the augmented reality device physically moves relative to the physical object in the physical environment over time, wherein the initial virtual painting and the updated virtual painting are output.

22. The non-transitory computer-readable medium of claim 21, wherein identifying the physical object within the visual scene comprises:

analyzing the visual scene to determine the one or more edges of the physical object within the visual scene, wherein the physical object is detected based on the determined one or more edges.

23. The non-transitory computer-readable medium of claim 21, wherein identifying the physical object within the visual scene comprises:

retrieving the predefined geometrical data, which specifies geometrical characteristics of the specified type of the physical object; and

analyzing the visual scene using the predefined geometrical data in order to determine that the physical object is an instance of one of the specified type of physical object within the visual scene.

24. The non-transitory computer-readable medium of claim 21, wherein the area is of a specific shape.

25. The non-transitory computer-readable medium of claim 21,

wherein the one or more illumination characteristics include at least one of a position of a light source within the physical environment, an angle of the light source, an indication of whether the light source is omnidirectional, a color of the light source, an intensity of the light source, and a reflectivity value of the physical object.

26. The non-transitory computer-readable medium of claim 21, wherein the request specifies to remove content at a first location within the visual scene, wherein adjusting the visual scene by painting at least a portion of the identified first object comprises:

removing the portion of the identified physical object from the visual scene as portrayed by the augmented reality device; and

replacing the removed portion with replacement content that is displayed by the augmented reality device, wherein the replacement content is determined based at least in part on content within the visual scene surrounding the removed portion of the identified physical object in the physical environment.

27. The non-transitory computer-readable medium of claim 21, the operation further comprising:

upon detecting that at least a predefined threshold of the physical object has been painted, rendering a series of frames depicting the physical object as an animated virtual object within the visual scene.