AUGMENTED REALITY SURFACE PAINTING

• US 20140002472A1
• Filed: 06/29/2012
• Published: 01/02/2014
• Est. Priority Date: 06/29/2012
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Mitchell, Kenny, Sobeski, David, Leake, Brian
• Granted Patent: US 10,068,547 B2
• Time in Patent Office: Days
• US Class Current: 345/582
• CPC Class Codes: G09G 5/026 Control of mixing and/or overlay of colours in general G09G5/022 and G09G5/024 take precedence

Abstract

Techniques for simulating interactions using an augmented reality device are described. Embodiments receive a request to paint over portions of a visual scene. Here, the visual scene is captured using one or more camera devices of the augmented reality device and is presented on a display of the augmented reality device. A first object in the visual scene corresponding to the first location is identified. Additionally, embodiments paint at least a portion of the first object as specified by the received request, and render a series of frames depicting the first object, such that the painted at least a portion of the first object is shown as painted when viewed from different perspectives using the augmented reality device.

Claims

1. A method for displaying content using an augmented reality device, comprising:

receiving a request to paint within a visual scene, wherein the visual scene is captured using one or more camera devices of the augmented reality device and is presented on a display of the augmented reality device;

identifying a first object depicted by the visual scene corresponding to the first location;

painting, by operation of one or more computer processors of the augmented reality device, at least a portion of the first object as specified by the received request; and

rendering a series of frames depicting the first object, such that the painted at least a portion of the first object is shown as painted when viewed from different perspectives using the augmented reality device.

2. The method of claim 1, wherein identifying a first object within the visual scene further comprises:

analyzing the visual scene to determine edges of one or more physical objects within the visual scene,

wherein the first object is detected based on the determined edges.

3. The method of claim 1, wherein identifying a first object within the visual scene further comprises:

retrieving predefined geometrical data specifying geometrical characteristics of at least one physical object; and

analyzing the visual scene using the predefined geometrical data to determine that the first object is an instance of one of the at least one physical objects within the visual scene.

4. The method of claim 1, wherein the received request specifies a first shape to paint at the first location within the visual scene and further specifies at least one color for the first shape, and wherein the first shape comprises at least one of (i) a selection of one of a plurality of predefined shapes and (ii) a free form shape created based on user input.

5. The method of claim 4, wherein the received request further specifies a texture for the first shape.

6. The method of claim 1, wherein the received request specifies at least one of (i) a color to paint the at least a portion of the first object and (ii) a pattern to paint the at least a portion of the first object.

7. The method of claim 1, further comprising:

measuring one or more environmental illumination characteristics of an environment in which the augmented reality device is located, wherein the environmental illumination characteristics include at least one of a position of a light source within an environment in which the augmented reality device is located, an angle of the light source, an indication of whether the light source is omnidirectional, a color of the light source, an intensity of the light source and a reflectivity value of a physical object within the environment; and

adjusting the painted at least a portion of the first object based on the measured one or more environmental illumination characteristics.

8. The method of claim 1, wherein the request specifies to remove content at the first location within the visual scene, and wherein adjusting the visual scene by painting at least a portion of the identified first object as specified by the received request further comprises:

removing the at least a portion of the identified first object from the visual scene; and

replacing the removed at least a portion with replacement content, wherein the replacement content is determined based in part on content within the visual scene surrounding the removed at least a portion.

9. The method of claim 1, wherein the first object is a predefined virtual object, and further comprising:

upon detecting that at least a predefined threshold of the first object has been painted, rendering a series of frames depicting the first object as an animated virtual object within the visual scene.

10. An augmented reality device, comprising:

a processor; and

a memory containing a program that, when executed by the processor, performs an operation for displaying content, comprising;

receiving a request to paint within a visual scene, wherein the visual scene is captured using one or more camera devices of the augmented reality device and is presented on a display of the augmented reality device;

identifying a first object depicted by the visual scene corresponding to the first location;

painting at least a portion of the first object as specified by the received request; and

rendering a series of frames depicting the first object, such that the painted at least a portion of the first object is shown as painted when viewed from different perspectives using the augmented reality device.

11. The augmented reality device of claim 10, wherein identifying a first object within the visual scene further comprises:

analyzing the visual scene to determine edges of one or more physical objects within the visual scene,

wherein the first object is detected based on the determined edges.

12. The augmented reality device of claim 10, wherein identifying a first object within the visual scene further comprises:

retrieving predefined geometrical data specifying geometrical characteristics of at least one physical object; and

analyzing the visual scene using the predefined geometrical data to determine that the first object is an instance of one of the at least one physical objects within the visual scene.

13. The augmented reality device of claim 10, wherein the received request specifies a first shape to paint at the first location within the visual scene and further specifies at least one color for the first shape, and wherein the first shape comprises at least one of (i) a selection of one of a plurality of predefined shapes and (ii) a free form shape created based on user input.

14. The augmented reality device of claim 13, wherein the received request further specifies a texture for the first shape.

15. The augmented reality device of claim 10, wherein the received request specifies at least one of (i) a color to paint the at least a portion of the first object and (ii) a pattern to paint the at least a portion of the first object.

16. The augmented reality device of claim 10, the operation further comprising:

measuring one or more environmental illumination characteristics of an environment in which the augmented reality device is located, wherein the environmental illumination characteristics include at least one of a position of a light source within an environment in which the augmented reality device is located, an angle of the light source, an indication of whether the light source is omnidirectional, a color of the light source, an intensity of the light source and a reflectivity value of a physical object within the environment; and

adjusting the painted at least a portion of the first object based on the measured one or more environmental illumination characteristics.

17. The augmented reality device of claim 10, wherein the request specifies to remove content at the first location within the visual scene, and wherein adjusting the visual scene by painting at least a portion of the identified first object as specified by the received request further comprises:

removing the at least a portion of the identified first object from the visual scene; and

replacing the removed at least a portion with replacement content, wherein the replacement content is determined based in part on content within the visual scene surrounding the removed at least a portion.

18. The augmented reality device of claim 10, wherein the first object is a predefined virtual object, and the operation further comprising:

upon detecting that at least a predefined threshold of the first object has been painted, rendering a series of frames depicting the first object as an animated virtual object within the visual scene.

19. A computer-readable storage medium containing a program that, when executed, performs an operation for displaying content using an augmented reality device, comprising:

receiving a request to paint within a visual scene, wherein the visual scene is captured using one or more camera devices of the augmented reality device and is presented on a display of the augmented reality device;

identifying a first object depicted by the visual scene corresponding to the first location;

painting at least a portion of the first object as specified by the received request; and

rendering a series of frames depicting the first object, such that the painted at least a portion of the first object is shown as painted when viewed from different perspectives using the augmented reality device.

20. The computer-readable storage medium of claim 19, wherein identifying a first object within the visual scene further comprises:

analyzing the visual scene to determine edges of one or more physical objects within the visual scene,

wherein the first object is detected based on the determined edges.

21. The computer-readable storage medium of claim 19, wherein identifying a first object within the visual scene further comprises:

retrieving predefined geometrical data specifying geometrical characteristics of at least one physical object; and

analyzing the visual scene using the predefined geometrical data to determine that the first object is an instance of one of the at least one physical objects within the visual scene.

22. The computer-readable storage medium of claim 19, wherein the received request specifies a first shape to paint at the first location within the visual scene and further specifies at least one color for the first shape, and wherein the first shape comprises at least one of (i) a selection of one of a plurality of predefined shapes and (ii) a free form shape created based on user input.

23. The computer-readable storage medium of claim 22, wherein the received request further specifies a texture for the first shape.

24. The computer-readable storage medium of claim 19, wherein the received request specifies at least one of (i) a color to paint the at least a portion of the first object and (ii) a pattern to paint the at least a portion of the first object.

25. The computer-readable storage medium of claim 19, the operation further comprising:

measuring one or more environmental illumination characteristics of an environment in which the augmented reality device is located, wherein the environmental illumination characteristics include at least one of a position of a light source within an environment in which the augmented reality device is located, an angle of the light source, an indication of whether the light source is omnidirectional, a color of the light source, an intensity of the light source and a reflectivity value of a physical object within the environment; and

adjusting the painted at least a portion of the first object based on the measured one or more environmental illumination characteristics.

26. The computer-readable storage medium of claim 19, wherein the request specifies to remove content at the first location within the visual scene, and wherein adjusting the visual scene by painting at least a portion of the identified first object as specified by the received request further comprises:

removing the at least a portion of the identified first object from the visual scene; and

replacing the removed at least a portion with replacement content, wherein the replacement content is determined based in part on content within the visual scene surrounding the removed at least a portion.

27. The computer-readable storage medium of claim 19, wherein the first object is a predefined virtual object, and the operation further comprising:

upon detecting that at least a predefined threshold of the first object has been painted, rendering a series of frames depicting the first object as an animated virtual object within the visual scene.