Method and apparatus for gesture interaction with a photo-active painted surface

• US 9,804,683 B1
• Filed: 02/09/2017
• Issued: 10/31/2017
• Est. Priority Date: 10/22/2012
• Status: Active Grant

• Current Assignee: X Development LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Lee, Johnny, Teller, Eric, Patrick, William Graham, Peeters, Eric
• Primary Examiner(s): Balaoing, Ariel
• Application Number:
  US15/428,751
• Time in Patent Office: 264 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0354 with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G09G 2300/026 Video wall, i.e. juxtaposition of a plurality of screens to create a display screen of bigger dimensions
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: H04M 1/2755 by optical scanning

Abstract

A method and apparatus for gesture interaction with a photo-active painted surface is described. The method may include driving a spatial electromagnetic modulator to emit electromagnetic stimulation in the form of an image to cause photo-active paint to display the image. The method may also include capturing, with at least a camera of a painted surface display system, image data of the image displayed on the photo-active paint applied to a surface and a user motion performed relative to the image. The method may also include analyzing the captured image data to determine a sequence of one or more physical movements of the user relative to the image displayed on the photo-active paint. The method may also include determining, based on the analysis, that the user motion is indicative of a gesture, and driving the spatial electromagnetic modulator to update.

Claims

1. A non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the processor to:

monitor a surface and an area in front of the surface;

determine, based on the monitoring, when a user approaches the surface or is in the area in front of the surface;

generate, based on the determination, an image of a control interface on the surface;

capture image data of the image of the control interface displayed on the surface and user motion performed relative to the image of the control interface;

analyze the captured image data to determine if the user motion is indicative of a gesture associated with the image of the control interface displayed on the surface; and

control a connected system in response to the gesture, wherein the connected system is different than a system including the processor, and wherein the control interface is associated with the connected system.

2. The non-transitory computer readable storage medium of claim 1, wherein the connected system is one of a remote server, a home entertainment system, a home security system, or a lighting control system.

3. The non-transitory computer readable storage medium of claim 1 further including instructions that, when executed by the processor cause the processor to:

determine, due to the monitoring, an identity of the user, wherein the image of the control interface is a specific control interface associated with the identity of the user, and wherein different control interfaces are generated for different users.

4. The non-transitory computer readable storage medium of claim 1, wherein the gesture is one of a formal gesture, a user-created gesture, a natural gesture, or a user-interface control gesture.

5. The non-transitory computer readable storage medium of claim 1 further including instructions that, when executed by the processor cause the processor to:

cause the image of the control interface to be updated in response to the gesture.

6. The non-transitory computer readable storage medium of claim 1, wherein to analyze the captured image data to determine if the user motion is indicative of a gesture associated with the image of the control interface further includes instructions that, when executed by the processor cause the processor to:

analyze the captured image data to determine a sequence of one or more physical movements of the user relative to the image of the control interface, wherein the sequence of one or more physical movements is indicative of the gesture.

7. The non-transitory computer readable storage medium of claim 1 further including instructions that, when executed by the processor cause the processor to:

determine, based on the monitoring, whether the user is approaching the surface; and

cause the image of the control interface to be enlarged in response to the determination.

8. The non-transitory computer readable storage medium of claim 1 further including instructions that, when executed by the processor cause the processor to:

determine, based on the monitoring, a user context, wherein the user context includes an identity of the user, a user'"'"'s proximity to the surface, a user'"'"'s posture and attention relative to the surface, and a user'"'"'s proximity to different areas of the surface.

9. The non-transitory computer readable storage medium of claim 8, wherein the user context further determines the control interface to generate on the surface.

10. The non-transitory computer readable storage medium of claim 8, wherein the user'"'"'s proximity to different areas of the surface determines a control interface of a plurality of control interfaces to generate on the surface, and wherein different areas of the surface are associated with different control interfaces.

11. A system, comprising:

an audio-visual sensor coupled to monitor a surface and an area in front of the surface, and provide sensor data in response;

an electromagnetic (EM) modulator coupled to stimulate generation of an image on the surface, wherein photoactive compounds disposed on the surface generate the image in response to the stimulation, and wherein the EM modulator stimulates image generation in response to one or more control signals; and

a non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the system to;

receive first sensor data from the audio-visual sensor;

determine, based on the first sensor data, a context for a user with respect to the surface;

provide the one or more control signals to the EM modulator in response to the determination of the context for the user, wherein the one or more control signals cause the EM modulator to stimulate generation of an image of a control interface, the control interface associated with a connected system different than the system;

receive second sensor data from the audio-visual sensor, the second sensor data including user motion; and

determine, based on the second sensor data, that the user motion is indicative of a gesture associated with the image of the control interface displayed on the surface; and

control the connected system in response to the gesture.

12. The system of claim 11, wherein the connected system is one of a remote server, a home entertainment system, a home security system, or a lighting control system.

13. The system of claim 11, wherein the audio-visual sensor is a camera.

14. The system of claim 11, wherein a photoactive paint is disposed on the surface, and wherein the photoactive paint generates the image of the control interface in response to the stimulation.

15. The system of claim 14, wherein the photoactive paint includes one of a photoluminescent, photochromic, or thermo-chromic material.

16. The system of claim 11, wherein the non-transitory computer readable storage medium further includes instructions that, when executed by the processor, cause the system to:

determine, based on the first sensor data, an identity of the user, wherein the image of the control interface is a specific control interface associated with the identity of the user, and wherein different control interfaces are generated for different users.

17. The system of claim 11, wherein the gesture is one of a formal gesture, a user-created gesture, a natural gesture, or a user-interface control gesture.

18. The system of claim 11, wherein the non-transitory computer readable storage medium further includes instructions that, when executed by the processor, cause the system to:

cause the image of the control interface to be updated in response to the gesture.

19. The system of claim 11, wherein the user context includes an identity of the user, a user'"'"'s proximity to the surface, a user'"'"'s posture and attention relative to the surface, and a user'"'"'s proximity to different areas of the surface.

20. The system of claim 19, wherein the user'"'"'s proximity to different areas of the surface determines a control interface of a plurality of control interfaces to generate on the surface, wherein different areas of the surface are associated with different control interfaces.