Method and apparatus for gesture interaction with a photo-active painted surface

• US 9,164,596 B1
• Filed: 10/22/2012
• Issued: 10/20/2015
• Est. Priority Date: 10/22/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lee, Johnny, Teller, Eric, Patrick, William Graham, Peeters, Eric
• Primary Examiner(s): BALAOING, ARIEL A
• Application Number:
  US13/657,363
• Time in Patent Office: 1,093 Days
• Field of Search: 345/158
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0354 with detection of 2D relative movements between the device, or an operating part thereof, and a plane or surface, e.g. 2D mice, trackballs, pens or pucks
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0483 Interaction with page-structured environments, e.g. book metaphor
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G09G 2300/026 Video wall, i.e. juxtaposition of a plurality of screens to create a display screen of bigger dimensions
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: H04M 1/2755 by optical scanning

Abstract

A method and apparatus for gesture interaction with a photo-active painted surface is described. The method may include driving a spatial electromagnetic modulator to emit electromagnetic stimulation in the form of an image to cause photo-active paint to display the image. The method may also include capturing, with at least a camera of a painted surface display system, image data of the image displayed on the photo-active paint applied to a surface and a user motion performed relative to the image. The method may also include analyzing the captured image data to determine a sequence of one or more physical movements of the user relative to the image displayed on the photo-active paint. The method may also include determining, based on the analysis, that the user motion is indicative of a gesture, and driving the spatial electromagnetic modulator to update.

Claims

1. A method, comprising:

driving a spatial electromagnetic modulator to emit electromagnetic stimulation in the form of an image to cause photo-active paint to display the image;

capturing, with at least a camera of a painted surface display system, image data of the image displayed on the photo-active paint applied to a surface and a user motion performed relative to the image, wherein the image displayed on the photo-active paint persists for a period of time even when the user motion obstructs a path between the spatial electromagnetic modulator and the image displayed on photo-active paint;

analyzing the captured image data to determine a sequence of one or more physical movements of the user relative to the image displayed on the photo-active paint;

determining, based on the analysis, that the user motion is indicative of a gesture; and

processing data by the painted surface display system based on the determined gesture.

2. The method of claim 1, further comprising:

determining a real-world context associated with the user based on results of the analysis of the sequence of one or more physical movements of the user relative to the image in the captured image data;

generating a control interface for display on the photo-active paint applied to the surface based on the determined real-world context; and

driving the spatial electromagnetic modulator to emit electromagnetic stimulation in the form the generated control interface to cause the photo-active paint to display an image of the control interface.

3. The method of claim 2, wherein the real-world context includes one or more of the user'"'"'s movement relative to the surface, the user'"'"'s position relative to a plurality of different locations of the surface, a determined identify of the user, a determination of the user'"'"'s direction of attention relative to the surface, or one or more user control interface preferences.

4. The method of claim 2, wherein the determined gesture is a gesture to interact with a control in the displayed image of the control interface, further comprising:

driving the spatial electromagnetic modulator to emit electromagnetic stimulation in the form of an update to the image to cause the photo-active paint to display an updated image of the control interface.

5. The method of claim 1, further comprising:

transmitting the processed data to a remote system.

6. The method of claim 5, wherein the remote system is one of a remote server, a home entertainment system, home security system, or a lighting control system.

7. The method of claim 4, wherein the image displayed on photo-active paint applied to the surface is an image of a page from a book, the determined gesture is a gesture to change which page of the book is displayed, and the update to the image is the display of an image of a new page in the book.

8. The method of claim 4, further comprising:

detecting that the user motion indicative of the gesture is indicative of a data entry gesture; and

driving the spatial electromagnetic modulator to emit electromagnetic stimulation in the form of the detected data entry to cause the photo-active paint to display the data entry as the update to the image.

9. The method of claim 8, further comprising:

detecting a second user motion indicative of a second gesture, the second gesture a command to be applied to the data entry; and

performing the command on data entered with the data entry gesture.

10. The method of claim 8, wherein the data entry is text entry, further comprising:

performing optical character recognition on the text entry; and

driving the spatial electromagnetic modulator to emit electromagnetic stimulation in the form a clean type font version of the text entry to cause the photo-active paint to display the clean type font version of the text entry.

11. The method of claim 8, wherein the data entry is entry of a user generated drawing, further comprising:

performing image recognition analysis on the user generated drawing;

determining a second version of the user generated drawing based on results of the image recognition analysis; and

driving the spatial electromagnetic modulator to emit electromagnetic stimulation in the form the second version of the user generated drawing to cause the photo-active paint to display the second version of the user generated drawing.

12. The method of claim 1, further comprising:

analyzing a second image drawn on the photo-active paint applied to a surface by a hand-held electromagnetic modulator within the captured image data; and

analyzing the second image to determine whether the second image is indicative of a gesture.

13. The method of claim 1, wherein the gesture is one of a formal gesture, a user-created gesture, a natural gesture, or a user-interface control gesture.

19. The non-transitory computer readable storage medium of claim 1, further comprising:

analyzing a second image drawn on the photo-active paint applied to a surface by a hand-held electromagnetic modulator within the captured image data; and

analyzing the second image to determine whether the second image is indicative of a gesture.

14. A non-transitory computer readable storage medium including instructions that, when executed by a processor, cause the processor to perform a method comprising:

driving a spatial electromagnetic modulator to emit electromagnetic stimulation in the form of an image to cause photo-active paint to display the image;

capturing, with at least a camera of a painted surface display system, image data of the image displayed on the photo-active paint applied to a surface and a user motion performed relative to the image, wherein the image displayed on the photo-active paint persists for a period of time even when the user motion obstructs a path between the spatial electromagnetic modulator and the image displayed on photo-active paint;

analyzing the captured image data to determine a sequence of one or more physical movements of the user relative to the image displayed on the photo-active paint;

determining, based on the analysis, that the user motion is indicative of a gesture; and

processing data by the painted surface display system based on the determined gesture.

15. The non-transitory computer readable storage medium of claim 14, further comprising:

determining a real-world context associated with the user based on results of the analysis of the sequence of one or more physical movements of the user relative to the image in the captured image data;

generating a control interface for display on the photo-active paint applied to the surface based on the determined real-world context; and

driving the spatial electromagnetic modulator to emit electromagnetic stimulation in the form the generated control interface to cause the photo-active paint to display an image of the control interface.

16. The non-transitory computer readable storage medium of claim 14, wherein the image displayed on photo-active paint applied to the surface is an image of a page from a book, the determined gesture is a gesture to change which page of the book is displayed, and the update to the image is the display of an image of a new page in the book.

17. The non-transitory computer readable storage medium of claim 14, further comprising:

detecting that the user motion indicative of the gesture is indicative of a data entry gesture;

driving the spatial electromagnetic modulator to emit electromagnetic stimulation in the form the detected data entry as the update to the image to cause the photo-active paint to display the data entry.

18. The non-transitory computer readable storage medium of claim 17, wherein the data entry is text entry, further comprising:

performing optical character recognition on the text entry; and

driving the spatial electromagnetic modulator to emit electromagnetic stimulation in the form a clean type font version of the text entry to cause the photo-active paint to display the clean type font version of the text entry.

20. A painted surface display system comprising:

a display engine to drive a spatial electromagnetic modulator to emit electromagnetic stimulation in the form of an image to cause photo-active paint to display the image;

an audiovisual capture device to capture image data of an image displayed on the photo-active paint applied to a surface and a user motion performed relative to the image with at least a camera of the painted surface display system, wherein the image displayed on the photo-active paint persists for a period of time even when the user motion obstructs a path between the spatial electromagnetic modulator and the image displayed on photo-active paint;

a gesture recognition engine to analyze the captured image data to determine a sequence of one or more physical movements of the user relative to the image, and determine, based on the analysis, that the user motion is indicative of a gesture; and

a control engine to process data based on the determined gesture.