SYSTEM AND METHOD FOR INTERACTING WITH A DISPLAY

• US 20080143975A1
• Filed: 10/25/2006
• Published: 06/19/2008
• Est. Priority Date: 10/25/2006
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: SCHUNZEL, Mark J., McCULLOCH, Douglas J., TREVATHAN, Matthew B., DENNARD, Mark D.
• Granted Patent: US 8,356,254 B2
• Time in Patent Office: Days
• US Class Current: 353/42
• CPC Class Codes: G03B 21/26 Projecting separately subsidiary matter simultaneously with main image

Abstract

A system and method of interacting with a display. The method comprises recognizing a disturbance in a display zone of a projected image and displaying a selected state in response to the recognized disturbance. The method further includes recognizing a gesture which interrupts a light source and is associated with an action to be taken on or associated with the displayed selected state. An action is executed in response to the recognized gesture. The system includes a server having a database containing data associated with at least one or more predefined gestures, and at least one of a hardware and software component for executing an action based on the at least one or more predefined gestures.

Claims

1. A method, comprising:

recognizing a disturbance in a display zone of a projected image;

displaying a selected state in response to the recognized disturbance;

recognizing a gesture which interrupts a source and is associated with an action to be taken on or associated with the displayed selected state; and

executing the action in response to the recognized gesture.

2. The method of claim 1, wherein the recognized gesture is a motion of a non-tethered object between the source and the projected image.

3. The method of claim 1, wherein the recognizing the disturbance and gesture is captured by a camera and sent to a processor which performs the recognition and execution of the action.

4. The method of claim 1, wherein the display zone is one or more areas on the projected image which are associated with an application program.

5. The method of claim 1, further comprising validating the disturbance and gesture using a template.

6. The method of claim 1, further comprising validating the gesture via a look-up table.

7. The method of claim 1, further comprising making a determination as to whether a shape of a disturbing object is a valid shape in order to display the selected state.

8. The method of claim 7, wherein when the disturbing object is a valid shape, the selected state of a selected item is displayed.

9. The method of claim 1, wherein the executed action is a menu driven command.

10. The method of claim 1, wherein:

the recognizing the disturbance is recognizing a first object; and

recognizing the gesture is recognizing a motion of a second object.

11. The method of claim 10, wherein the first object is a finger on a first hand and the second object is a finger or hand motion on another hand.

12. The method of claim 1, wherein the disturbance is recognized by an action in a field of vision of a camera.

13. The method of claim 1, wherein the disturbance is recognized as a trajectory for several specific patterns that indicate a type of motion.

14. The method of claim 1, wherein the action to be executed requires at least two command steps.

15. The method of claim 1, wherein the gesture is captured, reconciled and executed by a computer to perform the action.

16. The method of claim 1, wherein the steps of claim 1 are performed on at least one of a subscription, advertising and fee basis.

17. The method of claim 1, wherein the steps of claim 1 are provided by a provider which at least one of creates, maintains and supports a computer infrastructure that performs the steps of claim 1.

18. The method of claim 17, wherein the provider receives payment from sale of advertising content to one or more third parties.

19. The method of claim 1, wherein the action is performed by a computer which communicates with a camera that captures the disturbance and gesture, and sends the captured disturbance and gesture to the computer for recognition, verification and execution.

20. The method of claim 1, further comprising correcting for distortion of the projected image caused by at least one of oblique projection and a shape of a surface.

21. The method of claim 1, wherein the source is a light source.

22. The method of claim 1, wherein the interrupting is a disturbance of a field of vision of a camera.

23. The method of claim 1, wherein the executing of the action is based on the recognition of the disturbance and the recognition of the gesture.

24. A method, comprising:

projecting an image on a surface using at least a source of light and a processor configured to store and execute application programs associated with the image;

sensing a first action in a display zone of the image;

validating the first action and displaying a selected state in response to the validated first action;

sensing a gesture interrupting the light source;

validating that the gesture is associated with a pre-defined command and the displayed selected state; and

executing the pre-defined command in response to the validated gesture.

25. The method of claim 24, wherein the validating the gesture is performed through a look-up table having a plurality of pre-defined gestures.

26. The method of claim 24, wherein the display zone is one or more areas on the image.

27. The method of claim 24, further comprising making a determination as to whether a shape associated with the first action is a valid shape in order to display the selected state.

28. The method of claim 24, wherein the executed pre-defined action is a menu driven command.

29. The method of claim 24, wherein:

the sensing a first action is sensing a first object; and

sensing the gesture is sensing a pre-defined motion of a second object.

30. The method of claim 24, wherein the first action and gesture are recognized by a disturbance in a field of vision of a camera.

31. The method of claim 30, wherein the first action is recognized as a trajectory for defined patterns that indicate at least one of a type and place of motion.

32. The method of claim 24, wherein the steps of claim 24 are performed on at least one of a subscription, advertising and fee basis.

33. The method of claim 24, wherein the steps of claim 24 are provided by a provider which at least one of creates, maintains and supports a computer infrastructure that performs the steps of claim 24.

34. The method of claim 33, wherein the provider receives payment from sale of advertising content to one or more third parties.

35. A system comprising a server having a database containing data associated with at least one or more predefined gestures, and at least one of a hardware and software component for executing an action based on the at least one or more predefined gestures, the hardware and software:

comparing a first action in an interaction zone to a predefined template of a shape;

comparing a second action, which interrupts a light source, to the at least one or more predefined gestures;

validating the first action and the second action based on the comparing to the predefined template and the at least one or more predefined gestures; and

executing an action based on the validating of the first action and the second action.

36. The system of claim 35, wherein the system includes an architecture comprising a services layer, an integration layer and an application layer.

37. The system of claim 36, wherein the services layer comprises at least:

a vision interface module responsible for recognizing the gesture and converting an appropriate action to an application;

a geometric reasoning module which enables automatic selection of a display based on proximity of the interaction zone to a user and non-occlusion of the interaction zone by the user or other object; and

a user localization module to determine a position of the user.

38. The system of claim 37, wherein one of the geometric reasoning module and user localization module recognize the first action and the second action.

39. The system of claim 35, wherein the at least one of a hardware and software component resides on a server provided by a service provider.

40. A computer program product comprising a computer usable medium having readable program code embodied in the medium, the computer program product includes at least one component to:

recognize a disturbance in a display zone of a projected image;

display a selected state in response to the recognized disturbance;

recognize a gesture interrupting a light source and associated with an action to be taken on or associated with the displayed selected state; and

execute the action in response to the recognized gesture.

41. A method for deploying an application for interacting with a display, comprising:

providing a computer infrastructure being operable to;

project an image on a surface;

sense a first action in a predefined interaction zone of the image;

validate the first action and displaying a selected state;

sense a gesture;

validate that the gesture is associated with a pre-defined action; and

execute the pre-defined action in response to the validated gesture.

42. A method, comprising:

recognizing a first action of a first object and a second action of a object;

validating a movement comprising a combination of the first action and the second action by comparison to predefined gestures; and

executing a complex command based on the validating of the combination of the first action and the second action.