Method, system, and program for gesture based option selection

• US 6,283,860 B1
• Filed: 08/14/2000
• Issued: 09/04/2001
• Est. Priority Date: 11/07/1995
• Status: Expired due to Term

• Current Assignee: Philips Electronics North America Corporation (Koninklijke Philips N.V.)
• Original Assignee: Philips Electronics North America Corporation (Koninklijke Philips N.V.)
• Inventors: Murphy, Thomas G., Lyons, Damian M.
• Primary Examiner(s): Vo, Peter
• Assistant Examiner(s): PARADISO, JOHN ROGER
• Application Number:
  US09/638,132
• Time in Patent Office: 386 Days
• Field of Search: 345/358, 345/121, 345/156, 345/127, 345/131, 382/313, 600/587, 353/52, 434/410, 434/379, 434/307.R, 434/308.R, 434/309, 434/314, 434/323-324, 434/365, 434/367, 434/408, 434/428-430, 463/36
• US Class Current: 463/36
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/42 by mapping the input signals into game commands, e.g. mapping the displacement of a stylus on a touch screen to the steering angle of a virtual vehicle
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: A63F 2300/8029 Fighting without shooting
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 7/254 involving subtraction of images

Abstract

The system displays on a screen a set of options that are selectable by a user. The user standing in front of the screen points at a desired option and a camera of the system takes an image of the user while pointing. The system calculates from the pose of the user in the image whether the user is pointing to any of the displayed options. If such is the case, that particular option is selected and an action corresponding with that option is executed.

Claims

1. A method for enabling a user to select a particular option from a set of selectable options, the method comprising the steps of:

displaying the set of selectable options at respective option positions;

while allowing the user to physically point at the particular option, taking a single image of the user, wherein the single image does not include an image of the selectable options;

ascertaining from the single image whether the user points at the particular option;

if such is the case, concluding that the user has selected the particular option; and

based on the single image, determining a feature position of a body feature of the user in 3 dimensional space;

the step of ascertaining from the single image whether the user has pointed at the particular option being executed on the basis of the feature position of the body feature and the option position of the particular option.

2. A method as claimed in claim 1, wherein the set of selectable objects is displayed on a video screen facing the user.

3. A method as claimed in claim 1, wherein the body feature is the pointing end of an arm of the user.

4. A method as claimed in claim 3, wherein the method further comprises, based on the single image, determining a foot position of a foot of the user, a projection of the arm of the user on the plane of the single image, and a long axis of the body of the user on the plane of the single image and wherein determining the feature position comprises:

determining the distance between the option position of the particular option and the foot position;

determining a first angle between the projection of the arm of the user and the projection of the long axis of the body of the user; and

determining a second-angle between the arm of the user and the plane of the single image on the basis of an estimated arm length of the arm of the user and a determined length of the projection of the arm.

5. A method as claimed in claim 4 comprising an initialization step in which the height of the user is determined, wherein the estimated arm length is obtained from biometrics data on the basis of the height of the user.

6. A method as claimed in claim 4, wherein the first angle between the projection of the arm of the user and the projection of the long axis of the body of the user is a projection of the arm of the user and a projection of the long axis of the body taken at an eye level of the user.

7. A method as claimed in claim 6, wherein the projection of the long axis of the body at the eye level of the user is determined utilizing stored biometric data.

8. A system for enabling a user to select a particular option from a set of selectable options, the sysytem comprising:

display means for displaying the set of selectable options at respective option positions;

a single image taking means for taking a single image of the user, wherein the single image does not include an image of the selectable options;

ascertaining means for ascertaining from the single image whether the user points at the particular option;

concluding means for concluding that the user has selected the particular option, in the case that the ascertaining means have ascertained that the user points at the particular option; and

feature determining means for based on the single image determining a feature position of a body feature of the user in 3 dimensional space;

the ascertaining means being arranged to ascertain whether the user points at the particular option on the basis of the feature position of the body feature and the option position of the particular option.

9. A system as claimed in claim 8, wherein the display means comprise a video screen facing the user for displaying the set of selectable objects.

10. A system as claimed in claim 8, wherein the body feature is the pointing end of an arm of the user.

11. A system as claimed in claim 10, wherein

the system further comprises further determining means for, based on the single image, determining a foot position of a foot of the user, a projection of the arm of the user on the plane of the single image and a long axis of the body of the user on the plane of the single image; and

the feature determining means are arranged;

to determine the distance between the option position of the particular option and the foot position;

to determine a first angle between the projection of the arm of the user and the projection of the long axis of the body of the user; and

to determine a second angle between the arm of the user and the plane of the single image on the basis of an estimated arm length of the arm of the user and a determined length of the projection of the arm.

12. A system as claimed in claim 11 comprising initialization means for determining the height of the user, wherein the feature determining means are arranged to obtain the estimated arm length from biometrics data on the basis of the height of the user.

13. A system as claimed in claim 11, wherein the feature determining means is arranged to determine the first angle between the projection of the arm of the user and the projection of the long axis of the body of the user utilizing a projection of the arm of the user and a projection of the long axis of the body at an eye level of the user.

14. A system as claimed in claim 13, wherein the projection of the long axis of the body at the eye level of the user is determined utilizing stored biometric data.

15. A program stored on a computer readable medium for enabling a user to select a particular option from a set of selectable options, the program comprising:

an execution portion for displaying the set of selectable options at respective option positions on a display;

an execution portion for initiating a single imaging of the user pointing at the particular option, wherein the single imaging does not include an imaging of the selectable options;

an execution portion for determining from the single image whether the user points at the particular option on the basis of a feature position of a body feature of the user in a three-dimensional space and the option position of the particular option.

16. The program of claim 15, wherein the body feature is the pointing end of an arm of the user.

17. The program of claim 16, wherein the execution portion for determining whether the user points at the particular option comprises:

an execution portion for determining a distance between the option position of the particular option and a foot position of the user; and

an execution portion for determining a first angle between a projection of an arm of the user on the plane of the single image and a projection of a long axis of the body of the user on the plane of the single image; and

an execution portion for determining a second angle between the arm of the user and the plane of the image on the basis of an estimated arm length of the arm of the user and a determined length of the projection of the arm.

18. The program of claim 17, wherein the first angle between the projection of the arm of the user and the projection of the long axis of the body of the user is a projection of the arm of the user and a projection of the long axis of the body at eye level of the user.

19. The program of claim 18, wherein the projection of the long axis of the body at eye level of the user is determined utilizing stored biometric data.