Method and Apparatus for Controlling a Mobile Device Using a Camera

• US 20100159981A1
• Filed: 12/23/2008
• Published: 06/24/2010
• Est. Priority Date: 12/23/2008
• Status: Active Grant

• Current Assignee: HTC Corporation
• Original Assignee: HTC Corporation
• Inventors: Chang, Wen-Chi, Chiang, Ching-Liang, Hsu, Chih-Feng
• Granted Patent: US 9,417,699 B2
• Time in Patent Office: Days
• US Class Current: 455/556.100
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04M 1/725 Cordless telephones user interfaces specially adapted therefor H04M1/724
• CPC Class Codes: H04M 2250/52 including functional features of a camera

Abstract

A method and system for using an image sensor to control applications on a mobile device is disclosed. The system uses image processing to detect control gestures by the user. Control gestures may include, for example, moving a hand laterally, moving the hand closer or farther away from the camera, or opening and closing a hand. These control motions are linked to user commands to be executed by the application being controlled. The system initializes by storing a background image showing the field of view of the image sensor before any objects are present. It then detects new objects in the field of view by comparing new images to the background image. If a new object is detected, the system waits until the object is stationary in the field of view. After the object is stationary, the system detects if there is a change in the object indicating a control gesture. After detecting a change in the stationary object, the system determines a set of parameters defining the change and matches the parameters to a user command. The parameters may include the object'"'"'s change in position or size or change in hand state (e.g. open or closed). The system then passes the command to an application to be executed.

Claims

1. A method for interpreting a control gesture to a mobile device having an image sensor capable of providing a sequence of images, comprising:

detecting an object in the sequence of images;

determining if the detected object is stationary;

if the detected object is determined to be stationary, detecting a change in the detected object;

determining one or more parameters associated with the detected change;

interpreting a user command indicated by the determined one or more parameters; and

executing an action associated with the interpreted user command.

2. The method as claimed in claim 1, wherein detecting an object comprises:

acquiring a background image;

comparing a current image to the background image; and

if a part of the current image differs substantially from a corresponding part of the background image, designating the part of the current image as the detected object.

3. The method as claimed in claim 1, wherein detecting a change comprises:

determining a value of a parameter associated with the object; and

detecting if the value of the parameter changes in a later image of the sequence of images.

4. The method as claimed in claim 1, further comprising:

classifying the detected object by object type; and

determining parameters of the one or more parameters selected depending on the object type of the detected object.

5. The method as claimed in claim 1, further comprising ignoring the interpreted user command if the interpreted user command is received within a specified time after a previous user command.

6. The method as claimed in claim 1, further comprising:

determining a first location of the object in a first image of the sequence of images; and

determining a second location of the object in a second image of the sequence of images;

wherein determining one or more parameters comprises determining a direction parameter by determining the direction of change from the first location to the second location.

7. The method as claimed in claim 1, wherein determining one or more parameters associated with the detected change comprises:

determining a first hand state associated with the detected object when the detected object is stationary;

determining a second hand state associated with the detected object after the detected change; and

if the first hand state differs from the second hand state, generating a parameter indicating a change between the first hand state and the second hand state,

wherein determining a first hand state and determining a hand state comprise;

determining a first point of the detected object;

determining a second point of the detected object;

if a segment between the first point and the second point is substantially a single color, setting the hand state to closed; and

if the segment between the first point and the second point is not substantially a single color, setting the hand state to open.

8. The method as claimed in claim 1, wherein determining one or more parameters associated with the detected change comprises:

when the detected object is stationary, determining a first hand state associated with the detected object;

after the change is detected, determining a second hand state associated with the detected object; and

generating a parameter indicating a change between the first hand state and the second hand state.

9. The method as claimed in claim 1, wherein detecting an object comprises;

comparing a second image of the sequence of images to a first image of the sequence of images;

if the second image is substantially identical to the first image, setting the second image as a background image; and

detecting an object if a third image of the sequence of images differs substantially from the background image.

10. The method as claimed in claim 1, wherein determining if the detected object is stationary further comprises:

comparing a current image of the sequence of images to a previous image of the sequence of images; and

designating the object as stationary if the current image is substantially identical to the previous image.

11. An apparatus for interpreting a control gesture to a mobile device, comprising:

a camera configured to generate a sequence of images;

a memory; and

a processor coupled to the camera and the memory, wherein the processor is configured to execute various components, wherein the components comprise;

an object detector component configured to detect an object in the sequence of images;

a stationary object detector component configured to determine if the object is stationary;

a change detector component configured detect a change in the object after the object is determined to be stationary;

a gesture identifier component configured to determine one or more parameters associated with the detected change;

an application control component configured to interpret a user command indicated by the one or more parameters; and

a mobile application configured to receive the user command from the application control component and to execute an action associated with the user command.

12. The apparatus as claimed in claim 11, further wherein the processor is further configured to execute:

a background image generator component configured to acquire a background image;

wherein the object detector component is configured to detect the object by comparing the current image to the background image and, if a part of the current image differs substantially from a corresponding part of the background image, designating the part of the current image as the detected object.

13. The apparatus as claimed in claim 11, wherein the change detector is configured to detect a change by determining a value of a parameter associated with the object and detecting if the value of the parameter changes in a later image of the sequence of images.

14. The apparatus as claimed in claim 11, wherein the application control component is further configured to ignore the interpreted user command if the interpreted user command is received within a specified time after a previous user command.

15. The apparatus as claimed in claim 11, wherein the gesture identifier component is further configured to:

determine a first dimension of the object in a first image of the sequence of images; and

determine a second dimension of the object in a second image of the sequence of images;

wherein determining one or more parameters comprises determining a distance change parameter by determining a change in magnitude between the first dimension and the second dimension.

16. The apparatus as claimed in claim 11, wherein the processor is further configured to execute:

a hand state detector component configured to determine a first hand state associated with the detected object when the detected object is stationary and a second hand state associated with the detected object after the detected change,

wherein the gesture identifier is further configured to generate a parameter indicating a change between the first hand state and the second hand state.

17. The apparatus as claimed in claim 11, wherein the processor is further configured to execute:

a background image generator component configured to compare a second image of the sequence of images to a first image of the sequence of images and set the second image as the background image if the second image is substantially identical to the first image;

wherein the object detector component is further configured to detect an object if a third image of the sequence of images differs substantially from the background image.

18. An apparatus for interpreting a control gesture to a mobile device, comprising:

a means for generating a sequence of images;

a means for detecting an object in the sequence of images;

a means for determining if the detected object is stationary;

a means for detecting a change in the detected object if the detected object is determined to be stationary;

a means for determining one or more parameters associated with the detected change;

a means for interpreting a user command indicated by the determined one or more parameters; and

a means for executing an action associated with the interpreted user command.

19. The apparatus as claimed in claim 18, wherein the means for detecting an object comprises:

a means for acquiring a background image;

a means for comparing a current image to the background image; and

a means for designating a part of the current image as the detected object if the part of the current image differs substantially from a corresponding part of the background image.

20. The apparatus as claimed in claim 18, wherein the means for detecting a change comprises:

a means for determining a value of a parameter associated with the object; and

a means for detecting if the value of the parameter changes in a later image of the sequence of images.

21. The apparatus as claimed in claim 18, further comprising:

a means for classifying the detected object by object type;

wherein the one or more parameters are selected depending on the object type of the detected object.

22. The apparatus as claimed in claim 18, further comprising a means for ignoring the interpreted user command if the interpreted user command is received within a specified time after a previous user command.

23. The apparatus as claimed in claim 18, wherein the means for determining one or more parameters associated with the detected change comprises:

a means for determining a first width of the detected object when the detected object is stationary;

a means for determining a second width of the detected object after the detected change; and

a means for generating a parameter indicating a distance change parameter by determining a change in magnitude between the first width and the second width.

24. The apparatus as claimed in claim 18, wherein the means for detecting an object comprises:

a means for comparing a second image of the sequence of images to a first image of the sequence of images;

a means for setting the second image as a background image if the second image is substantially identical to the first image;

a means for detecting an object if a third image of the sequence of images differs substantially from the background image.

25. The apparatus as claimed in claim 18, wherein the means for determining if the detected object is stationary comprises:

a means for comparing a current image of the sequence of images to a previous image of the sequence of images; and

a means for designating the object as stationary if the current image is substantially identical to the previous image.

26. The apparatus as claimed in claim 18, wherein the means for determining one or more parameters further comprises:

a means for determining a first hand state associated with the detected object when the detected object is stationary;

a means for determining a second hand state associated with the detected object after the change is detected; and

a means for generating a parameter indicating a change between the first hand state and the second hand state.

27. The apparatus as claimed in claim 18, wherein the means for detecting an object further comprises:

a means for comparing a second image of the sequence of images to a first image of the sequence of images;

a means for setting the second image as a background image if the second image is substantially identical to the first image;

a means for detecting an object if a third image of the sequence of images differs substantially from the background image.