Method and apparatus for controlling a mobile device using a camera

• US 9,417,699 B2
• Filed: 12/23/2008
• Issued: 08/16/2016
• Est. Priority Date: 12/23/2008
• Status: Expired due to Fees

• Current Assignee: HTC Corporation
• Original Assignee: HTC Corporation
• Inventors: Chiang, Ching-Liang, Hsu, Chih-Feng, Chang, Wen-Chi
• Primary Examiner(s): Yodichkas, Aneeta
• Assistant Examiner(s): ELNAFIA, SAIFELDIN E
• Application Number:
  US12/343,055
• Publication Number:
  US 20100159981A1
• Time in Patent Office: 2,793 Days
• Field of Search: 345156-184, 345173-178, 715/863, 382/104, 382/106, 382/107, 382/210, 382/236
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04M 1/725 Cordless telephones user interfaces specially adapted therefor H04M1/724
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264

Abstract

A method and system for using an image sensor to control applications on a mobile device is disclosed. The system uses image processing to detect control gestures by the user. Control gestures may include, for example, moving a hand laterally, moving the hand closer or farther away from the camera, or opening and closing a hand. These control motions are linked to user commands to be executed by the application being controlled. The system initializes by storing a background image showing the field of view of the image sensor before any objects are present. It then detects new objects in the field of view by comparing new images to the background image. If a new object is detected, the system waits until the object is stationary in the field of view. After the object is stationary, the system detects if there is a change in the object indicating a control gesture. After detecting a change in the stationary object, the system determines a set of parameters defining the change and matches the parameters to a user command. The parameters may include the object'"'"'s change in position or size or change in hand state (e.g. open or closed). The system then passes the command to an application to be executed.

Claims

1. A method for interpreting a control gesture to a mobile device having an image sensor capable of providing a sequence of images, comprising:

comparing a second image of the sequence of images to a first image of the sequence of images, the comparing including a comparison between content of the second image and the first image;

if the second image is substantially identical to the first image and has the same content, setting the second image as a background image;

detecting an object if a third image of the sequence of images differs substantially from the background image;

storing only the pixels of said object in the third image that differ substantially from the background image;

using the stored pixels of the object, determining if the detected object is stationary based at least in part on an absence of change of the object among multiple images of the sequence of images;

if the detected object has been determined to be stationary, storing only the pixels of the detected stationary object and determining initial object characteristics from the stored pixels;

detecting a change in the detected object among multiple images of the sequence of images;

using the initial object characteristics, determining one or more parameters associated with the detected change;

interpreting a user command indicated by the determined one or more parameters,

wherein the interpreted user command is ignored if the interpreted user command is received within a specified time after a previous user command; and

executing an action associated with the interpreted user command.

2. The method as claimed in claim 1, wherein detecting an object further comprises:

if a part of the third image differs substantially from a corresponding part of the background image, designating the part of the current image as the detected object.

3. The method as claimed in claim 1, wherein detecting a change comprises:

determining a value of a parameter associated with the object; and

detecting if the value of the parameter changes in a later image of the sequence of images.

4. The method as claimed in claim 1, further comprising:

classifying the detected object by object type; and

determining parameters of the one or more parameters selected depending on the object type of the detected object.

5. The method as claimed in claim 1, further comprising:

determining a first location of the object in a first image of the sequence of images; and

determining a second location of the object in a second image of the sequence of images;

wherein determining one or more parameters comprises determining a direction parameter by determining the direction of change from the first location to the second location.

6. The method as claimed in claim 1, wherein determining one or more parameters associated with the detected change comprises:

determining a first hand state associated with the detected object if the detected object is stationary;

determining a second hand state associated with the detected object after the detected change; and

if the first hand state differs from the second hand state, generating a parameter indicating a change between the first hand state and the second hand state,

wherein determining a first hand state and determining a hand state comprise;

determining a first point of the detected object;

determining a second point of the detected object;

if a segment between the first point and the second point is substantially a single color, setting the hand state to closed; and

if the segment between the first point and the second point is not substantially a single color, setting the hand state to open.

7. The method as claimed in claim 1, wherein determining one or more parameters associated with the detected change comprises:

if the detected object is stationary, determining a first hand state associated with the detected object;

after the change is detected, determining a second hand state associated with the detected object; and

generating a parameter indicating a change between the first hand state and the second hand state.

8. The method as claimed in claim 1, wherein determining if the detected object is stationary further comprises:

comparing a current image of the sequence of images to a previous image of the sequence of images; and

designating the object as stationary if the current image is substantially identical to the previous image.

9. An apparatus for interpreting a control gesture to a mobile device, comprising:

a camera configured to generate a sequence of images;

a memory; and

a processor coupled to the camera and the memory, wherein the processor is configured to execute various components, wherein the components comprise;

a background image generator component configured to compare a second image of the sequence of images to a first image of the sequence of images and set the second image as the background image if the second image is substantially identical to the first image, the comparing including a comparison between content of the second image and the first image;

an object detector component configured to detect an object in the sequence of images if a third image of the sequence of images differs substantially from the background image, the object detector storing only the pixels of said object in the third image that differ substantially from the background image;

a stationary object detector component configured to use the stored pixels of the object to determine if the object is stationary based at least in part on an absence of change among multiple images of the sequence of images, and to store only the pixels of the stationary object;

a change detector component configured detect a change in the object among multiple images of the sequence of images after the object is determined to be stationary;

a gesture identifier component configured to determine one or more parameters associated with the detected change;

an application control component configured to interpret a user command indicated by the one or more parameters,

wherein the application control component is further configured to ignore the interpreted user command if the interpreted user command is received within a specified time after a previous user command; and

a mobile application configured to receive the user command from the application control component and to execute an action associated with the user command.

10. The apparatus as claimed in claim 9, further wherein the processor is further configured to execute:

wherein the object detector component is configured to detect the object by comparing the third image to the background image and, if a part of the third image differs substantially from a corresponding part of the background image, designating the part of the third image as the detected object.

11. The apparatus as claimed in claim 9, wherein the change detector is configured to detect a change by determining a value of a parameter associated with the object and detecting if the value of the parameter changes in a later image of the sequence of images.

12. The apparatus as claimed in claim 9, wherein the gesture identifier component is further configured to:

determine a first dimension of the object in a first image of the sequence of images; and

determine a second dimension of the object in a second image of the sequence of images;

wherein determining one or more parameters comprises determining a distance change parameter by determining a change in magnitude between the first dimension and the second dimension.

13. The apparatus as claimed in claim 9, wherein the processor is further configured to execute:

a hand state detector component configured to determine a first hand state associated with the detected object if the detected object is stationary and a second hand state associated with the detected object after the detected change,

wherein the gesture identifier is further configured to generate a parameter indicating a change between the first hand state and the second hand state.