Enabling gesture input for controlling a presentation of content

• US 9,377,860 B1
• Filed: 12/19/2012
• Issued: 06/28/2016
• Est. Priority Date: 12/19/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Weber, Jason Robert, Bell, Matthew Paul, Noble, Isaac Scott, Strutt, Guenael Thomas, Polansky, Stephen Michael, Zhou, Dong
• Primary Examiner(s): Chang, Kent
• Assistant Examiner(s): Rosario, Nelson
• Application Number:
  US13/720,172
• Time in Patent Office: 1,287 Days
• Field of Search: 348148-155, 705/14.63, 715/863, 345/156, 345/165, 345/166, 345173-179, 178 1801- 1804
• US Class Current: 1/1
• CPC Class Codes: B60K 2360/146 Instrument input by gesture
• CPC Class Codes: B60K 2360/1464 3D-gesture
• CPC Class Codes: B60K 35/10 Input arrangements, i.e. from user to vehicle, associated with vehicle functions or specially adapted therefor
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G11B 27/102 Programmed access in sequence to addressed parts of tracks of operating record carriers access by moving the head G11B3/08, G11B5/54, G11B7/085, G11B21/022; by moving the record carrier G11B15/005, G11B17/005, by driving of both record carrier and head G11B15/1816

Abstract

Approaches are described for providing input to a portable computing device, such as a mobile phone. A user'"'"'s hand can be detected based on data (e.g., one or more images) obtained by at least one sensor of the device, such as camera, and the images can be analyzed to locate the hand of the user. As part of the location computation, the device can determine a motion being performed by the hand of the user, and the device can determine a gesture corresponding to the motion. In the situation where the device is controlling a media player capable of playing media content, the gesture can be interpreted by the device to cause the device to, e.g., pause a media track or perform another function with respect to the media content being presented via the device.

Claims

1. A computer implemented method for determining user input to a mobile computing device, the method comprising:

under control of one or more computer systems configured with executable instructions,

communicating with an automobile to determine that a mobile computing device is located within a detection range relative to the automobile;

determining a movement of the automobile;

causing the mobile computing device to activate an air gesture input mode in association with the mobile computing device;

disabling a touch input mode of the mobile computing device based at least in part on the movement of the automobile;

determining a movement of a user using an infrared detector of the mobile computing device, the infrared detector having a resolution;

activating a first front-facing camera of the mobile computing device, the first front-facing camera having a first resolution higher than the resolution of the infrared detector;

activating a second front facing camera of the mobile computing device, the second front-facing camera having a second resolution higher than the resolution of the infrared detector;

obtaining a first image using the first front-facing camera of the mobile computing device, the first front-facing camera having a first field of view;

obtaining a second image using the second front-facing camera of the mobile computing device, the second front-facing camera separated a distance from the first front-facing camera and having a second field of view that partially overlaps the first field of view;

analyzing the first image to determine a feature point corresponding to a representation of a hand of the user with respect to the mobile computing device, the hand being located within an area relative to the mobile computing device;

analyzing the second image to determine a same feature point;

determining a first location of the feature point in the first image and a second location of the same feature point in the second image;

determining a path of the feature point, the path corresponding to a gesture, the gesture being one of a plurality of gestures associated with controlling presentation of media content; and

causing a function to be performed with respect to the media content being presented, the function corresponding to the gesture.

2. The computer implemented method of claim 1, wherein determining the path of the feature point includes:

determining a wave of the hand of the user in a left-to-right direction or right-to-left direction relative to the mobile computing device, the wave of the hand being the gesture, wherein the function causes the media content to be skipped or repeated.

3. The computer implemented method of claim 1, further comprising:

determining that the mobile computing device is positioned within the automobile; and

pairing the mobile computing device with a media player of the automobile, wherein the function to be performed with respect to the media content is presented via the media player of the automobile.

4. The computer implemented method of claim 1, further comprising:

determining, based at least in part on the first location and the second location of the feature point, a relative location of the feature point with respect to the mobile computing device, the relative location being in three dimensions.

5. The computer implemented method of claim 4, wherein the gesture comprises one or more movements of the feature point in three dimensions.

6. A computer implemented method, comprising:

under control of one or more computer systems configured with executable instructions,

detecting a connection with an electronic media system of an automobile by a computing device;

determining a movement of the automobile;

causing the computing device to activate an air gesture input mode based at least in part on detecting the connection;

disabling a touch input mode of the computing device based at least in part on the movement of the automobile;

determining a movement of a user using an infrared detector of the computing device, the infrared detector having a resolution;

activating a first front-facing camera of the computing device, the first front-facing camera having a first resolution higher than the resolution of the infrared detector;

activating a second front-facing camera of the computing device, the second front-facing camera having a second resolution higher than the resolution of the infrared detector;

obtaining a first image using the first front-facing camera of the computing device, the first front-facing camera having a first field of view;

obtaining a second image using the second front-facing camera of the computing device, the second front-facing camera separated a distance from the first front-facing camera and having a second field of view that partially overlaps the first field of view;

analyzing the first image to determine a feature point corresponding to a representation of a hand of the user with respect to the computing device, the hand being located within an area relative to the computing device;

analyzing the second image to determine a same feature point;

determining a first location of the feature point in the first image and a second location of the same feature point in the second image;

determining a path of the feature point, the path corresponding to a motion being performed by the user;

determining a gesture corresponding to the motion, the gesture being one of a plurality of gestures corresponding to one or more actions to be performed by the computing device; and

performing the one or more actions corresponding to the gesture.

7. The computer implemented method of claim 6, wherein determining the gesture includes:

determining a forward motion of a hand of the user relative to the computing device, the forward motion of the hand being the gesture,

the gesture causing the computing device to pause a media track being presented.

8. The computer implemented method of claim 7, further comprising:

determining a subsequent forward motion of the hand of the user relative to the computing device; and

causing the media track being presented to resume play.

9. The computer implemented method of claim 6, wherein determining the gesture includes:

determining a shaking motion of the hand of the user relative to the computing device, the shaking motion of the hand being the gesture;

the gesture causing the computing device to activate a voice input mode;

detecting, using at least one sensor of the computing device, an audio command being spoken by the user; and

causing the computing device to perform a second function with respect to an application on the computing device based at least in part on the audio command.

10. The computer implemented method of claim 9, wherein the at least one sensor includes a camera, a capacitive sensor, an ultrasonic sensor, or combinations thereof.

11. The computer implemented method of claim 6, further comprising:

receiving an incoming call;

determining a left-to-right motion of the hand of the user relative to the computing device; and

causing the computing device to answer the incoming call.

12. The computer implemented method of claim 6, further comprising:

pairing the computing device with the automobile, wherein based on the pairing, the computing device is capable of controlling at least one of a media player or a navigation system of the automobile.

13. The computer implemented method of claim 6, wherein determining the gesture includes:

determining a counter-clockwise motion of a hand of the user relative to the computing device, the counter-clockwise motion of the hand being the gesture,

the gesture causing the computing device to rewind a media track being presented.

14. The computer implemented method of claim 6,

wherein determining the gesture includes:

determining a clockwise motion of a hand of the user relative to the computing device, the clockwise motion of the hand being the gesture,

the gesture causing the computing device to fast-forward a media track being presented.

15. The computer implemented method of claim 6, wherein determining the gesture includes:

determining, based at least in part on analyzing image information obtained from one or more images captured by the computing device, a position of the hand of the user with respect to the computing device; and

based at least in part on analyzing the image information and the position of the hand of the user, determining the gesture being performed by the hand of the user.

16. A computing system, comprising:

at least one processor; and

memory including instructions that, when executed by the processor, cause the computing system to;

detect a connection with an electronic media system of an automobile by a computing device;

determining a movement of the automobile;

cause the computing device to activate an air gesture input mode based at least in part on detecting the connection;

disabling a touch input mode of the computing device based at least in part on the movement of the automobile;

determining a movement of a user using an infrared detector of the computing device, the infrared detector having a resolution;

activating a first front-facing camera of the computing device, the first front-facing camera having a first resolution higher than the resolution of the infrared detector;

activating a second front-facing camera of the computing device, the second front-facing camera having a second resolution higher than the resolution of the infrared detector;

obtain a first image using the first front-facing camera of the computing device, the first front-facing camera having a first field of view;

obtain a second image using the second front-facing camera of the computing device, the second front-facing camera separated a distance from the first front-facing camera and having a second field of view that partially overlaps the first field of view;

analyze the first image to determine a feature point corresponding to a representation of a hand of the user with respect to the mobile computing device, the hand being located within an area relative to the mobile computing device;

analyze the second image to determine a same feature point;

determine a first location of the feature point in the first image and a second location of the same feature point in the second image;

determine a path of the feature point, the path corresponding to a gesture, the gesture being one of a plurality of gestures corresponding to one or more actions to be performed by the computing device; and

perform the one or more actions corresponding to the gesture.

17. The computing system of claim 16, wherein the instructions, when executed, further cause the computing device to:

determine a down-to-up motion of the hand of the user relative to the computing device, the down-to-up motion of the hand being the gesture,

the gesture causing the computing device to increase a volume of a media track being presented.

18. The computing system of claim 16, wherein the instructions, when executed, further cause the computing device to:

receive an incoming call;

determine the hand of the user being held over the computing device; and

cause the computing device, the computing device to reject the incoming call.

19. The computing system of claim 16, wherein the instructions, when executed, further cause the computing device to:

communicate with the automobile to establish the connection;

activate the air gesture input mode based at least in part on the connection being established; and

pair the computing device with at least one of a media player or a navigation system of the automobile, the computing device being able to control one of the media player or the navigation system of the automobile based at least in part on the pairing.

20. The computing system of claim 16, wherein the instructions, when executed, further cause the computing device to:

determine a right-to-left motion of the hand of the user relative to the computing device, the right-to-left motion of the hand being the gesture,

the gesture causing the computing device to repeat a media track being presented.

21. The computing system of claim 20, wherein the instructions, when executed, further cause the computing device to:

determine a subsequent right-to-left motion of the hand of the user relative to the computing device within a determined period of time of repeating the media track; and

cause the computing device to skip to a previous media track being presented.

22. A non-transitory computer readable storage medium storing one or more sequences of instructions executable by one or more processors to perform a set of operations comprising:

detecting a connection with an electronic media system of an automobile by a computing device;

determining a movement of the automobile;

causing the computing device to activate an air gesture input mode based at least in part on detecting the connection;

disabling a touch input mode of the computing device based at least in part on the movement of the automobile;

determining a movement of a user using an infrared detector of the computing device, the infrared detector having a resolution;

activating a first front-facing camera of the computing device, the first front-facing camera having a first resolution higher than the resolution of the infrared detector;

activating a second front-facing camera of the computing device, the second front-facing camera having a second resolution higher than the resolution of the infrared detector;

obtaining a first image using the first front-facing camera of the computing device, the first front-facing camera having a first field of view;

obtaining a second image using the second front-facing camera of the computing device, the second front-facing camera separated a distance from the first front-facing camera and having a second field of view that partially overlaps the first field of view;

analyzing the first image to determine a feature point corresponding to a representation of a hand of the user with respect to the computing device, the hand being located within a gesture area relative to the computing device;

analyzing the second image to determine a same feature point;

determining a first location of the feature point in the first image and a second location of the same feature point in the second image;

determining a path of the feature point, the path corresponding to a gesture, the gesture being one of a plurality of gestures corresponding to one or more actions to be performed by the computing device; and

performing the one or more actions corresponding to the gesture.

23. The non-transitory computer readable storage medium of claim 22, further comprising instructions executed by the one or more processors to perform the operations of:

determining a shaking motion of the hand of the user relative to the computing device, the shaking motion of the hand being the gesture, the gesture causing the computing device to activate a voice input mode;

detecting, using at least one sensor of the computing device, an audio command being spoken by the user; and

causing the computing device to perform a second function with respect to an application on the computing device based at least in part on the audio command.

24. The non-transitory computer readable storage medium of claim 22,

determine an up-to-down motion of the hand of the user relative to the computing device, the up-to-down motion of the hand being the gesture, the gesture causing the computing device to decrease a volume of a media track being presented via the computing device.

25. A computer implemented method for determining user input to a mobile computing device, the method comprising:

under control of one or more computer systems configured with executable instructions,

communicating with an automobile to determine that a mobile computing device is located within a detection range relative to the automobile;

causing the mobile computing device to activate an air gesture input mode in association with the mobile computing device;

disabling a touch input mode of the mobile computing device;

determining a movement of a user using an infrared detector of the mobile computing device, the infrared detector having a resolution;

activating a first front-facing camera of the mobile computing device the first front-facing camera having a first resolution higher than the resolution of the infrared detector;

activating a second front facing camera of the mobile computing device, the second front-facing camera having a second resolution higher than the resolution of the infrared detector;

obtaining a first image using the first front-facing camera of the mobile computing device, the first front-facing camera having a first field of view;

obtaining a second image using the second front-facing camera of the mobile computing device, the second front-facing camera separated a distance from the first front-facing camera and having a second field of view that partially overlaps the first field of view;

analyzing the first image to determine a feature point corresponding to a representation of a hand of the user with respect to the mobile computing device, the hand being located within an area relative to the mobile computing device;

analyzing the second image to determine a same feature point;

determining a first location of the feature point in the first image and a second location of the same feature point in the second image;

determining a path of the feature point, the path corresponding to a gesture, the gesture being one of a plurality of gestures associated with controlling presentation of media content; and

causing a function to be performed with respect to the media content being presented, the function corresponding to the gesture.