Movement recognition as input mechanism

• US 8,788,977 B2
• Filed: 12/10/2008
• Issued: 07/22/2014
• Est. Priority Date: 11/20/2008
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Bezos, Jeffrey P.
• Primary Examiner(s): Kim, Matt
• Assistant Examiner(s): TRAPANESE, WILLIAM C
• Application Number:
  US12/332,049
• Publication Number:
  US 20100125816A1
• Time in Patent Office: 2,050 Days
• Field of Search: 715/863
• US Class Current: 715/863
• CPC Class Codes: G01C 21/367 Details, e.g. road map scale, orientation, zooming, illumination, level of detail, scrolling of road map or positioning of current position marker
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus

Abstract

The detection of relative motion or orientation between a user and a computing device can be used to control aspects of the device. For example, the computing device can include an imaging element and software for locating positions, shapes, separations, and/or other aspects of a user'"'"'s facial features relative to the device, such that an orientation of the device relative to the user can be determined. A user then can provide input to the device by performing actions such as tilting the device, moving the user'"'"'s head, making a facial expression, or otherwise altering an orientation of at least one aspect of the user with respect to the device. Such an approach can be used in addition to, or as an alternative to, conventional input devices such as keypads and touch screens.

Claims

1. A method of providing input to a virtual keyboard displayed on a portable computing device, comprising:

obtaining a first image using an imaging element of the portable computing device, the first image including a facial feature of a user of the portable computing device;

obtaining a second image using the imaging element, the second image including the facial feature of the user of the portable computing device;

analyzing the obtained first image and second image, with the portable computing device, to determine a change in the position of the facial feature of the user with respect to the portable computing device;

using the determined change, of the position of the facial feature with respect to the portable computing device, as input to perform a selection of one of a plurality of alphanumeric characters of the virtual keyboard, the selection corresponding to a position of a cursor configured to move in at least two degrees of freedom with respect to the virtual keyboard;

altering information displayed within a graphical user interface including the virtual keyboard based at least in part on the determined change;

in response to the selection, reducing a movement capability of the cursor with respect to the virtual keyboard by one degree of freedom such that the cursor is limited to being positioned with respect to a subset of the plurality of alphanumeric characters of the virtual keyboard corresponding to the reduced movement capability, the information displayed within the graphical user interface excluding a representation of either or both of the obtained first and second images;

determining a subsequent change of the position of the facial feature with respect to the portable computing device, wherein the subsequent change, of the position of the facial feature with respect to the portable computing device, corresponds to the one degree of freedom by which the movement capability of the cursor is reduced;

preventing use of data for the subsequent change, corresponding to the one degree of freedom, as input for performing a second selection using the cursor while the movement capability of the cursor is reduced;

determining an additional change of the position of the facial feature with respect to the portable computing device after the selection is performed, wherein the additional change, of the position of the facial feature with respect to the portable computing device, is outside the one degree of freedom by which the movement capability of the cursor is reduced; and

utilizing data for the additional change as input for performing the second selection using the cursor while the movement capability of the cursor is reduced by the one degree of freedom.

2. The method according to claim 1, wherein:

obtaining the second image is in response to a movement of the portable computing device.

3. The method according to claim 1, further comprising:

analyzing the obtained first and second images to determine a change in a set of measurable aspects corresponding to facial features of the user of the computing device,

wherein altering the information displayed within the graphical user interface is based at least in part on the determined change of the set of measurable aspects.

4. The method according to claim 3, wherein the set of measurable aspects includes at least one of a dimension of at least one of the user'"'"'s eyes, a separation of features on the user'"'"'s face, a relative location of a feature on the user'"'"'s face, and a relative orientation between features on the user'"'"'s face.

5. The method according to claim 1, wherein:

analyzing the obtained first image and second image to determine a change in the position of the facial feature of the user with respect to the portable computing device includes determining a change in relative orientation between the user and the portable computing device.

6. The method according to claim 1, further comprising:

obtaining a third image using the imaging element, the third image including the facial feature of the user of the portable computing device;

analyzing the obtained second image and third image, with the portable computing device, to determine a second change in the position of the facial feature of the user with respect to the portable computing device, the second change based at least in part on an expression of the user adjusting at least one of a shape, position, and dimension of the facial feature of the user; and

altering the information displayed within the graphical user interface based at least in part on the second change.

7. The method according to claim 1, wherein:

altering the information displayed within the graphical user interface includes moving the cursor over the virtual keyboard in a direction corresponding to the reduced movement capability.

8. A method for selecting an alphanumeric character in a computing device, comprising:

analyzing first image information of a user of the computing device, the first image information obtained using an imaging element of the computing device, to detect a first position of the computing device at a first distance from the user;

in response to detecting the first position of the computing device at a first distance from the user, identifying a first alphanumeric character of a virtual keyboard available for selection;

receiving a selection of the first alphanumeric character;

in response to receiving the selection, reducing the movement capability of a cursor displayed on the computing device by one degree of freedom with respect to the virtual keyboard, the cursor being configured to move in at least two degrees of freedom with respect to the virtual keyboard before receiving the selection;

determining a subsequent change in at least one of a position of the computing device with respect to the user or a distance from the user at which the computing device is, wherein the subsequent change, in the at least one of the position of the computing device with respect to the user or the distance from the user at which the computing device is, corresponds to the one degree of freedom by which the movement capability of the cursor is reduced;

preventing use of data for the subsequent change, corresponding to the one degree of freedom, as input for performing a selection using the cursor while the movement capability of the cursor is reduced;

analyzing second image information of the user of the computing device, the second image information obtained using the imaging element of the computing device, to detect a second position of the computing device at a second distance from the user, wherein the second position indicates an additional change of the position of the user with respect to the computing device after receiving the selection; and

in response to detecting the second position of the computing device at a second distance from the user, utilizing the additional change of the position of the user with respect to the computing device to identify a second alphanumeric character of the virtual keyboard available for selection.

9. The method according to claim 8, further comprising:

displaying a set of characters of the virtual keyboard using a display element of the computing device; and

at a time when the cursor is in a position to select at least one character in the set of characters, performing the step of receiving the selection of the at least one character.

10. The method according to claim 9, further comprising:

segmenting the set of characters of the virtual keyboard into a plurality of subsets of characters, wherein the selection of at least one character in the set of characters comprises receiving a selection of at least one of the subsets of characters; and

segmenting the selection of the at least one subset of characters into a second plurality of subsets of characters.

11. The method according to claim 8, further comprising:

using an orientation detection element of the computing device in conjunction with analyzing the image information to determine the first position of the computing device;

wherein the cursor is configured to be directed toward the at least one character to be selected according to the first position of the computing device.

12. The method according to claim 11, wherein the orientation detection element is at least one of an accelerometer, a motion detector, an electronic compass, a global-positioning system receiver, and an electronic level.

13. The method according to claim 8, wherein reducing the movement capability of the cursor by one degree of freedom reduces the degrees of freedom from two degrees to one degree.

14. The method according to claim 8 further comprising:

in response to detecting a change in the orientation of the computing device, identifying a third alphanumeric character of the virtual keyboard available for selection, wherein the change in orientation corresponds to the user tilting the computing device.

15. The method according to claim 8, wherein:

identifying the first alphanumeric character available for selection includes at least one of;

hovering the cursor over the first alphanumeric character;

changing a color of the first alphanumeric character;

changing a shape of the first alphanumeric character;

changing a background color around the first alphanumeric character; and

displaying a shape around the first alphanumeric character.

16. A method of interacting with a computing device, comprising:

analyzing image information obtained using an imaging element of the computing device to determine a first relative orientation of at least one feature of a user to the computing device;

analyzing device orientation information obtained using an orientation detection element of the computing device to determine a second relative orientation of the computing device with respect to a surrounding environment;

detecting a change in at least one of the first and second relative orientations;

in response to detecting the change, performing a desired function corresponding to a position of a cursor configured to move in at least two degrees of freedom with respect to a menu rendered on a display of the computing device;

in response to performing the desired function, reducing a movement capability of the cursor by one degree of freedom with respect to the menu;

determining a subsequent change in a relative orientation of the computing device, the subsequent change in the relative orientation corresponding to the one degree of freedom by which the movement capability of the cursor is reduced;

preventing use of data for the subsequent change, corresponding to the one degree of freedom, as input for performing a selection using the cursor while the movement capability of the cursor is reduced;

determining an additional change in a relative orientation of the computing device outside the one degree of freedom by which the movement capability of the cursor is reduced; and

utilizing data for the additional change as input for performing the selection using the cursor while the movement capability of the cursor is reduced by the one degree of freedom.

17. The method according to claim 16, wherein:

analyzing image information includes capturing at least one image from the portable computing device to determine the position of the at least one feature of a user with respect to the computing device, and

detecting a change in at least one of position of the at least one feature of the user and the orientation of the computing device includes capturing at least one additional image from the portable computing device and determining a difference in position of the at least one feature.

18. The method according to claim 16, wherein:

the step of performing a desired function for the detected change includes altering the information displayed within a graphical user interface based at least in part on the detected change, wherein the graphical user interface of the device excludes a representation of the image information.

19. The method according to claim 16, further comprising:

determining whether the detected change at least meets an input threshold corresponding to the change before determining the desired function.

20. The method according to claim 16, wherein:

the orientation detection element is one of an accelerometer, a motion detector, an electronic compass, and an electronic level.

21. The method according to claim 16, wherein:

the at least one feature corresponding to the user includes at least one facial feature of the user.

22. The method according to claim 16, wherein:

analyzing the image information to determine the first relative location of the at least one feature of the user to the computing device includes determining an approximate location of each eye of the user in the image information.

23. The method according to claim 16, wherein:

performing the desired function includes at least one of viewing and selecting an item.

24. The method according to claim 16, further comprising:

analyzing the image information to determine an identity of the user.

25. The method according to claim 24, further comprising:

determining a plurality of inputs corresponding to a plurality of changes for the determined identity of the user.

26. The method according to claim 24, further comprising:

capturing calibration information for each determined identity before analyzing the image information.

27. The method according to claim 16, further comprising:

prompting the user to confirm the desired function before performing the function.

28. The method according to claim 27, further comprising:

enabling the user to confirm the input by changing at least one of the first and second relative orientations.

29. A computing device, comprising:

a processor;

a display element for displaying a graphical user interface;

a memory device including instructions operable to be executed by the processor to perform a set of actions; and

an imaging element operable to obtain image information corresponding to a relative viewing area of the computing device,

the computing device being operable to;

obtain a first image and a second image using the imaging element, the first and second images each including a position of at least one of eyes, a mouth, and a head of a user of the computing device;

analyze the obtained first image and second image to determine a change in the positions of at least one of the eyes, the mouth, and the head of the user with respect to the computing device;

use the determined change as input to perform a selection of one of a plurality of alphanumeric characters displayed as part of a menu through the graphical user interface, the selection corresponding to a position of a cursor configured to move in at least two degrees of freedom with respect to the menu;

in response to the selection, reduce a movement capability of the cursor by one degree of freedom with respect to the menu such that the cursor is limited to being positioned with respect to a subset of the plurality of alphanumeric characters of the menu, the choices of the alphanumeric characters in the subset being limited by the reduced movement capability, the information displayed within the graphical user interface excluding a representation of the obtained first image, the obtained second image, or both the obtained first and second images;

determine a subsequent change in the positions of at least one of the eyes, the mouth, or the head of the user with respect to the computing device, wherein the subsequent change, in the positions of the at least one of the eyes, the mouth, or the head of the user with respect to the computing device, corresponds to the one degree of freedom by which the movement capability of the cursor is reduced;

prevent use of data for the subsequent change, corresponding to the one degree of freedom, as input for performing a selection using the cursor while the movement capability of the cursor is reduced;

determine an additional change in the positions of at least one of the eyes, the mouth, or the head of the user with respect to the computing device, wherein the additional change is outside the one degree of freedom by which the movement capability of the cursor is reduced; and

utilize data for the additional change as input for performing the selection using the cursor while the movement capability of the cursor is reduced by the one degree of freedom.

30. A method of receiving input for a computing device, comprising:

analyzing image information to determine at least one relative location of at least one feature of a user with respect to the computing device, the image information being obtained using an imaging element of the computing device;

in response to detecting a change in the at least one relative location of at least one feature of the user that at least meets an input threshold corresponding to the change, selecting an input function corresponding to the detected change and executing the selected input function, the execution corresponding to a position of a cursor configured to move in at least two degrees of freedom with respect to a directory displayed on a display of the computing device;

in response to executing the selected input, reducing a movement capability of the cursor by one degree of freedom with respect to the directory, wherein the user is able to define the input function corresponding to the detected change;

determining a subsequent change in at least one relative location of at least one feature of the user with respect to the computing device, wherein the subsequent change, in the at least one relative location of the at least one feature of the user with respect to the computing device, corresponds to the one degree of freedom by which the movement capability of the cursor is reduced;

preventing use of data for the subsequent change, corresponding to the one degree of freedom, as input for performing a selection using the cursor while the movement capability of the cursor is reduced;

determining an additional change in at least one relative location of at least one feature of the user with respect to the computing device, wherein the additional change is outside the one degree of freedom by which the movement capability of the cursor is reduced; and

utilizing data for the additional change as input for performing the selection using the cursor while the movement capability of the cursor is reduced by the one degree of freedom.

31. The method according to claim 30, wherein:

detecting the change in the at least one relative location of at least one feature of the user includes detecting a change in at least one facial feature of the user.

32. The method according to claim 31, wherein:

detecting the change in the at least one relative location of at least one feature of the user includes detecting a change in at least one of a plurality of relative locations for the at least one facial feature of the user, at least one facial feature of the user having a plurality of relative locations associated therewith such that a change in expression for that facial feature results in a change in at least one relative position of the facial feature.

33. The method according to claim 30, wherein:

selecting the input function corresponding to the detected change and executing the selected input function includes altering information displayed within a graphical user interface associated with the computing device based at least in part on the detected change, wherein the graphical user interface of the computing device excludes a representation of the image information.

34. A computing device, comprising:

a processor;

a display element;

an imaging element operable to capture image information, the imaging element being oriented such that a user viewing the display element will be at least partially captured in the image information; and

a memory device including instructions that, when executed by the processor, cause the processor to;

analyze the image information to determine a set of measurable aspects corresponding to facial features of the user of the computing device;

determine a relative orientation of the user with respect to the computing device based at least in part upon values for the set of measurable aspects;

determine a change in the relative orientation of the user with respect to the computing device, the change in orientation being based at least in part on a change in position of the computing device;

perform a function corresponding to the determined change in the relative orientation and corresponding to a position of a cursor configured to move in at least two degrees of freedom with respect to an arranged set of characters displayed on the display element;

in response to performing the function, reduce a movement capability of the cursor by one degree of freedom with respect to the arranged set of characters such that the user of the computing device is limited to defining a function to be performed in response to each of a plurality of determinable changes in the relative orientation;

determine a subsequent change in the relative orientation of the user with respect to the computing device, wherein the subsequent change, in the relative orientation of the user with respect to the computing device, corresponds to the one degree of freedom by which the movement capability of the cursor is reduced;

prevent use of data for the subsequent change, corresponding to the one degree of freedom, as input for performing a selection using the cursor while the movement capability of the cursor is reduced;

determine an additional change in the relative orientation of the user with respect to the computing device, wherein the additional change is outside the one degree of freedom by which the movement capability of the cursor is reduced; and

utilizing data for the additional change as input for performing the selection using the cursor while the movement capability of the cursor is reduced by the one degree of freedom.

35. The computing device according to claim 34 wherein the memory device further includes instructions that, when executed by the processor, cause the processor to:

determine whether the change at least meets an input threshold corresponding to the change before performing the action.

36. The computing device according to claim 34, wherein:

the step of performing the function includes altering the information displayed within a graphical user interface on the display element of the computing device based at least in part on the determined change, wherein the information displayed within the graphical user interface of the device excludes a representation of the image information.

37. A computer program product embedded in a non-transitory computer-readable medium for controlling a computing device, comprising:

program code for analyzing first image information of a user to determine a first position of the computing device, the first position being a first distance away from the user and the first image information being obtained using at least one imaging element of the computing device positioned such that a representation of the user of the computing device is at least partially included in the image information;

program code for determining, in response to detecting a change in the first position of the computing device at the first distance away from the user, whether the change at least meets an input threshold corresponding to the change;

program code for determining an input to perform a selection of one of a plurality of alphanumeric characters displayed as part of a window panel through the graphical user interface, the determined input corresponding to a detected change that at least meets the input threshold, and executing an action on the computing device in response to the determined input, wherein a movement capability of a cursor with respect to the window panel is reduced by one degree of freedom with respect to the window panel in response to executing the action;

program code for determining a subsequent change in the first position of the computing device at the first distance away from the user, wherein the subsequent change, in the first position of the computing device at the first distance away from the user, corresponds to the one degree of freedom by which the movement capability of the cursor is reduced;

program code for preventing use of data for the subsequent change, corresponding to the one degree of freedom, as input for performing a selection using the cursor while the movement capability of the cursor is reduced; and

program code for analyzing second image information of the user to determine a second position of the computing device, the second position being a second distance away from the user and the second image information being obtained using at least one imaging element of the computing device, wherein the second position indicates an additional change of the position of the user with respect to the computing device,

wherein the second position is further away from the user than the first position and the program code for executing the action on the computing device includes altering information displayed within a graphical user interface on a display element of the computing device based at least in part on the detected change and the additional change, the information displayed on the graphical user interface of the computing device excluding a representation of the user.

38. The computer program product according to claim 37, further comprising:

program code for determining an orientation of the portable computing device, the portable computing device including at least one element operable to determine the orientation; and

program code for determining a change in the orientation of the computing device,

wherein the program code for determining whether the change at least meets an input threshold corresponding to the change includes instructions for analyzing both changes in orientation and changes in relative location of at least one feature of the user.

39. The computer program product according to claim 38, wherein:

a user of the computing device is able to define an action to be executed in response to each of a plurality of detectable changes.