Camera Gestures for User Interface Control

• US 20090309765A1
• Filed: 06/11/2008
• Published: 12/17/2009
• Est. Priority Date: 06/11/2008
• Status: Active Grant

• Current Assignee: RPX Corporation
• Original Assignee: Nokia Corporation
• Inventors: Shi, Yuanchun, Wang, Kongqiao, Zhang, Liang
• Granted Patent: US 8,269,842 B2
• Time in Patent Office: Days
• US Class Current: 341/20
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

A system and method for using images captured from a digital camera to control navigation through a three-dimensional user interface. The sequence of images may be examined to identify feature points to be tracked through successive frames of the images captured by the camera. A plurality of classifiers may be used to discern shift from rotation gestures, based on expected behavior of feature points in the image when the camera is shifted or rotated in position. The various classifiers may generate voting values for shift and rotation gestures, and the system can use historical gesture information to assist in categorizing a current gesture.

Claims

1. A method, comprising:

identifying a plurality of feature points from a current video image from a camera;

comparing positions of said feature points with corresponding positions of said feature points from a prior video image from said camera;

determining, from changes in said positions of said feature points, three-dimensional movement of the camera; and

outputting an identification of the three-dimensional movement of the camera as a gesture output.

2. The method of claim 1, wherein said determining differentiates between shift and rotation gestures of said camera by applying a plurality of classifiers to differences in positions of said feature points between said current and prior images.

3. The method of claim 2, wherein said plurality of classifiers includes a first classifier based on whether a plurality of feature points shares a common movement distance.

4. The method of claim 2, wherein said plurality of classifiers includes a second classifier based on whether different ones of said feature points moved by different distances.

5. The method of claim 2, wherein said plurality of classifiers includes a third classifier based on an average moving speed of said feature points.

6. The method of claim 1, further comprising:

counting the number of identified feature points in the current video image that were also found in the prior video image;

comparing the number of identified feature points with a predetermined minimum number of feature points; and

updating the feature points by identifying one or more new feature points in the current video image if the number of identified feature points falls below the predetermined minimum.

7. The method of claim 6, wherein said determining differentiates between shift and rotation gestures of said camera based on how often said updating is performed.

8. The method of claim 1, further comprising:

determining that a movement of the camera was a shift or rotation movement; and

recording a corresponding shift or rotation movement in a movement history of the camera, but outputting a gesture different from the movement recorded in the movement history.

9. The method of claim 8, wherein

if the movement was determined to be a shift movement, identifying the movement as a rotation movement in said outputting;

and

if the movement was determined to be a rotation movement, identifying the movement as a shift movement in said outputting.

10. The method of claim 2, wherein said plurality of classifiers includes:

a first classifier based on whether a plurality of the feature points have a common movement distance;

a second classifier based on a difference in movement distance for two of said feature points;

a third classifier based on how often new feature points are required; and

a fourth classifier based on an average movement speed of said feature points.

11. The method of claim 1, further comprising:

using a cellular telephone camera to navigate through a three-dimensional user interface on a computer.

12. The method of claim 1, wherein said determining determines that the three-dimensional movement is a zoom movement.

13. An apparatus, comprising:

a video input configured to receive video image data from a camera; and

a processor, configured to perform the following;

identifying a plurality of feature points from a current video image from said camera;

comparing positions of said feature points with corresponding positions of said feature points from a prior video image from said camera;

determining, from changes in said positions of said feature points, three-dimensional movement of the camera; and

outputting an identification of the three-dimensional movement of the camera as a gesture output.

14. The apparatus of claim 13, wherein said determining differentiates between shift and rotation gestures of said camera by applying a plurality of classifiers to differences in positions of said feature points between said current and prior images.

15. The apparatus of claim 14, wherein said plurality of classifiers includes a first classifier based on whether a plurality of feature points shares a common movement distance.

16. The apparatus of claim 14, wherein said plurality of classifiers includes a second classifier based on whether different ones of said feature points moved by different distances.

17. The apparatus of claim 14, wherein said plurality of classifiers includes a third classifier based on an average moving speed of said feature points.

18. The apparatus of claim 13, said processor being further configured to perform the following:

counting the number of identified feature points in the current video image that were also found in the prior video image;

comparing the number of identified feature points with a predetermined minimum number of feature points; and

updating the feature points by identifying one or more new feature points in the current video image if the number of identified feature points falls below the predetermined minimum.

19. The apparatus of claim 18, wherein said determining differentiates between shift and rotation gestures of said camera based on how often said updating is performed.

20. The apparatus of claim 13, said processor being further configured to perform the following:

determine that a movement of the camera was a shift or rotation movement; and

record a corresponding shift or rotation movement in a movement history of the camera, but output a gesture different from the movement recorded in the movement history.

21. The apparatus of claim 20, wherein

if the movement was determined to be a shift movement, identifying the movement as a rotation movement in said outputting;

and

if the movement was determined to be a rotation movement, identifying the movement as a shift movement in said outputting.

22. The apparatus of claim 14, wherein said plurality of classifiers includes:

a first classifier based on whether a plurality of the feature points have a common movement distance;

a second classifier based on a difference in movement distance for two of said feature points;

a third classifier based on how often new feature points are required; and

a fourth classifier based on an average movement speed of said feature points.

23. The apparatus of claim 13, wherein said apparatus is a computer communicatively coupled to said camera, and said camera is part of a computer mouse input device.

24. The apparatus of claim 13, wherein said determining determines that the three-dimensional movement is a zoom movement.

25. A computer-readable medium storing computer-executable instructions for performing the following:

identifying a plurality of feature points from a current video image from a camera;

comparing positions of said feature points with corresponding positions of said feature points from a prior video image from said camera;

determining, from changes in said positions of said feature points, three-dimensional movement of the camera; and

outputting an identification of the three-dimensional movement of the camera as a gesture output.

26. The computer-readable medium of claim 25, wherein said determining differentiates between shift and rotation gestures of said camera by applying a plurality of classifiers to differences in positions of said feature points between said current and prior images.

27. The computer-readable medium of claim 26, wherein said plurality of classifiers includes a first classifier based on whether a plurality of feature points shares a common movement distance.

28. The computer-readable medium of claim 26, wherein said plurality of classifiers includes a second classifier based on changes in relative positioning between pairs of said feature points.

29. The computer-readable medium of claim 26, wherein said plurality of classifiers includes a third classifier based on an average moving speed of said feature points.

30. The computer-readable medium of claim 25, further storing computer-executable instructions for performing the following:

counting the number of identified feature points in the current video image that were also found in the prior video image;

comparing the number of identified feature points with a predetermined minimum number of feature points; and

updating the feature points by identifying one or more new feature points in the current video image if the number of identified feature points falls below the predetermined minimum.

31. The computer-readable medium of claim 30, wherein said determining differentiates between shift and rotation gestures of said camera based on how often said updating is performed.

32. The computer-readable medium of claim 25, further storing computer-executable instructions for performing the following:

determining that a movement of the camera was a shift movement; and

recording a corresponding shift movement in a movement history of the camera;

but

identifying the movement as a rotation movement in said outputting.

33. The computer-readable medium of claim 26, wherein said plurality of classifiers includes:

a first classifier based on whether a plurality of the feature points have a common movement distance;

a second classifier based on a difference in relative positioning between pairs of said feature points;

a third classifier based on how often new feature points are required; and

a fourth classifier based on an average movement speed of said feature points.

34. The computer-readable medium of claim 25, further storing computer-executable instructions for performing the following:

determining that the movement of the camera was a zoom movement.

35. The computer-readable medium of claim 25, further storing computer-executable instructions for performing the following:

determining that a movement of the camera was a shift or rotation movement; and

recording a corresponding shift or rotation movement in a movement history of the camera, but outputting a gesture different from the movement recorded in the movement history.

36. The computer-readable medium of claim 35, wherein the computer-executable instructions further cause the following to occur when executed:

if the movement was determined to be a shift movement, identifying the movement as a rotation movement in said outputting; and

if the movement was determined to be a rotation movement, identifying the movement as a shift movement in said outputting.