Computing device interface

• US 8,818,027 B2
• Filed: 01/25/2011
• Issued: 08/26/2014
• Est. Priority Date: 04/01/2010
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Forutanpour, Babak, Ren, Jianfeng
• Primary Examiner(s): Newman, Michael A
• Application Number:
  US13/013,657
• Publication Number:
  US 20110243380A1
• Time in Patent Office: 1,309 Days
• Field of Search: 382/103, 382/181, 382/209, 382/217, 345/156, 345/157, 345/175, 345/181, 345/182
• US Class Current: 382/103
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/0418 for error correction or compensation, e.g. based on parallax, calibration or alignment
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A computing device configured for providing an interface is described. The computing device includes a processor and instructions stored in memory. The computing device projects a projected image from a projector. The computing device also captures an image including the projected image using a camera. The camera operates in a visible spectrum. The computing device calibrates itself, detects a hand and tracks the hand based on a tracking pattern in a search space. The computing device also performs an operation.

Claims

1. A computing device configured for providing an interface, comprising:

a processor;

memory in electronic communication with the processor; and

instructions stored in the memory, the instructions being executable to;

project a projected image from a projector;

capture an image using a camera, wherein the camera operates in a visible spectrum;

calibrate the computing device;

detect a hand;

track the hand by scanning a search space within the captured image for a tracking pattern representative of the hand, the search space size being based on the hand size, the hand size being determined by a change in intensity of a portion of the captured image; and

perform an operation.

2. The computing device of claim 1, wherein calibrating the computing device comprises:

finding corners of the projected image; and

mapping coordinates of the projected image to a desktop space.

3. The computing device of claim 1, wherein calibrating the computing device comprises determining the hand size.

4. The computing device of claim 3, wherein the hand size is determined based on the projected image, wherein the projected image comprises a plurality of ellipses.

5. The computing device of claim 3, wherein the hand size is determined based on facial recognition.

6. The computing device of claim 1, wherein the hand is detected in an initial interaction area.

7. The computing device of claim 6, wherein the instructions are further executable to:

track a body; and

move the initial interaction area based on tracking the body.

8. The computing device of claim 1, wherein the instructions are further executable to determine that the tracking pattern is matched to a hand.

9. The computing device of claim 1, wherein the instructions are further executable to dynamically change the search space based on a motion of the hand.

10. The computing device of claim 1, wherein the tracking pattern comprises a rectangular center dark region adjoined on three sides by rectangular light outer regions and adjoined on one side by a rectangular dark outer region.

11. The computing device of claim 1, wherein the operation is performed based on tracking the hand.

12. The computing device of claim 1, wherein the instructions are further executable to recognize a hand gesture based on a gesture pattern.

13. The computing device of claim 12, wherein recognizing a hand gesture further comprises:

determining whether a tracked hand has stopped moving for a period of time; and

scaling a tracked hand image to a fixed size.

14. The computing device of claim 12, wherein the gesture pattern comprises a rectangular center dark region adjoined on two sides by rectangular light outer regions.

15. The computing device of claim 12, wherein the instructions are further executable to perform an operation based on the recognized hand gesture.

16. The computing device of claim 12, wherein the instructions are further executable to perform an operation when the hand gesture is recognized for a percentage of a number of frames.

17. The computing device of claim 12, wherein the hand gesture is recognized based on an angle between a thumb and index finger, and wherein the instructions are further executable to move a cursor away from the hand based on the angle between the thumb and index finger.

18. The computing device of claim 1, wherein the instructions are further executable to illuminate the projected image more brightly in a region around a hand.

19. The computing device of claim 1, wherein the instructions are further executable to track a plurality of hands.

20. The computing device of claim 1, wherein the instructions are further executable to recognize a plurality of hand gestures.

21. The computing device of claim 1, wherein the instructions are further executable to remove dark portions of the image before tracking the hand to avoid tracking a shadow of the hand.

22. The computing device of claim 1, wherein the instructions are further executable to provide improved focus and exposure of the image in an area of the projected image.

23. The computing device of claim 15, wherein the operation releases a cursor.

24. A method for providing an interface, comprising:

projecting a projected image from a projector;

capturing an image using a camera, wherein the camera operates in a visible spectrum;

calibrating a computing device;

detecting, by the computing device, a hand;

tracking, by the computing device, the hand by scanning a search space within the captured image for a tracking pattern representative of the hand, the search space size being based on the hand size, the hand size being determined by a change in intensity of a portion of the captured image; and

performing an operation by the computing device.

25. The method of claim 24, wherein calibrating the computing device comprises:

finding corners of the projected image; and

mapping coordinates of the projected image to a desktop space.

26. The method of claim 24, wherein calibrating the computing device comprises determining the hand size.

27. The method of claim 26, wherein the hand size is determined based on the projected image, wherein the projected image comprises a plurality of ellipses.

28. The method of claim 26, wherein the hand size is determined based on facial recognition.

29. The method of claim 24, wherein the hand is detected in an initial interaction area.

30. The method of claim 29, further comprising:

tracking a body; and

moving the initial interaction area based on tracking the body.

31. The method of claim 24, further comprising determining that the tracking pattern is matched to a hand.

32. The method of claim 24, further comprising dynamically changing the search space dynamically based on a motion of the hand.

33. The method of claim 24, wherein the tracking pattern comprises a rectangular center dark region adjoined on three sides by rectangular light outer regions and adjoined on one side by a rectangular dark outer region.

34. The method of claim 24, wherein the operation is performed based on tracking the hand.

35. The method of claim 24, further comprising recognizing a hand gesture based on a gesture pattern.

36. The method of claim 35, wherein recognizing a hand gesture further comprises:

determining whether a tracked hand has stopped moving for a period of time; and

scaling a tracked hand image to a fixed size.

37. The method of claim 35, wherein the gesture pattern comprises a rectangular center dark region adjoined on two sides by rectangular light outer regions.

38. The method of claim 35, further comprising performing an operation based on the recognized hand gesture.

39. The method of claim 35, further comprising performing an operation when the hand gesture is recognized for a percentage of a number of frames.

40. The method of claim 35, wherein the hand gesture is recognized based on an angle between a thumb and index finger, and the method further comprises moving a cursor away from the hand based on the angle between the thumb and index finger.

41. The method of claim 24, further comprising illuminating the projected image more brightly in a region around a hand.

42. The method of claim 24, further comprising tracking a plurality of hands.

43. The method of claim 24, further comprising recognizing a plurality of hand gestures.

44. The method of claim 24, further comprising removing dark portions of the image before tracking the hand to avoid tracking a shadow of the hand.

45. The method of claim 24, further comprising providing improved focus and exposure of the image in an area of the projected image.

46. The method of claim 38, wherein the operation releases a cursor.

47. A computer-program product for providing an interface, the computer-program product comprising a non-transitory tangible computer-readable medium having instructions thereon, the instructions comprising:

code for causing a computing device to project a projected image from a projector;

code for causing the computing device to capture an image using a camera, wherein the camera operates in a visible spectrum;

code for causing the computing device to calibrate the computing device;

code for causing the computing device to detect a hand;

code for causing the computing device to track the hand by scanning a search space within the captured image for a tracking pattern representative of the hand, the search space size being based on the hand size, the hand size being determined by a change in intensity of a portion of the captured image; and

code for causing the computing device to perform an operation.

48. The computer-program product of claim 47, wherein the hand is detected in an initial interaction area.

49. The computer-program product of claim 47, wherein the instructions further comprise code for causing the computing device to recognize a hand gesture based on a gesture pattern.

50. An apparatus for providing an interface, comprising:

means for projecting a projected image from a projector;

means for capturing an image using a camera, wherein the camera operates in a visible spectrum;

means for calibrating the apparatus;

means for detecting a hand;

means for tracking the hand by scanning a search space within the captured image for a tracking pattern representative of the hand, the search space size being based on the hand size, the hand size being determined by a change in intensity of a portion of the captured image; and

means for performing an operation.

51. The apparatus of claim 50, wherein the hand is detected in an initial interaction area.

52. The apparatus of claim 50, further comprising means for recognizing a hand gesture based on a gesture pattern.