CAMERA-BASED USER INPUT FOR COMPACT DEVICES

• US 20080273755A1
• Filed: 05/02/2008
• Published: 11/06/2008
• Est. Priority Date: 05/04/2007
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: GestureTek Incorporated (Winning Brands Corp.)
• Inventors: HILDRETH, Evan
• Granted Patent: US 8,472,665 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: A63F 2300/1075 using a touch screen
• CPC Class Codes: A63F 2300/6045 for mapping control signals received from the input arrangement into game commands
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0233 Character input methods
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: H04M 1/236 including keys on side or rear faces
• CPC Class Codes: H04M 1/72427 for supporting games or graphical animations
• CPC Class Codes: H04M 2250/52 including functional features of a camera

Abstract

A camera is used to detect a position and/or orientation of an object such as a user'"'"'s finger as an approach for providing user input, for example to scroll through data, control a cursor position, and provide input to control a video game based on a position of a user'"'"'s finger. Input may be provided to a handheld device, including, for example, cell phones, video games systems, portable music (MP3) players, portable video players, personal data assistants (PDAs), audio/video equipment remote controls, and consumer digital cameras, or other types of devices.

Claims

1. A computer-implemented method comprising:

detecting an orientation of a finger of a user'"'"'s hand in an image of the user'"'"'s hand interacting with an electronic device using an anatomical model of a finger;

mapping the detected orientation to a control input; and

controlling an application based on the control input.

2. The method of claim 1, further comprising detecting a position of the finger of the user'"'"'s hand in the image of the user'"'"'s hand interacting with the electronic device, wherein the control input is mapped based on the detected position.

3. The method of claim 2, wherein:

the control input defines a position and an orientation in a virtual environment that corresponds to the detected position and detected orientation of the finger; and

controlling the application based on the control input comprises;

controlling a virtual object in the virtual environment to move to the position and the orientation in the virtual environment that corresponds to the detected position and detected orientation of the finger; and

rendering a display of the virtual object at the position and the orientation in the virtual environment that corresponds to the detected position and detected orientation of the finger.

4. The method of claim 1, wherein detecting the orientation of the finger of the user'"'"'s hand in the image of the user'"'"'s hand interacting with the electronic device using the anatomical model of the finger comprises identifying a first part of the finger that corresponds to the fingertip, identifying a second part of the finger that corresponds to a base of the finger, calculating a vector between the first part of the finger and the second part of the finger, and determining the orientation of the finger based on the vector.

5. The method of claim 1, wherein detecting the orientation of the finger of the user'"'"'s hand in the image of the user'"'"'s hand interacting with the electronic device using the anatomical model of the finger comprises:

acquiring an image of the finger, and

edge processing the image.

6. The method of claim 5, further comprising segmenting the image in scanlines having an orientation that is substantially perpendicular to the finger.

7. The method of claim 1, further comprising detecting contours around the finger.

8. The method of claim 7, wherein the contours around the finger are detected by calculating a gradient of image intensity within the image and identifying at least one of local minima and local maxima in the gradient of image intensity.

9. The method of claim 1, wherein detecting the orientation of the finger of the user'"'"'s hand in the image of the user'"'"'s hand interacting with the electronic device using the anatomical model of the finger comprises identifying an area of the image that corresponds to creases in skin of the finger and ignoring the identified area of the image that corresponds to the creases in the skin of the finger in detecting the finger within the image.

10. The method of claim 1, wherein detecting the orientation of the finger of the user'"'"'s hand in the image of the user'"'"'s hand interacting with the electronic device using the anatomical model of the finger comprises selecting an object in the image as the finger based on a size of the object, a shape of the object, and whether the object intersects a border of the image.

11. The method of claim 1 wherein detecting the orientation of the finger using the anatomical model comprises detecting the orientation of the finger using an anatomical model of a finger oriented in a plane parallel to a display of the electronic device, the anatomical model being a model of a finger extending from a base of the finger to a tip of the finger.

12. A device comprising:

an image detector configured to detect an orientation of a finger of a user'"'"'s hand in an image of the user'"'"'s hand interacting with an electronic device using an anatomical model of a finger; and

a processor configured to map the detected orientation to a control input and control an application based on the control input.

13. The device of claim 12, wherein the image detector is disposed in a mobile phone, a portable video game system, a portable music player, a personal data assistant, a remote control or a digital camera.

14. A computer readable medium comprising a computer program product, the computer program product comprising instructions that, when executed, operate to cause a computer to perform operations comprising:

detecting an orientation of a finger of a user'"'"'s hand in an image of the user'"'"'s hand interacting with an electronic device using an anatomical model of a finger;

mapping the detected orientation to a control input; and

controlling an application based on the control input.

15. A computer-implemented method comprising:

accessing an image of a user'"'"'s hand interacting with an electronic device;

identifying an object in the image as a finger of the user'"'"'s hand based on a size of the object, a shape of the object, and whether the object intersects a border of the image;

determining a position of the object identified in the image as the finger of the user'"'"'s hand;

mapping the detected position to a control input; and

controlling an application based on the control input.

16. The method of claim 15 wherein identifying the object in the image as the finger of the user'"'"'s hand comprises identifying the object in the image as the finger of the user'"'"'s hand based on the object having an elongated shape that corresponds to an anatomical model of a finger.

17. The method of claim 15 wherein identifying the object in the image as the finger of the user'"'"'s hand comprises identifying the object in the image as the finger of the user'"'"'s hand based on an anatomical model of a finger oriented in a plane parallel to a display of the electronic device, the anatomical model being a model of a finger extending from a base of the finger to a tip of the finger.

18. The method of claim 15, wherein:

mapping the detected position to the control input comprises mapping the detected position to a position in a display region that corresponds to the detected position of the finger; and

controlling the application based on the control input comprises moving a displayed indicator to the position in the display region that corresponds to the detected position of the finger.

19. The method of claim 15, wherein:

mapping the detected position to the control input comprises identifying an input control at a position in a displayed user interface that corresponds to the detected position of the finger and determining an operation associated with the identified input control; and

controlling the application based on the control input comprises performing the operation associated with the identified input control.

20. The method of claim 15, wherein:

determining the position of the object identified in the image as the finger of the user'"'"'s hand comprises;

determining a width of the finger in the image, and

estimating a depth position of the finger with respect to a camera that captured the image based on the determined width; and

the control input is mapped based on the estimated depth position.

21. The method of claim 15, wherein determining the position of the object identified in the image as the finger of the user'"'"'s hand comprises detecting a part of the finger that corresponds to the fingertip, and determining the position as a centroid of the part of the finger that corresponds to the fingertip.

22. The method of claim 15, further comprising defining an image region within the image such that borders of the image region are spaced apart from borders of the image by a distance of at least one half of an expected fingertip width, wherein the position of the finger is detected within the defined image region.

23. The method of claim 22, wherein a position of the fingertip is mapped to a position on a border of the image region when the fingertip is detected outside of the image region.

24. A device comprising:

a camera configured to capture an image of a user'"'"'s hand interacting with an electronic device;

a processor configured to;

access the image of the user'"'"'s hand interacting with the electronic device;

identify an object in the image as a finger of the user'"'"'s hand based on a size of the object, a shape of the object, and whether the object intersects a border of the image;

determine a position of the object identified in the image as the finger of the user'"'"'s hand;

map the detected position to a control input; and

control an application based on the control input.

25. A computer readable medium comprising a computer program product, the computer program product comprising instructions that, when executed, operate to cause a computer to perform operations comprising:

accessing an image of a user'"'"'s hand interacting with an electronic device;

identifying an object in the image as a finger of the user'"'"'s hand based on a size of the object, a shape of the object, and whether the object intersects a border of the image;

determining a position of the object identified in the image as the finger of the user'"'"'s hand;

mapping the detected position to a control input; and

controlling an application based on the control input.