Enhanced camera-based input

• US 8,659,548 B2
• Filed: 05/21/2008
• Issued: 02/25/2014
• Est. Priority Date: 07/27/2007
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Hildreth, Evan
• Primary Examiner(s): Beck, Alexander S
• Assistant Examiner(s): STEINBERG, JEFFREY S
• Application Number:
  US12/124,375
• Publication Number:
  US 20090027337A1
• Time in Patent Office: 2,106 Days
• Field of Search: 345156-184
• US Class Current: 345/158
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/03 Arrangements for converting the position or the displacement of a member into a coded form
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence

Abstract

Enhanced camera-based input, in which a detection region surrounding a user is defined in an image of the user within a scene, and a position of an object (such as a hand) within the detection region is detected. Additionally, a control (such as a key of a virtual keyboard) in a user interface is interacted with based on the detected position of the object.

Claims

1. A non-transitory computer readable medium encoded with a computer program product, the computer program product comprising instructions that, when executed, operate to cause a computer to perform operations comprising:

generating an image of a user within a scene;

defining, in the image, a detection region surrounding the user, further comprising;

determining a position of a torso and a reach of an arm of the user,

defining the detection region to exclude the torso and at least a portion of a region of the image unreachable by the arm;

determining a portion of the detection region in which a second user could be detected; and

defining the detection region to exclude the portion of the detection region in which the second user could be detected;

detecting a position of a hand of the user within the detection region; and

interacting with a control in a user interface based on the detected position of the hand, the control comprising items aligned with a guide line defined relative to an avatar representation of the user on the user interface.

2. A computer-implemented method comprising:

defining, in an image of a user within a scene, a detection region surrounding the user, the defining comprising;

determining an unreachable region of the image not reachable by an object associated with the user,

defining the detection region to exclude at least a portion of the unreachable region;

determining a portion of the detection region in which a second user could be detected; and

defining the detection region to exclude the portion of the detection region in which the second user could be detected;

detecting a position of the object within the detection region; and

interacting with a control in a user interface based on the detected position of the object.

3. The method of claim 2, further comprising detecting an engagement gesture of the user, wherein the position of the object is detected based on detecting the engagement gesture.

4. The method of claim 2, wherein the object is a hand of the user.

5. The method of claim 2, wherein interacting with the control further comprises selecting a character.

6. The method of claim 2, wherein the user interface further comprises a representation of the user.

7. The method of claim 6, further comprising displaying a segmented image as the representation of the user.

8. The method of claim 6, wherein the control comprises items aligned with a guide line defined relative to the representation, wherein the items are displayed without obscuring the representation.

9. The method of claim 6, wherein the representation of the user further comprises an avatar or a cursor.

10. The method of claim 9, further comprising animating the avatar to mimic motion of the user, based on the detected position of the object.

11. The method of claim 6, further displaying the control in the user interface above or aside the representation.

12. The method of claim 2, wherein defining the detection region further comprises:

determining a position of a torso of the user; and

defining the detection region to exclude the torso, based on the determined position of the torso.

13. The method of claim 12, wherein the position of the torso of the user is determined using image segmentation.

14. The method of claim 12, wherein defining the detection region comprises:

determining the unreachable region of the image not reachable by the object, based on the determined position of the torso; and

defining the detection region to exclude the determined unreachable region of the image.

15. The method of claim 2, wherein the object comprises at least a portion of an arm of the user, and wherein defining the detection region further comprises:

determining a reach of the arm of the user; and

defining the detection region to exclude at least a portion of a region of the image unreachable by the arm, based on the determined reach of the arm.

16. The method of claim 15, wherein the reach of the arm of the user is based on a determined position of a torso, head or shoulder of the user.

17. The method of claim 2, wherein defining the detection region further comprises:

determining a position of a head of the user; and

defining the detection region to exclude the head, based on the determined position of the head.

18. The method of claim 17, wherein the position of the head of the user is determined using face detection.

19. The method of claim 2, further comprising:

cropping the image; and

displaying the image within the user interface.

20. The method of claim 19, further comprising centering the image on the user, wherein the image is cropped based on centering the image.

21. The method of claim 20, further comprising determining a magnification factor that makes the control reachable to the user within the centered image, wherein the image is cropped based on the magnification factor.

22. The method of claim 21, wherein the magnification factor is determined using an anatomical model.

23. The method of claim 2, further comprising:

detecting a face of the user;

determining an identity of the user based on the detected face; and

adjusting the control based on the identity of the user.

24. The method of claim 2, further comprising:

detecting a position of a second object within the detection region; and

adjusting the control based on the position of the second object.

25. The method of claim 2, further comprising capturing the image using a camera.

26. The method of claim 2, wherein the detection region is shaped as an arc-shaped detection sub-region above the user, contiguous with two linear detection sub-regions, one on each side of the user.

27. The method of claim 2, wherein detecting the position of the object within the detection region further comprises:

detecting a position of the object in relation to a guideline mapped within the detection region.

30. The method of claim 2, wherein the detection region is defined to overlap at least one portion of the user.

31. The method of claim 2, wherein the detection region is defined not to overlap the user.

32. The method of claim 2, wherein the detection region is defined in response to an engagement gesture being performed.

28. A non-transitory computer readable medium encoded with a computer program product, the computer program product comprising instructions that, when executed, operate to cause a computer to perform operations comprising:

defining, in an image of a user within a scene, a detection region surrounding the user, the defining comprising;

determining an unreachable region of the image not reachable by an object associated with the user,

defining the detection region to exclude at least a portion of the unreachable region;

determining a portion of the detection region in which a second user could be detected; and

defining the detection region to exclude the portion of the detection region in which the second user could be detected;

detecting a position of the object within the detection region; and

interacting with a control in a user interface based on the detected position of the object.

29. A device comprising a processor configured to:

define, in an image of a user within a scene, a detection region surrounding the user, at least in part by;

determining an unreachable region of the image not reachable by an object associated with the user,

defining the detection region to exclude at least a portion of the unreachable region;

determining a portion of the detection region in which a second user could be detected; and

defining the detection region to exclude the portion of the detection region in which the second user could be detected;

detect a position of the object within the detection region; and

interact with a control in a user interface based on the detected position of the object.

33. An apparatus comprising:

means for defining, in an image of a user within a scene, a detection region surrounding the user, the means for defining comprising;

means for determining an unreachable region of the image not reachable by an object associated with the user,

means for defining the detection region to exclude at least a portion of the unreachable region;

means for determining a portion of the detection region in which a second user could be detected; and

means for defining the detection region to exclude the portion of the detection region in which the second user could be detected;

means for detecting a position of the object within the detection region; and

means for interacting with a control in a user interface based on the detected position of the object.