Pointer projection for natural user input

• US 9,823,764 B2
• Filed: 12/03/2014
• Issued: 11/21/2017
• Est. Priority Date: 12/03/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Schwesinger, Mark, Leyvand, Tommer, Stachniak, Szymon
• Primary Examiner(s): EURICE, MICHAEL
• Application Number:
  US14/559,870
• Publication Number:
  US 20160162082A1
• Time in Patent Office: 1,084 Days
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 2203/012 Walk-in-place systems for allowing a user to walk in a virtual environment while constraining him to a given position in the physical environment
• CPC Class Codes: G06F 2203/04101 2.5D-digitiser, i.e. digitiser detecting the X/Y position of the input means, finger or stylus, also when it does not touch, but is proximate to the digitiser's interaction surface and also measures the distance of the input means within a short range in the Z direction, possibly with a separate measurement setup
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 2203/04808 Several contacts: gestures triggering a specific function, e.g. scrolling, zooming, right-click, when the user establishes several contacts with the surface simultaneously; e.g. using several fingers or a combination of fingers and pen
• CPC Class Codes: G06F 2206/20 Indexing scheme related to audio interfaces for computers, indexing schema related to group G06F3/16
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0412 Digitisers structurally integrated in a display
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps

Abstract

A method to identify a targeted object based on eye tracking and gesture recognition. The method is enacted in a compute system controlled by a user and operatively coupled to a machine vision system. In this method, the compute system receives, from the machine vision system, video imaging a head and pointer of the user. Based on the video, the compute system computes a geometric line of sight of the user, which is partly occluded by the pointer. Then, with reference to position data for one or more objects, the compute system identifies the targeted object, situated along the geometric line of sight.

Claims

1. In a compute system operatively coupled to a machine vision system and to a display, a method comprising:

receiving, from the machine vision system, prior and subsequent video imaging a head and pointer of a user;

computing, based on the prior video, a prior geometric line of sight of the user partly occluded by the pointer;

assigning spatial coordinates to one or more non-display objects situated along the prior geometric line of sight, the spatial coordinates defining a position of each of the non-display objects within a field of view of the machine vision system;

computing, based on the subsequent video, a subsequent geometric line of sight of the user partly occluded by the pointer;

identifying, by reference to the spatial coordinates of the one or more non-display objects, a targeted display object situated on the display and along the subsequent geometric line of sight;

identifying in the subsequent video movement of the pointer along the subsequent geometric line of sight;

and executing code for a process associated with the targeted display object in response to the movement.

2. The method of claim 1, further comprising presenting the targeted display object on the display.

3. The method of claim 1, wherein computing the subsequent geometric line of sight includes computing spatial coordinates of the head from the subsequent video.

4. The method of claim 1, wherein computing the subsequent geometric line of sight includes computing spatial coordinates of an eye or pupil of the user from the subsequent video.

5. The method of claim 1, wherein computing the subsequent geometric line of sight includes

computing spatial coordinates of the pointer from the subsequent video,

and wherein the subsequent geometric line of sight is a straight line passing through the pointer and through the eye of the user.

6. The method of claim 1, wherein identifying the targeted display object includes determining whether the targeted display object is situated along the subsequent geometric line of sight.

7. The method of claim 1, wherein the pointer includes a finger of the user.

8. The method of claim 1, wherein the machine vision system includes a depth camera, and wherein each of the prior and subsequent video includes a series of time-resolved depth images from the depth camera.

9. The method of claim 1, wherein the machine vision system includes a flat-image camera, and wherein each of the prior and subsequent video includes a series of time-resolved images from the flat-image camera.

10. The method of claim 1, wherein depth and flat-image cameras of the machine vision system have parallel optical axes oriented in the same direction.

11. The method of claim 1, further comprising representing the targeted display object visually as a selected object in response to a selection command spoken by the user.

12. The method of claim 1, further comprising receiving user input defining the spatial coordinates.

13. The method of claim 1, wherein the one or more non-display objects define a viewing area of the display.

14. The method of claim 1, further comprising:

receiving, from the machine vision system, final video imaging the head and pointer of the user;

computing, based on the final video, a final line of sight of the user partly occluded by the pointer; and

forming on the display, along the final line of sight, an image to represent the targeted display object.

15. The method of claim 1, wherein the code is further executable to select the targeted display object in response to a selection command from the user.

16. The method of claim 1, wherein the machine vision system and the display are integrated in a head-mounted device worn by the user.

17. In a compute system operatively coupled to a machine vision system and to a display, a method comprising:

receiving, from the machine vision system, video imaging a head and hand of a user;

computing, based on the video, a geometric line of sight of the user partly occluded by the hand;

identifying, with reference to position data for one or more objects, a targeted display object situated on the display and along the geometric line of sight;

identifying, in the video, movement of the hand away from the user, along the geometric line of sight;

and shifting a depth coordinate of the display object in response to movement of the hand away from the user, along the geometric line of sight.

18. The method of claim 17 further comprising:

processing the video to identify closure of the hand, with the hand remaining in the geometric line of sight; and

reducing the size of the display object in response to closure of the hand, with the hand maintained in the geometric line of sight.

19. In a compute system operatively coupled to a machine vision system, a method comprising:

associating a process of the compute system to a non-display object arranged in a field of view of the machine-vision system, the process being one of a plurality of processes associated with a corresponding plurality of non-display objects in the field of view of the machine-vision system;

receiving, from the machine vision system, video imaging a head and pointer of a user;

computing, based on the video, a geometric line of sight of the user partly occluded by the pointer;

identifying the non-display object by reference to position data defining spatial coordinates of the plurality of non-display objects, the non-display object being situated along the geometric line of sight;

identifying in the video movement of the pointer along the geometric line of sight;

and executing the process associated with the targeted non-display object in response to the movement.

20. The method of claim 19, wherein the machine vision system is integrated into a head-mounted display device worn by the user.