Processing of gesture-based user interactions using volumetric zones

• US 9,811,166 B2
• Filed: 06/15/2015
• Issued: 11/07/2017
• Est. Priority Date: 09/14/2007
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Intellectual Ventures Holding 81 LLC (Intellectual Ventures LLC)
• Inventors: Bell, Matthew, Chennavasin, Tipatat, Clanton, Charles H., Hulme, Michael, Ophir, Eyal, Vieta, Matthew
• Primary Examiner(s): Bycer, Eric J
• Application Number:
  US14/739,922
• Publication Number:
  US 20150346835A1
• Time in Patent Office: 876 Days
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Systems and methods for processing gesture-based user interactions within an interactive display area are provided. The display of one or more virtual objects and user interactions with the one or more virtual objects may be further provided. Multiple interactive areas may be created by partitioning an area proximate a display into multiple volumetric spaces or zones. The zones may be associated with respective user interaction capabilities. A representation of a user on the display may change as the ability of the user to interact with one or more virtual object changes.

Claims

1. A method comprising:

receiving, by a computing system, information corresponding to a physical space from one or more cameras, wherein the physical space comprises a plurality of zones; and

generating, by the computing system, an image comprising one or more virtual objects and a first status indicator indicating a first portion of a user in the physical space that is capable of providing input to the computing system,

wherein the first status indicator is not reactive to movement of the user within a first zone of the plurality of zones unless the user approaches a boundary to either another of the plurality of zones or to an outside physical space outside of any of the plurality of zones, and

wherein, in response to detection of the at least the first portion of the user approaching an inter-zone boundary between the first zone and a second zone from a first direction, adjusting a display characteristic of the first status indicator to indicate that a change to portions of the user that are capable of interacting with the one or more virtual objects will occur if the user continues to move in the first direction.

2. The method of claim 1, wherein the first zone is positioned less than a first distance from a display device upon which the image is displayed, the second zone is positioned at a distance greater than the first distance from the display device and less than a second distance from the display device, and wherein the plurality of zones further comprises a third zone that is positioned greater than the second distance from the display device and less than a third distance from the display device.

3. The method of claim 1, further comprising selecting the first status indicator from a plurality of status indicators in response to detecting the at least the first portion of the user within the first zone, wherein the first status indicator indicates that the user is not capable of interacting with the one or more virtual objects with hands or body of the user.

4. The method of claim 3, wherein the first status indicator comprises at least one of:

a disappearing puff of smoke;

or

a fading image of at least a portion of the user.

5. The method of claim 1, further comprising selecting a second status indicator in response to detecting at least hands and body of the user within the second zone, wherein the second status indicator indicates that the user is capable of interacting with the one or more virtual objects in the second zone with the user'"'"'s hands and body and the second status indicator does not indicate specific input from the user that is used to interact with the one or more virtual objects.

6. The method of claim 5, further comprising selecting a third status indicator in response to detecting at least the portion of the user within a third zone, wherein the third status indicator indicates that the user is capable of interacting with the one or more virtual objects in the third zone with the body of the user.

7. The method of claim 1, wherein said detection of the at least the first portion of the user approaching the inter-zone boundary is based on at least one of a distance of the at least the first portion of the user from a display device or a distance of the least the first portion of the user from one or more of the one or more cameras.

8. The method of claim 1, wherein the first status indicator indicates a third portion of the user that is not capable of providing input to the computing system in order to interact with the one or more virtual objects.

9. The method of claim 1, wherein the information corresponding to the physical space includes a display distance indicating a distance of the user to a display device upon which the image is displayed, the method further comprising adjusting the boundary between the first zone and a second zone using the display distance.

10. The method of claim 1, further comprising:

determining an activation level of the user based on a first distance from a first position of the first portion of the user to a position of another portion of the user in the physical space; and

in response to detection of the at least the first portion of the user approaching the inter-zone boundary between the first zone and the second zone from the first direction, updating the activation level of the user based on a second distance from a second position of a second portion of the user in the physical space to the position of another portion of the user in the physical space.

11. A non-transitory computer readable storage having executable instructions stored thereon, the executable instructions configured to cause a computing system having one or more hardware processors to perform operations comprising:

receiving information indicative of a physical space having a plurality of zones;

generating an image, the image comprising;

a virtual object; and

a first status indicator indicating a first portion of a user in the physical space;

wherein the first status indicator is not reactive to movement of the user within a first zone of the plurality of zones unless the user approaches a boundary to either another of the plurality of zones or to an outside physical space outside of any of the plurality of zones; and

adjusting, in response to detection of the at least the first portion of the user approaching an inter-zone boundary between the first zone and a second zone from a first direction, a display characteristic of the first status indicator to indicate that a change to portions of the user that are capable of interacting with the one or more virtual objects will occur if the user continues to move in the first direction.

12. The non-transitory computer readable storage of claim 11, wherein the user is capable of different interactions with the virtual object in each of the plurality of zones.

13. The non-transitory computer readable storage of claim 11, wherein the first zone is positioned less than a first distance from a display device upon which the image is displayed, the second zone is positioned at a distance greater than the first distance from the display device and less than a second distance from the display device, and wherein the plurality of zones further comprises a third zone that is positioned greater than the second distance from the display device and less than a third distance from the display device.

14. The non-transitory computer readable storage of claim 11, wherein the computer system further performs operations comprising selecting the first status indicator from a plurality of comprises in response to detecting the at least the first portion of the user within the first zone, wherein the first status indicator indicates that the user is not capable of interacting with the one or more virtual objects with hands or body of the user.

15. The non-transitory computer readable storage of claim 11, wherein the computer system further performs operations comprising:

determining an activation level of the user based on a first distance from a first position of the first portion of the user to a position of another portion of the user in the physical space; and

updating, in response to detection of the at least the first portion of the user approaching the inter-zone boundary between the first zone and a second zone from the first direction, the activation level of the user based on a second distance from a second position of a second portion of the user in the physical space to the position of another portion of the user in the physical space.

16. A computing system, comprising:

a camera; and

one or more hardware processors configured to execute instructions in order to cause the computing system to;

receive information corresponding to a physical space from the camera, wherein the physical space comprises a plurality of zones; and

generate an image comprising one or more virtual objects and a first status indicator indicating a first portion of a user that is capable of providing input to the computing system,

wherein the first status indicator is not reactive to movement of the user within a first zone of the plurality of zones unless the user approaches a boundary to either another of the plurality of zones or to an outside physical space outside of any of the plurality of zones, and

wherein, in response to detection of the at least the first portion of the user approaching an inter-zone boundary between the first zone and a second zone from a first direction, adjusting a display characteristic of the first status indicator to indicate that a change in which portions of the user are capable of interacting with the one or more virtual objects will occur if the user continues to move in the first direction.

17. The system of claim 16, wherein the first zone is positioned less than a first distance from a display device upon which the image is displayed, the second zone is positioned at a distance greater than the first distance from the display device and less than a second distance from the display device, and a third zone is positioned greater than the second distance from the display device and less than a third distance from the display device.

18. The system of claim 16, wherein the instructions further cause the computer system to select the first status indicator in response to detection of the at least the first portion of the user within the first zone, wherein the first status indicator indicates that the user is not capable of interacting with the one or more virtual objects with hands or body of the user.

19. The system of claim 16, wherein the information corresponding to the physical space includes a display distance indicating a distance of the user to a display device upon which the image is displayed, wherein the instructions further cause the computer system to adjust the inter-zone boundary between the first zone and a second zone using the display distance.

20. The system of claim 16, further comprising instructions to cause the computer system to:

determine an activation level of the user based on a first distance from a first position of the first portion of the user to a position of another portion of the user in the physical space; and

in response to detection of the at least the first portion of the user approaching the inter-zone boundary between the first zone and a second zone from the first direction, update the activation level of the user based on a second distance from a second position of a second portion of the user in the physical space to the position of another portion of the user in the physical space.