Processing of gesture-based user interactions using volumetric zones

• US 10,564,731 B2
• Filed: 11/03/2017
• Issued: 02/18/2020
• Est. Priority Date: 09/14/2007
• Status: Active Grant

• Current Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Original Assignee: Meta Platforms, Inc. (f/k/a Facebook, Inc.)
• Inventors: Bell, Matthew, Chennavasin, Tipatat, Clanton, Charles H., Hulme, Michael, Ophir, Eyal, Vieta, Matthew
• Primary Examiner(s): Bycer, Eric J.
• Application Number:
  US15/803,517
• Publication Number:
  US 20180157334A1
• Time in Patent Office: 837 Days
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: G01B 11/14 for measuring distance or clearance between spaced objects or spaced apertures G01B11/26 takes precedence; rangefinders G01C3/00
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Systems and methods for processing gesture-based user interactions within an interactive display area are provided. The display of one or more virtual objects and user interactions with the one or more virtual objects may be further provided. Multiple interactive areas may be created by partitioning an area proximate a display into multiple volumetric spaces or zones. The zones may be associated with respective user interaction capabilities. A representation of a user on the display may change as the ability of the user to interact with one or more virtual object changes.

Claims

1. A method comprising:

receiving information corresponding to a physical space from one or more cameras;

mapping a position of a user in the physical space to a position in a virtual space based at least in part on the received information;

presenting, via an output device, a representation of the user in the position within the virtual space;

presenting, via the output device, a virtual object within the virtual space in a first state;

detecting, via the one or more cameras, an interaction with the virtual object based on a movement of a first part of the user from a first position in the physical space to a second position in the physical space;

determining a second state for presenting the virtual object based at least in part on the interaction and the first state;

presenting, via the output device, the representation of the user in the second position;

presenting, via the output device, the virtual object within the virtual space in the second state;

identifying a first zone within the physical space;

mapping the first zone to a portion of the virtual space; and

in response to detection of at least a portion of the user approaching an inter-zone boundary between the first zone and a second zone from a first direction, adjusting a display characteristic of the virtual object to indicate that a change to portions of the user that are capable of interacting with the virtual object will occur if the user continues to move in the first direction.

2. The method of claim 1, wherein the information corresponding to the physical space includes a display distance indicating a distance of the user to the output device, the method comprising determining the second state for presenting the virtual object based at least in part on the interaction, the first state, and the display distance.

3. The method of claim 1, wherein the information corresponding to the physical space includes a display distance indicating a distance of the user to the output device, the method comprising:

determining that the display distance corresponds to a threshold distance; and

detecting the interaction in response to said determining.

4. The method of claim 1, further comprising:

identifying a first zone within the physical space;

mapping the first zone to a portion of the virtual space; and

in response to detection of at least a portion of the user approaching an inter-zone boundary between the first zone and a second zone from a first direction, adjusting a display characteristic of the virtual object to indicate that a change to portions of the user that are capable of interacting with the virtual object will occur if the user continues to move in the first direction.

5. The method of claim 4, wherein the representation of the user comprises a representation of a hand of the user.

6. The method of claim 1, wherein the information corresponding to the physical space includes a display distance indicating a distance of the user to the output device, the method comprising generating the representation of the user at least in part by:

(a) projecting the second position along two dimensions of the virtual space and (b) projecting the distance of the user along a third dimension of the virtual space.

7. The method of claim 6 further comprising:

determining that the distance along the third dimension is less than a distance of the virtual object along the third dimension; and

overlaying a portion of the virtual object with a portion of the representation of the user.

8. The method of claim 6 further comprising:

determining that the distance along the third dimension is greater than a distance of the virtual object along the third dimension; and

overlaying a portion of the representation of the user with a portion of the virtual object.

9. A non-transitory computer readable storage having executable instructions stored thereon, the executable instructions configured to cause a computing system having one or more hardware processors to perform operations comprising:

receiving information corresponding to a physical space from one or more cameras;

mapping a position of a user in the physical space to a position in a virtual space based at least in part on the received information;

presenting, via an output device, a representation of the user in the position within the virtual space;

presenting, via the output device, a virtual object within the virtual space in a first state;

detecting, via the one or more cameras, an interaction with the virtual object based on a movement of a first part of the user from a first position in the physical space to a second position in the physical space;

determining a second state for presenting the virtual object based at least in part on the interaction and the first state;

presenting, via the output device, the representation of the user in the second position; and

presenting, via the output device, the virtual object within the virtual space in the second state;

identifying a first zone within the physical space;

mapping the first zone to a portion of the virtual space; and

in response to detection of at least a portion of the user approaching an inter-zone boundary between the first zone and a second zone from a first direction, adjusting a display characteristic of the virtual object to indicate that a change to portions of the user that are capable of interacting with the virtual object will occur if the user continues to move in the first direction.

10. The non-transitory computer readable storage of claim 9,

wherein the information corresponding to the physical space includes a display distance indicating a distance of the user to the output device, and

wherein the computer system further performs operations comprising determining the second state for presenting the virtual object based at least in part on the interaction, the first state, and the display distance.

11. The non-transitory computer readable storage of claim 9,

wherein the information corresponding to the physical space includes a display distance indicating a distance of the user to the output device, and

wherein the computer system further performs operations comprising:

determining that the display distance corresponds to a threshold distance; and

further comprising detecting the interaction in response to said determining.

12. The non-transitory computer readable storage of claim 9, wherein the representation of the user comprises a representation of a hand of the user.

13. The non-transitory computer readable storage of claim 9,

wherein the information corresponding to the physical space includes a display distance indicating a distance of the user to the output device, and

wherein the computer system further performs operations comprising generating the representation of the user at least in part by:

(a) projecting the second position along two dimensions of the virtual space and (b) projecting the distance of the user along a third dimension of the virtual space.

14. The non-transitory computer readable storage of claim 13, wherein the computer system further performs operations comprising:

determining that the distance along the third dimension is greater than a distance of the virtual object along the third dimension; and

overlaying a portion of the representation of the user with a portion of the virtual object.

15. A computing system comprising:

a camera; and

one or more hardware processors configured to execute instructions to cause the computing system to;

receive information corresponding to a physical space from the camera;

map a position of a user in the physical space to a position in a virtual space based at least in part on the received information;

present, via an output device, a representation of the user in the position within the virtual space;

present, via the output device, a virtual object within the virtual space in a first state;

detect, via the camera, an interaction with the virtual object based on a movement of a first part of the user from a first position in the physical space to a second position in the physical space;

determine a second state for presenting the virtual object based at least in part on the interaction and the first state;

present, via the output device, the representation of the user in the second position; and

present, via the output device, the virtual object within the virtual space in the second state;

identify a first zone within the physical space;

map the first zone to a portion of the virtual space; and

in response to detection of the at least a first portion of the user approaching an inter-zone boundary between the first zone and a second zone from a first direction, adjust a display characteristic of the virtual object to indicate that a change to portions of the user that are capable of interacting with the virtual object will occur if the user continues to move in the first direction.

16. The computing system of claim 15,

wherein the information corresponding to the physical space includes a display distance indicating a distance of the user to the output device, and

wherein the instructions further cause the computer system to determine the second state for presenting the virtual object based at least in part on the interaction, the first state, and the display distance.

17. The computing system of claim 15,

wherein the information corresponding to the physical space includes a display distance indicating a distance of the user to the output device, and

wherein the instructions further cause the computer system to:

determine that the display distance corresponds to a threshold distance; and

in response to determining that the display distance corresponds to the threshold distance, detect the interaction.

18. The computing system of claim 15, wherein the representation of the user comprises a representation of a hand of the user.

19. The computing system of claim 15,

wherein the information corresponding to the physical space includes a display distance indicating a distance of the user to the output device, and

wherein the instructions further cause the computer system to generate the representation of the user at least in part by:

(a) projecting the second position along two dimensions of the virtual space and (b) projecting the distance of the user along a third dimension of the virtual space.

20. The computing system of claim 19, wherein the instructions further cause the computer system to:

determine that the distance along the third dimension is less than a distance of the virtual object along the third dimension; and

overlay a portion of the virtual object with a portion of the representation of the user.