Systems and methods for gesture-based interaction

• US 10,303,259 B2
• Filed: 04/03/2017
• Issued: 05/28/2019
• Est. Priority Date: 04/03/2017
• Status: Active Grant

• Current Assignee: Hasivision, LLC
• Original Assignee: YouSpace, Inc.
• Inventors: Brunner, Ralph, Seligman, Andrew Emmett, Chien, Hsin-Yi, Stoddard, John Philip, Chen, Po-Jui, Sharton-Bierig, Elon, Gaston, Sierra Justine, Brunner, Vivi
• Primary Examiner(s): Taylor, Jr., Duane N
• Application Number:
  US15/478,201
• Publication Number:
  US 20180284901A1
• Time in Patent Office: 785 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

Various of the disclosed embodiments present depth-based user interaction systems facilitating natural and immersive user interactions. Particularly, various embodiments integrate immersive visual presentations with natural and fluid gesture motions. This integration facilitates more rapid user adoption and more precise user interactions. Some embodiments may take advantage of the particular form factors disclosed herein to accommodate user interactions. For example, dual depth sensor arrangements in a housing atop the interface'"'"'s display may facilitate depth fields of view accommodating more natural gesture recognition than may be otherwise possible. In some embodiments, these gestures may be organized into a framework for universal control of the interface by the user and for application-specific control of the interface by the user.

Claims

1. A user interface comprising:

one or more displays;

a depth sensor housing frame, the depth sensor housing frame comprising a first depth sensor associated with a first field of view passing through a first panel of the depth-sensor housing frame, the first panel substantially opaque to visual frequencies, and a second depth sensor associated with a second field of view passing through a second panel of the depth sensor housing frame, the second panel substantially opaque to visual frequencies, the first depth sensor mechanically coupled above the second depth sensor, the depth sensor housing frame above the one or more displays;

at least one processor; and

at least one memory comprising instructions configured, in conjunction with the at least one processor, to cause the user interface to perform a method comprising;

acquiring a first frame of depth data using the first depth sensor;

acquiring a second frame of depth data using the second depth sensor;

associating at least a portion of the first frame of depth data and the second frame of depth data with a classification corresponding to a first portion of a user'"'"'s body;

determining a gesture based, at least in part, upon the classification; and

notifying an application running on the user interface of the determined gesture.

2. The user interface of claim 1, wherein the depth sensor housing frame further comprises:

a back panel;

a top panel;

a bottom panel;

at least one depth sensor mount; and

at least two spacers located within an enclosure formed by the back panel, top panel, bottom panel, first panel substantially opaque to visual frequencies, and the second panel substantially opaque to visual frequencies, wherein

the at least one depth sensor mount is located between two of the at least two spacers, and wherein

the at least one depth sensor mount is configured to receive the first depth sensor and the second depth sensor such that a field of view associated with each depth sensor passes through at least one of the two panels substantially opaque to visual frequencies.

3. The user interface of claim 2, wherein the one or more displays comprises a 3×

3 grid of substantially adjacent displays.

4. The user interface of claim 3, wherein the depth sensor housing frame is one of three depth sensor housing frames comprising:

a first depth sensor housing frame above the top-left display of the 3×

3 grid;

a second depth sensor housing frame above the top-middle display of the 3×

3 grid; and

a third depth sensor housing frame above the top-right display of the 3×

3 grid.

5. The user interface of claim 1, wherein the method further comprises:

detecting a location of at least a portion of the user relative to the interface; and

adjusting a vanishing point of an image presented on the one or more displays to correspond with the location.

6. The user interface of claim 1, wherein the method further comprises:

detecting a first distance between the user and the one or more displays;

determining that the first distance is greater than a threshold distance from the one or more displays;

displaying a virtual environment comprising one or more rooms from a first perspective based, at least in part, upon the determination that the first distance is greater than the threshold distance;

detecting a second distance between the user and the one or more displays;

determining that the second distance is less than a threshold distance from the one or more displays; and

displaying the virtual environment from a second perspective based, at least in part, upon the determination that the second distance is less than the threshold distance, wherein

the first perspective displays the virtual environment from a virtual location further from the rooms than the second perspective.

7. The user interface of claim 6, wherein the method further comprises:

detecting a push gesture while presenting the virtual environment from the second perspective; and

activating an application associated with the room displayed closest to the user on the one or more displays.

8. A computer-implemented method running on a user interface, the method comprising:

acquiring a first frame of depth data from a first depth sensor;

acquiring a second frame of depth data from a second depth sensor, the first depth sensor and the second depth sensor within a depth sensor housing frame above one or more displays, the first depth sensor associated with a first field of view passing through a first panel of the depth sensor housing frame, the first panel substantially opaque to visual frequencies, and the second depth sensor associated with a second field of view passing through a second panel substantially opaque to visual frequencies, the first depth sensor mechanically coupled above the second depth sensor;

associating at least a portion of the first depth frame and the second depth frame with a classification corresponding to a first portion of a user'"'"'s body;

determining a gesture based, at least in part, upon the classification; and

notifying an application running on the user interface of the determined gesture.

9. The computer-implemented method of claim 8, wherein the depth sensor housing frame further comprises:

a back panel;

a top panel;

a bottom panel;

at least one depth sensor mount; and

at least two spacers located within an enclosure formed by the back panel, top panel, bottom panel, first panel substantially opaque to visual frequencies, and the second panel substantially opaque to visual frequencies, wherein

the at least one depth sensor mount is located between two of the at least two spacers, and wherein

the at least one depth sensor mount is configured to receive the first depth sensor and the second depth sensor such that a field of view associated with each depth sensor passes through at least one of the two panels substantially opaque to visual frequencies.

10. The computer-implemented method of claim 8, wherein the method further comprises:

detecting a location of at least a portion of the user relative to the interface; and

adjusting a vanishing point of an image presented on the one or more displays to correspond with the location.

11. The computer-implemented method of claim 8, wherein the method further comprises:

detecting a first distance between the user and the one or more displays;

determining that the first distance is greater than a threshold distance from the one or more displays;

displaying a virtual environment comprising one or more rooms from a first perspective based, at least in part, upon the determination that the first distance is greater than the threshold distance;

detecting a second distance between the user and the one or more displays;

determining that the second distance is less than a threshold distance from the one or more displays; and

displaying the virtual environment from a second perspective based, at least in part, upon the determination that the second distance is less than the threshold distance, wherein

the first perspective displays the virtual environment from a virtual location further from the rooms than the second perspective.

12. The computer-implemented method of claim 11, wherein the method further comprises:

detecting a push gesture while presenting the virtual environment from the second perspective; and

activating an application associated with the room displayed closest to the user on the one or more displays.

18. The computer-implemented method of claim 9, wherein the one or more displays comprise a 3×

3 grid of substantially adjacent displays.

19. The computer-implemented method of claim 18, wherein the depth sensor housing frame is one of three depth sensor housing frames comprising:

a first depth sensor housing frame above the top-left display of the 3×

3 grid;

a second depth sensor housing frame above the top-middle display of the 3×

3 grid; and

a third depth sensor housing frame above the top-right display of the 3×

3 grid.

13. A non-transitory computer-readable medium comprising instructions configured to cause a user interface to perform a method, the method comprising:

acquiring a first frame of depth data from a first depth sensor;

acquiring a second frame of depth data from a second depth sensor, the first depth sensor and the second depth sensor within a depth sensor housing frame above one or more displays, the first depth sensor associated with a first field of view passing through a first panel of the depth sensor housing frame, the first panel substantially opaque to visual frequencies, and the second depth sensor associated with a second field of view passing through a second panel, the second panel substantially opaque to visual frequencies, the first depth sensor mechanically coupled above the second depth sensor;

associating at least a portion of the first depth frame and the second depth frame with a classification corresponding to a first portion of a user'"'"'s body;

determining a gesture based, at least in part, upon the classification; and

notifying an application running on the user interface of the determined gesture.

14. The non-transitory computer-readable medium of claim 13, wherein the depth sensor housing frame comprises:

a back panel;

a top panel;

a bottom panel;

at least one depth sensor mount; and

at least two spacers located within an enclosure formed by the back panel, top panel, bottom panel, first panel substantially opaque to visual frequencies, and the second panel substantially opaque to visual frequencies, wherein

the at least one depth sensor mount is located between two of the at least two spacers, and wherein

the at least one depth sensor mount is configured to receive the first depth sensor and the second depth sensor such that a field of view associated with each depth sensor passes through at least one of the two panels substantially opaque to visual frequencies.

15. The non-transitory computer-readable medium of claim 13, wherein the method further comprises:

detecting a location of at least a portion of the user relative to the interface; and

adjusting a vanishing point of an image presented on the displays to correspond with the location.

16. The non-transitory computer-readable medium of claim 13, wherein the method further comprises:

detecting a first distance between the user and the displays;

determining that the first distance is greater than a threshold distance from the displays;

displaying a virtual environment comprising one or more rooms from a first perspective based, at least in part, upon the determination that the first distance is greater than the threshold distance;

detecting a second distance between the user and the displays;

determining that the second distance is less than a threshold distance from the displays; and

displaying the virtual environment from a second perspective based, at least in part, upon the determination that the second distance is less than the threshold distance, wherein

the first perspective displays the virtual environment from a virtual location further from the rooms than the second perspective.

17. The non-transitory computer-readable medium of claim 16, wherein the method further comprises:

detecting a push gesture while presenting the virtual environment from the second perspective; and

activating an application associated with the room displayed closest to the user on the one or more displays.

20. The non-transitory computer-readable medium of claim 14, wherein the one or more displays comprise a 3×

3 grid of substantially adjacent displays.

21. The non-transitory computer-readable medium of claim 20, wherein the depth sensor housing frame is one of three depth sensor housing frames comprising:

a first depth sensor housing frame above the top-left display of the 3×

3 grid;

a second depth sensor housing frame above the top-middle display of the 3×

3 grid; and

a third depth sensor housing frame above the top-right display of the 3×

3 grid.