Operating environment comprising multiple client devices, multiple displays, multiple users, and gestural control

• US 10,656,724 B2
• Filed: 03/12/2018
• Issued: 05/19/2020
• Est. Priority Date: 04/02/2009
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Kramer, Kwindla Hultman, Underkoffler, John, Sparrell, Carlton, Singh, Navjot, Hollenbach, Kate, Yarin, Paul
• Primary Examiner(s): Anya, Charles E
• Application Number:
  US15/918,636
• Publication Number:
  US 20180203520A1
• Time in Patent Office: 799 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Embodiments described herein includes a system comprising a processor coupled to display devices, sensors, remote client devices, and computer applications. The computer applications orchestrate content of the remote client devices simultaneously across at least one of the display devices and the remote client devices, and allow simultaneous control of the display devices. The simultaneous control includes automatically detecting a gesture of at least one object from gesture data received via the sensors. The gesture data is absolute three-space location data of an instantaneous state of the at least one object at a point in time and space. The detecting comprises aggregating the gesture data, and identifying the gesture using only the gesture data. The computer applications translate the gesture to a gesture signal, and control at least one of the display devices and the remote client devices in response to the gesture signal.

Claims

1. A method comprising:

with a first remote client device, using a first display system application to generate a first un-typed data structure that includes first typed graphics data of a first client application executed by the first remote client device;

with a second remote client device, using a second display system application to generate a second un-typed data structure that includes second typed graphics data of a second client application executed by the second remote client device;

with a display system processor;

receiving the first un-typed data structure from the first remote client device;

receiving second un-typed data structure from the second remote client device;

including the first un-typed data structure and the second un-typed data structure in a first repository for un-typed data structures;

rendering graphics data of the first repository;

controlling display of the rendered graphics data of the first repository, wherein the rendered graphics data includes rendered graphics data for the first typed graphics data and rendered graphics data for the second typed graphics data;

receiving gesture data that includes absolute three-space location data of an instantaneous state of a first object and a second object at a point in time and space;

simultaneously detecting a first gesture of the first object and a second gesture of the second object from the received gesture data; and

updating display of the rendered graphics data based on at least one of the detected first gesture and the detected second gesture.

2. The method of claim 1,

wherein controlling display of the rendered graphics data of the first repository comprises:

controlling a display system that includes a plurality of display devices to display the rendered graphics data of the first repository, and

wherein updating display of the rendered graphics data comprises;

controlling the display system to update the display of the rendered graphics data.

3. The method of claim 1,

wherein controlling display of the rendered graphics data of the first repository comprises:

controlling the first remote client device to display the rendered graphics data of the first repository, and

wherein updating display of the rendered graphics data comprises;

controlling the first remote client device to update the display of the rendered graphics data.

4. The method of claim 1,

wherein controlling display of the rendered graphics data of the first repository comprises:

controlling the second remote client device to display the rendered graphics data of the first repository, and

wherein updating display of the rendered graphics data comprises;

controlling the second remote client device to update the display of the rendered graphics data.

5. The method of claim 1, wherein the first un-typed data structure and the second un-typed data structure have a format that is application-independent.

6. The method of claim 1, wherein the first client application is different from the second client application.

7. The method of claim 1, wherein the first un-typed data structure and the second un-typed data structure have a same format.

8. The method of claim 1, wherein the display system processor receives the gesture data from a plurality of sensors.

9. The method of claim 1, wherein updating display of the rendered graphics data based on at least one of the detected first gesture and the detected second gesture comprises:

updating display of the rendered graphics data for the first typed graphics data based the detected second gesture.

10. The method of claim 1, wherein updating display of the rendered graphics data based on at least one of the detected first gesture and the detected second gesture comprises:

updating display of the rendered graphics data for the first typed graphics data based the detected first gesture.

11. The method of claim 1, wherein updating display of the rendered graphics data based on at least one of the detected first gesture and the detected second gesture comprises:

updating display of the rendered graphics data for the second typed graphics data based the detected second gesture.

12. The method of claim 1, wherein updating display of the rendered graphics data based on at least one of the detected first gesture and the detected second gesture comprises:

updating display of the rendered graphics data for the second typed graphics data based the detected first gesture.

13. The method of claim 1, wherein the first object is associated with a first user and the second object is associated with a second user.

14. The method of claim 13, wherein the first object is a wand.

15. The method of claim 13,

wherein the first object is an edge computing device, and

wherein the absolute three-space location data of the instantaneous state of the first object is determined by using a pointer application executed by the edge computing device.

16. The method of claim 13, wherein the second object is a wand.

17. The method of claim 13, wherein the second object is an input device.

18. The method of claim 13, wherein the first object is associated with the first remote client device, and wherein the second object is associated with the second remote client device.

19. The method of claim 13, wherein one of the first object and the second object is a phone.