Operating environment comprising multiple client devices, multiple displays, multiple users, and gestural control

• US 9,952,673 B2
• Filed: 10/08/2013
• Issued: 04/24/2018
• Est. Priority Date: 04/02/2009
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Kramer, Kwindla Hultman, Underkoffler, John, Sparrell, Carlton, Singh, Navjot, Hollenbach, Kate, Yarin, Paul
• Primary Examiner(s): Anya, Charles E
• Application Number:
  US14/048,747
• Publication Number:
  US 20140195988A1
• Time in Patent Office: 1,659 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883

Abstract

Embodiments described herein includes a system comprising a processor coupled to display devices, sensors, remote client devices, and computer applications. The computer applications orchestrate content of the remote client devices simultaneously across at least one of the display devices and the remote client devices, and allow simultaneous control of the display devices. The simultaneous control includes automatically detecting a gesture of at least one object from gesture data received via the sensors. The gesture data is absolute three-space location data of an instantaneous state of the at least one object at a point in time and space. The detecting comprises aggregating the gesture data, and identifying the gesture using only the gesture data. The computer applications translate the gesture to a gesture signal, and control at least one of the display devices and the remote client devices in response to the gesture signal.

Claims

1. A system for decoupling applications in multi-computer interactive configuration, the system comprising:

a display system processor that is communicatively coupled to a plurality of sensors and a display system that includes a plurality of display devices;

a first remote client device; and

a second remote client device,

wherein the first remote client device generates a first un-typed data structure that includes first typed graphics data of a first client application of the first remote client device, and provides the first un-typed data structure to the display system processor,

wherein the second remote client device generates a second un-typed data structure that includes second typed graphics data of a second client application of the second remote client device, and

provides the second un-typed data structure to the display system processor,

wherein the display system processor includes the first un-typed data structure and the second un-typed data structure in a first pool of the display system processor, wherein the display system processor renders graphics data of the first pool and controls at least one of the display system, the first remote client device and the second remote client device to display the rendered graphics data of the first pool, the rendered graphics data of the first pool including rendered graphics data for the first typed graphics data and rendered graphics data for the second typed graphics data,

wherein the display system processor simultaneously detects a first gesture of a first object associated with the first remote client device and a second gesture of a second object associated with the second remote client device from gesture data received via the plurality of sensors,

wherein the gesture data includes absolute three-space location data of an instantaneous state of the first object and the second object at a point in time and space,

wherein the display system processor controls at least one of the display system, the first remote client device, and the second remote client device to update the display of the rendered graphics data for the first typed graphics data based on the detected second gesture of the second object associated with the second remote client device,

wherein the first un-typed data structure and the second un-typed data structure have a format that is application-independent,

wherein the first client application is different from the second client application, and

wherein the first un-typed data structure and the second un-typed data structure have a same format.

2. The system of claim 1,

wherein the display system processor is constructed to control the display system, the first client device and the second client device to simultaneously display the rendered graphics data of the first pool across the plurality of display devices of the display system, a display device of first remote client device and a display device of the second remote client device.

3. The system of claim 2, wherein the display device of first remote client device displays the rendered graphics data for the first typed graphics data, and wherein the display system processor is constructed to control the first remote client device to update the display of the rendered graphics data for the first typed graphics data based on the detected second gesture.

4. The system of claim 2, wherein the display device of second remote client device displays the rendered graphics data for the first typed graphics data, and wherein the display system processor is constructed to control the second remote client device to update the display of the rendered graphics data for the first typed graphics data based on the detected second gesture.

5. The system of claim 2, wherein one of the plurality of display devices of the display system displays the rendered graphics data for the first typed graphics data, and wherein the display system processor is constructed to control the display system to update the display of the rendered graphics data for the first typed graphics data based on the detected second gesture.

6. The system of claim 1, wherein the display system processor is constructed to control the display system to update the display of the rendered graphics data for the first typed graphics data based on the detected first gesture of the first object associated with the first remote client device.

7. The system of claim 1, wherein the display system processor is constructed to control the display system to update the display of the rendered graphics data for the second typed graphics data based on the detected first gesture of the first object associated with the first remote client device.

8. The system of claim 1, wherein the display system processor is constructed to control the display system to update the display of the rendered graphics data for the second typed graphics data based on the detected second gesture of the second object associated with the second remote client device.

9. The system of claim 1, wherein the first object is a wand.

10. The system of claim 1, wherein the first object is an input device.

11. A method for decoupling applications in multi-computer interactive configuration, the method comprising:

a first remote client device generating a first un-typed data structure that includes first typed graphics data of a first client application of the first remote client device;

the first remote client device providing the first un-typed data structure to a display system processor;

a second remote client device generating a second un-typed data structure that includes second typed graphics data of a second client application of the second remote client device;

the second remote client device providing the second un-typed data structure to the display system processor;

the display system processor including the first un-typed data structure and the second un-typed data structure in a first pool of the display system processor the display system processor rendering graphics data of the first pool;

the display system processor controlling at least one of a display system that includes a plurality of display devices, the first remote client device and the second remote client device to display the rendered graphics data of the first pool, wherein the rendered graphics data of the first pool includes rendered graphics data for the first typed graphics data and rendered graphics data for the second typed graphics data;

the display system processor simultaneously detecting a first gesture of a first object associated with the first remote client device and a second gesture of a second object associated with the second remote client device from gesture data received via a plurality of sensors, wherein the gesture data includes absolute three-space location data of an instantaneous state of the first object and the second object at a point in time and space, and

the display system processor controlling at least one of the display system, the first remote client device, and the second remote client device to update the display of the rendered graphics data for the first typed graphics data based on the detected second gesture,

wherein the first un-typed data structure and the second un-typed data structure have a format that is application-independent,

wherein the first client application is different from the second client application, and

wherein the first un-typed data structure and the second un-typed data structure have a same format.

12. The method of claim 11, wherein the display system processor controls the display system, the first client device and the second client device to simultaneously display the rendered graphics data of the first pool across the plurality of display devices of the display system, a display device of first remote client device and a display device of the second remote client device.

13. The method of claim 12, wherein the display device of first remote client device displays the rendered graphics data for the first typed graphics data, and wherein the display system processor controls the first remote client device to update the display of the rendered graphics data for the first typed graphics data based on the detected second gesture.

14. The method of claim 12, wherein the display device of second remote client device displays the rendered graphics data for the first typed graphics data, and wherein the display system processor controls the first remote client device to update the display of the rendered graphics data for the first typed graphics data based on the detected second gesture.

15. The method of claim 12, wherein one of the plurality of display devices of the display system displays the rendered graphics data for the first typed graphics data, and wherein the display system processor controls the display system to update the display of the rendered graphics data for the first typed graphics data based on the detected second gesture.

16. The method of claim 11, wherein the display system processor controls the display system to update the display of the rendered graphics data for the first typed graphics data based on the detected first gesture of the first object associated with the first remote client device.

17. The method of claim 11, wherein the display system processor controls the display system to update the display of the rendered graphics data for the second typed graphics data based on the detected first gesture of the first object associated with the first remote client device.

18. The method of claim 11, wherein the display system processor controls the display system to update the display of the rendered graphics data for the second typed graphics data based on the detected second gesture of the second object associated with the second remote client device.

19. The method of claim 11, wherein the first object is a wand.

20. The method of claim 11, wherein the first object is an input device.