Operating environment with gestural control and multiple client devices, displays, and users

• US 10,296,099 B2
• Filed: 05/17/2017
• Issued: 05/21/2019
• Est. Priority Date: 04/02/2009
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Kramer, Kwindla Hultman, Underkoffler, John
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US15/597,687
• Publication Number:
  US 20170329413A1
• Time in Patent Office: 734 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G09G 3/002 to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT
• CPC Class Codes: H04L 67/025 for remote control or remote monitoring of applications

Abstract

Embodiments described herein includes a system comprising a processor coupled to display devices, sensors, remote client devices, and computer applications. The computer applications orchestrate content of the remote client devices simultaneously across the display devices and the remote client devices, and allow simultaneous control of the display devices. The simultaneous control includes automatically detecting a gesture of at least one object from gesture data received via the sensors. The detecting comprises identifying the gesture using only the gesture data. The computer applications translate the gesture to a gesture signal, and control the display devices in response to the gesture signal.

Claims

1. A system comprising:

a display processor that is communicatively coupled to a plurality of display devices;

a first remote client device that is communicatively coupled to the display processor, wherein the first remote client device includes a first input device and a first client application; and

a second remote client device that is communicatively coupled to the display processor, wherein the second remote client device includes a second input device and a second client application,

wherein the first remote client device is constructed to provide first video data to the display processor, wherein the first video data includes at least display output of the first client application,

wherein the second remote client device is constructed to provide second video data to the display processor, wherein the second video data includes at least display output of the second client application,

wherein the first remote client device and the second remote client device are remote to the display processor and the plurality of display devices, and

wherein the display processor is constructed to;

integrate the first video data and the second video data in a first application session of the display processor;

control the plurality of display devices to display rendered data of the first application session;

responsive to receiving first pointer data from the first input device via the first remote client device, the first pointer data corresponding to the displayed rendered data that corresponds to the second client application, controlling the second client application based on the received first pointer data; and

responsive to receiving second pointer data from the second input device via the second remote client device, the second pointer data corresponding to the displayed rendered data that corresponds to the first client application, controlling the first client application based on the received second pointer data.

2. The system of claim 1, wherein the display processor is communicatively coupled to a plurality of sensors.

3. The system of claim 1, wherein each of the first remote client device and the second remote client device provide an element of a workflow of the first application session.

4. The system of claim 3, wherein the display processor is constructed to integrate pixels of the element of the first remote client device and the element of the second remote client device into the workflow of the first application session.

5. The system of claim 4, wherein the display processor is constructed to provide a collaborative interface for a plurality of remote client devices communicatively coupled to the display processor, wherein the collaborative interface is constructed to provide any remote client device of the plurality of remote client devices with direct interaction with applications running on any other remote client device of the plurality of remote client devices, wherein direct interaction with an application running on a remote client device comprises control of an element of the remote client device that is included in the workflow of the first application session.

6. A method comprising:

at a display processor that is communicatively coupled to a plurality of display devices;

accessing first video data from a first remote client device that is communicatively coupled to the display processor, wherein the first video data includes at least display output of a first client application of the first remote client device;

integrating the accessed first video data in a first application session of the display processor;

controlling the plurality of display devices to display rendered data of the first application session;

receiving pointer data from an input device of a second remote client device, wherein the received pointer data corresponds to the displayed rendered data that corresponds to the first client application; and

controlling the first client application of the first remote client device based on the received pointer data,

wherein the first remote client device and the second remote client device are remote to the display processor and the plurality of display devices, and

wherein the first remote client device and the second remote client device are communicatively coupled to the display processor.

7. The method of claim 6, wherein the second remote client device is constructed to:

generate a first un-typed data structure that includes the pointer data of the input device of the second remote client device; and

provide the first un-typed data structure to the display processor,

wherein the pointer data has a data type that corresponds to a second client application of the second remote client device, and

wherein the first un-typed data structure that includes the pointer data has a format that is application-independent.

8. The method of claim 7, wherein the display processor controls the plurality of display devices to display rendered data of the first application session by using a first application of the display processor, wherein the first application of the display processor corresponds to a type that is different from a type of the second client application of the second remote client device.

9. The method of claim 8, further comprising:

at the display processor;

receiving pointer data from an input device of a third remote client device, wherein the received pointer data from the input device of the third remote client device corresponds to the displayed rendered data that corresponds to the first client application; and

controlling the first client application of the first remote client device based on the received pointer data of the input device of the third remote client device,

wherein the first remote client device, the second remote client device, and the third remote client device are remote to the display processor and the plurality of display devices, and

wherein the third remote client device is communicatively coupled to the display processor.

10. The method of claim 8, further comprising:

at the display processor;

accessing second video data from a third remote client device that is communicatively coupled to the display processor, wherein the second video data includes at least display output of a third client application of the third remote client device;

integrating the accessed first video data and the accessed second video data in the first application session of the display processor;

controlling the plurality of display devices to display rendered data of the first application session;

receiving pointer data from the input device of the second remote client device, wherein the received pointer data from the input device of the second remote client device corresponds to the displayed rendered data that corresponds to the third client application; and

controlling the third client application of the third remote client device based on the received pointer data from the input device of the second remote client device,

wherein the first remote client device, the second remote client device, and the third remote client device are remote to the display processor and the plurality of display devices, and

wherein the third remote client device is communicatively coupled to the display processor.

11. The method of claim 8, further comprising:

at the display processor;

accessing second video data from the second remote client device, wherein the second video data includes at least display output of the second client application of the second remote client device;

integrating the accessed first video data and the accessed second video data in the first application session of the display processor;

controlling the plurality of display devices to display rendered data of the first application session;

receiving pointer data from an input device of the first remote client device, wherein the received pointer data from the input device of the first remote client device corresponds to the displayed rendered data that corresponds to the second client application; and

controlling the second client application of the second remote client device based on the received pointer data from the input device of the first remote client device.

12. The method of claim 11, wherein the type of the first application of the display system processor is different from a type of the first client application of the first remote client device.

13. The method of claim 11, further comprising:

the display processor authenticating the first remote client device for access to the first application session; and

the display processor authenticating the second remote client device for access to the first application session.

14. The method of claim 6, wherein the first application session is an application session of a collaborative application.

15. The method of claim 6, wherein the display processor is communicatively coupled to a plurality of sensors.

16. The method of claim 6, wherein each of the first remote client device and the second remote client device provide an element of a workflow of the first application session.

17. The method of claim 16, wherein the display processor integrates pixels of the element of the first remote client device and the element of the second remote client device into the workflow of the first application session.

18. The method of claim 17, wherein the display processor provides a collaborative interface for a plurality of remote client devices communicatively coupled to the display processor, wherein the collaborative interface provides any remote client device of the plurality of remote client devices with direct interaction with applications running on any other remote client device of the plurality of remote client devices, wherein direct interaction with an application running on a remote client device comprises control of an element of the remote client device that is included in the workflow of the first application session.