Operating environment with gestural control and multiple client devices, displays, and users

• US 9,880,635 B2
• Filed: 04/28/2017
• Issued: 01/30/2018
• Est. Priority Date: 04/02/2009
• Status: Expired due to Fees

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Kramer, Kwindla Hultman, Underkoffler, John, Sparrell, Carlton, Singh, Navjot, Hollenbach, Kate, Yarin, Paul
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US15/582,243
• Publication Number:
  US 20170300122A1
• Time in Patent Office: 277 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: H04L 67/025 for remote control or remote monitoring of applications
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Embodiments described herein includes a system comprising a processor coupled to display devices, sensors, remote client devices, and computer applications. The computer applications orchestrate content of the remote client devices simultaneously across the display devices and the remote client devices, and allow simultaneous control of the display devices. The simultaneous control includes automatically detecting a gesture of at least one object from gesture data received via the sensors. The detecting comprises identifying the gesture using only the gesture data. The computer applications translate the gesture to a gesture signal, and control the display devices in response to the gesture signal.

Claims

1. A system comprising:

a display system processor that is communicatively coupled to a display system that includes a plurality of display devices;

a first remote client device that is communicatively coupled to the display system processor, wherein the first remote client device includes first content of a first application session of the display system processor and includes a first client application, and wherein the first remote client device is communicatively coupled to a first input device;

a second remote client device that is communicatively coupled to the display system processor, wherein the second remote client device includes second content of the first application session and includes a second client application, and wherein the second remote client device is communicatively coupled to a second input device,

wherein the first client application and the second client application are constructed to integrate the respective first content and second content simultaneously in the first application session of the display system processor,

wherein the display system processor is constructed to control the display system to display the integrated content of the first application session,

wherein the first client application and the second client application are constructed to simultaneously provide first event data of the first input device and second event data of the second the input device to the display system processor, and

wherein the display system processor is constructed to control the display system to update the display of the integrated content of the first application session based on the simultaneously received first event data and second event data.

2. A method comprising:

at a display system processor that is communicatively coupled to a display system that includes a plurality of display devices;

simultaneously accessing first content of a first remote client device that is communicatively coupled to the display system processor and second content of a second remote client device that is communicatively coupled to the display system processor, and integrating the accessed first content and the accessed second content in a first application session of the display system processor;

controlling the display system to display the integrated content of the first application session;

simultaneously receiving first event data from a first input device of the first remote client device and second event data from a second input device of the second remote client device, and controlling the display system to update the display of the integrated content of the first application session based on the simultaneously received first event data and second event data,

wherein the first remote client device and the second remote client device are remote to the display system processor.

3. The method of claim 2, further comprising:

the display system processor generating the first application session.

4. The method of claim 2, wherein the first remote client device is constructed to:

detect the first event data of the first input device;

generate a first un-typed data structure that includes the first event data; and

provide the first un-typed data structure to the display system processor,

wherein the first event data has a data type that corresponds to a first client application of the first remote client device, and

wherein the first un-typed data structure that includes the first event data has a format that is application-independent.

5. The method of claim 4, wherein the display system processor controls the display system to display the content of the first application session by using a first application of the display system processor, wherein the first application of the display system processor corresponds to a type that is different from a type of the first client application of the first remote client device.

6. The method of claim 5, wherein the second remote client device is constructed to:

detect the second event data of the second input device;

generate a second un-typed data structure that includes the second event data; and

provide the second un-typed data structure to the display system processor,

wherein the second event data has a data type that corresponds to a second client application of the second remote client device, and

wherein the second un-typed data structure that includes the second event data has a format that is application-independent.

7. The method of claim 6, wherein the type of the first application of the display system processor is different from a type of the second client application of the second remote client device.

8. The method of claim 7, wherein the type that corresponds to the second client application of the second remote client device is different from the type that corresponds to the first client application of the first remote client device.

9. The method of claim 8, wherein the first client application generates the first event data, and wherein the second client application generates the second event data.

10. The method of Claim 9, wherein one of a plurality of applications of the first remote client device generates the first content of the first remote client device, and wherein the application of the first remote client device that generates the first content is different from the first client application.

11. The method of claim 2, further comprising:

the display system processor authenticating the first remote client device for access to the first application session, wherein the display system processor adds the first content to the first application session responsive to the authentication of the first remote client device.

12. The method of claim 11, further comprising:

the display system processor authenticating the second remote client device for access to the first application session, wherein the display system processor adds the second content to the first application session responsive to the authentication of the second remote client device.

13. The method of claim 2, wherein the first application session is an application session of a collaborative application.

14. The method of claim 2, further comprising:

at the display system processor;

simultaneously accessing third content of a third remote client device that is communicatively coupled to the display system processor and fourth content of a fourth remote client device that is communicatively coupled to the display system processor, and integrating the accessed third content and the accessed fourth content in a second application session of the display system processor;

controlling the display system to display the integrated content of the second application session;

simultaneously receiving third event data from a third input device of the third remote client device and fourth event data from a fourth input device of the fourth remote client device, and controlling the display system to update the display of the integrated content of the second application session based on the simultaneously received third event data and fourth event data,

wherein the third remote client device and the fourth remote client device are remote to the display system processor.

15. The method of claim 2, wherein the display system processor provides simultaneous control of the content of the first application session by the first input device of the first remote client device and the second input device of the second remote client device.

16. The method of claim 2, wherein controlling the display system to display the integrated content of the first application session comprises:

controlling one display device of the display system to display the integrated content of the first application session.

17. The method of claim 2, wherein controlling the display system to display the integrated content of the first application session comprises:

controlling at least two display devices of the display system to display the integrated content of the first application session.

18. The method of claim 6, wherein the first remote client device is constructed to:

generate a third un-typed data structure that includes the first content; and

provide the third un-typed data structure to the display system processor,

wherein the third un-typed data structure that includes the first content has a format that is application-independent.

19. The method of claim 18, wherein the first content has a data type that corresponds to a third client application of the first remote client device.

20. The method of claim 18, wherein the first content has a data type that corresponds to the first client application of the first remote client device.