Operating environment with gestural control and multiple client devices, displays, and users

• US 10,067,571 B2
• Filed: 12/15/2017
• Issued: 09/04/2018
• Est. Priority Date: 04/24/2008
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Kramer, Kwindla Hultman, Underkoffler, John, Sparrell, Carlton, Singh, Navjot, Hollenback, Kate, Yarin, Paul
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US15/843,753
• Publication Number:
  US 20180107281A1
• Time in Patent Office: 263 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: H04L 67/025 for remote control or remote monitoring of applications
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Embodiments described herein includes a system comprising a processor coupled to display devices, sensors, remote client devices, and computer applications. The computer applications orchestrate content of the remote client devices simultaneously across the display devices and the remote client devices, and allow simultaneous control of the display devices. The simultaneous control includes automatically detecting a gesture of at least one object from gesture data received via the sensors. The detecting comprises identifying the gesture using only the gesture data. The computer applications translate the gesture to a gesture signal, and control the display devices in response to the gesture signal.

Claims

1. A telepresence system comprising:

a first multiuser collaboration system comprising a first collaboration server and a first plurality of display devices; and

a second multiuser collaboration system comprising a second collaboration server and a second plurality of display devices,

wherein the first collaboration server is constructed to;

control the first plurality of display devices to display content of a first application session of the telepresence system, and

integrate first content of a first remote client device that is communicatively coupled to the first collaboration server with the content of the first application session,

wherein the second collaboration server is constructed to;

control the second plurality of display devices to display the content of the first application session, and

integrate second content of a second remote client device that is communicatively coupled to the second collaboration server with the content of the first application session of the telepresence system,

wherein the first collaboration server is constructed to simultaneously receive first event data of a first input device of the first remote client device and second event data of a second input device of the second remote client device, and update display of the content of the first application session by the first plurality of display devices based on the simultaneously received first event data and second event data, and

wherein the second collaboration server is constructed to simultaneously receive the first event data and the second event data, and update display of the content of the first application session by the second plurality of display devices based on the simultaneously received first event data and second event data.

2. The system of claim 1, wherein the telepresence system is constructed to provide simultaneous control of content of the first application session by the first input device of the first remote client device and the second input device of the second remote client device.

3. The system of claim 2,

further comprising:

the first remote client device and the second remote client device,

wherein the first remote client device is constructed to;

detect the first event data of the first input device;

generate a first un-typed data structure that includes the first event data; and

provide the first un-typed data structure to the first collaboration server,

wherein the first event data has a data type that corresponds to a first client application of the first remote client device, and

wherein the first un-typed data structure that includes the first event data has a format that is application-independent.

4. The system of claim 3,

wherein the second remote client device is constructed to:

detect the second event data of the second input device;

generate a second un-typed data structure that includes the second event data; and

provide the second un-typed data structure to the second collaboration server,

wherein the second event data has a data type that corresponds to a second client application of the second remote client device, and

wherein the second un-typed data structure that includes the second event data has a format that is application-independent.

5. A multiuser collaboration system comprising:

a collaboration server; and

a plurality of display devices,

wherein the collaboration server is constructed to;

simultaneously integrate first content of a first remote client device and second content of a second remote client device in a first application session of the collaboration server,

control the plurality of display devices to display the integrated content of the first application session,

simultaneously receive first event data of a first input device of the first remote client device and second event data of a second the input device of the second remote client device, and

update display of the integrated content of the first application session by the plurality of display devices based on the simultaneously received first event data and second event data.

6. The system of claim 5, further comprising a tracking system constructed to provide spatial input to the collaboration server based on tracking of a wand.

7. The system of claim 6, further comprising a first camera, wherein the collaboration server is constructed to capture an image by using the first camera and integrate the captured image in the first application session.

8. The system of claim 7, further comprising a telepresence camera, wherein the collaboration server is constructed generate a telepresence video by using the telepresence camera.

9. The system of claim 5, further comprising:

a tracking system constructed to provide spatial input to the collaboration server based on tracking of a wand;

the wand;

a whiteboard camera;

a telepresence camera;

a microphone; and

a speaker.

10. The system of Claim 9, wherein the wand is a multi-modal input device.

11. The system of claim 5, wherein the collaboration server is constructed to authenticate the first remote client device for access to the first application session, wherein the collaboration server is constructed to add the first content to the first application session responsive to the authentication of the first remote client device, wherein the collaboration server is constructed to authenticate the second remote client device for access to the first application session, and wherein the collaboration server is constructed to add the second content to the first application session responsive to the authentication of the second remote client device.

12. The system of claim 5, wherein the collaboration server is constructed to:

simultaneously integrate third content of a third remote client device and fourth content of a fourth remote client device in a second application session of the collaboration server,

control the plurality of display devices to display the integrated content of the second application session,

simultaneously receive third event data of a third input device of the third remote client device and fourth event data of a fourth input device of the fourth remote client device, and

update display of the integrated content of the second application session by the plurality of display devices based on the simultaneously received third event data and fourth event data.

13. The system of claim 5, wherein the collaboration server is constructed to control the plurality of display devices to display the integrated content of the first application session by:

controlling one display device of the plurality of display devices to display the integrated content of the first application session.

14. The system of claim 5, wherein the collaboration server is constructed to control the plurality of display devices to display the integrated content of the first application session by:

controlling at least two display devices of the plurality of display devices to display the integrated content of the first application session.

15. The system of claim 8, further comprising:

the first remote client device and the second remote client device.

16. The system of claim 5, further comprising:

the first remote client device and the second remote client device.

17. The system of claim 16,

wherein the first remote client device is constructed to:

detect the first event data of the first input device;

generate a first un-typed data structure that includes the first event data; and

provide the first un-typed data structure to the collaboration server,

wherein the first event data has a data type that corresponds to a first client application of the first remote client device, and

wherein the first un-typed data structure that includes the first event data has a format that is application-independent.

18. The system of claim 17,

wherein the second remote client device is constructed to:

detect the second event data of the second input device;

generate a second un-typed data structure that includes the second event data; and

provide the second un-typed data structure to the collaboration server,

wherein the second event data has a data type that corresponds to a second client application of the second remote client device, and

wherein the second un-typed data structure that includes the second event data has a format that is application-independent.

19. A multiuser collaboration method comprising:

a collaboration server of a multiuser collaboration system simultaneously integrating first content of a first remote client device and second content of a second remote client device in a first application session of the collaboration server;

the collaboration server controlling a plurality of display devices of the multiuser collaboration system to display the integrated content of the first application session;

the collaboration server simultaneously receiving first event data of a first input device of the first remote client device and second event data of a second the input device of the second remote client device, and

the collaboration server updating the display of the integrated content of the first application session by the plurality of display devices based on the simultaneously received first event data and second event data.

20. The method of claim 19, further comprising:

the collaboration server authenticating the first remote client device for access to the first application session;

the collaboration server adding the first content to the first application session responsive to the authentication of the first remote client device;

the collaboration server authenticating the second remote client device for access to the first application session; and

the collaboration server adding the second content to the first application session responsive to the authentication of the second remote client device.