Operating environment with gestural control and multiple client devices, displays, and users

• US 10,353,483 B2
• Filed: 08/01/2018
• Issued: 07/16/2019
• Est. Priority Date: 04/24/2008
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Kramer, Kwindla Hultman, Underkoffler, John, Sparrell, Carlton, Singh, Navjot, Hollenback, Kate, Yarin, Paul
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US16/051,829
• Publication Number:
  US 20180348883A1
• Time in Patent Office: 349 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: H04L 67/025 for remote control or remote monitoring of applications
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Embodiments described herein includes a system comprising a processor coupled to display devices, sensors, remote client devices, and computer applications. The computer applications orchestrate content of the remote client devices simultaneously across the display devices and the remote client devices, and allow simultaneous control of the display devices. The simultaneous control includes automatically detecting a gesture of at least one object from gesture data received via the sensors. The detecting comprises identifying the gesture using only the gesture data. The computer applications translate the gesture to a gesture signal, and control the display devices in response to the gesture signal.

Claims

1. A method comprising:

a multi-user collaboration server integrating content streams received from a plurality of remote client devices in a single first collaboration session of the collaboration server;

the collaboration server controlling at least one display device to display content of the first collaboration session;

a tracking system detecting each wand of the collaboration server;

the tracking system simultaneously determining three-dimensional position and orientation information for each wand of the collaboration server;

the collaboration server controlling the at least one display device to display a unique pointer for each wand that is aimed at the at least one display device as determined by the three-dimensional position and orientation information of each of the plurality of wands; and

the collaboration server controlling the at least one display device to update display of at least a displayed first object of the first collaboration session based on a change in at least one of position and orientation of a first wand that is aimed at the displayed first object, as determined by the tracking system.

2. The method of claim 1,

wherein the collaboration server controlling the at least one display device to display a unique pointer for each wand that is aimed at the at least one display device as determined by the three-dimensional position and orientation information of each of the plurality of wands comprises:

the collaboration server controlling the at least one display device to display a first unique pointer for the first wand that is aimed at the at least one display device and a second unique pointer for a second wand that is aimed at the at least one display device.

3. The method of claim 2,

wherein the collaboration server controls the at least one display device to update display of at least the displayed first object of the first collaboration session based on a change in at least one of position and orientation of one of:

the first wand that is aimed at the at least one display device, and

the second wand that is aimed at the at least one display device.

4. The method of claim 2,

wherein the collaboration server controls the at least one display device to update display of the first displayed object of the first collaboration session based on a change in at least one of position and orientation of the first wand that is aimed at the at least one display device, and

wherein the collaboration server controls the at least one display device to update display of a second displayed object of the first collaboration session based on a change in at least one of position and orientation of the second wand that is aimed at the at least one display device.

5. The method of claim 1,

wherein the collaboration server simultaneously controls display of the first object and a second object of the first collaboration session at the at least one display device, and

wherein the collaboration server simultaneously:

updates display of the first object based on a change in at least one of position and orientation of the first wand that is aimed at the displayed first object, as determined by the tracking system, and

updates display of the second object based on a change in at least one of position and orientation of a second wand that is aimed at the displayed second object, as determined by the tracking system.

6. The method of claim 5, wherein the first object is an object of a first content stream received from a first remote client device and the second object is an object of a second content stream received from a second remote client device.

7. The method of claim 6, wherein the collaboration server simultaneously:

updates display of the first object based on a change in at least one of position and orientation of the second wand that is aimed at the displayed first object, as determined by the tracking system, and

updates display of the second object based on a change in at least one of position and orientation of the first wand that is aimed at the displayed second object, as determined by the tracking system.

8. The method of claim 6, wherein the collaboration server updates display of the first object based on a change in at least one of position and orientation of the second wand that is aimed at the displayed first object, as determined by the tracking system.

9. The method of claim 6, wherein the collaboration server updates display of the second object based on a change in at least one of position and orientation of the first wand that is aimed at the displayed second object, as determined by the tracking system.

10. The method of claim 1, wherein the collaboration server integrates a first content stream received from a first remote client device in the first collaboration session, and wherein the collaboration server integrates one of the first content stream and a second content stream received from the first remote client device in a second collaboration session.

11. A system comprising:

a multi-user collaboration server constructed to integrate content streams received from a plurality of remote client devices in a single first collaboration session of the collaboration server, and control at least one display device to display content of the first collaboration session;

a first wand; and

a tracking system constructed to detect each wand of the collaboration server,

wherein the tracking system is constructed to simultaneously determine three-dimensional position and orientation information for each wand of the collaboration server,

wherein the collaboration server is constructed to control the at least one display device to display a unique pointer for each wand that is aimed at the at least one display device as determined by the three-dimensional position and orientation information of each wand of the collaboration server, and

wherein the collaboration server is constructed to control the at least one display device to update display of at least a displayed first object of the first collaboration session based on a change in at least one of position and orientation of the first wand in a case where the first wand is aimed at the displayed first object, as determined by the tracking system.

12. The system of claim 11,

wherein the collaboration server is constructed to control the at least one display device to display a first unique pointer for the first wand that is aimed at the at least one display device and a second unique pointer for a second wand that is aimed at the at least one display device.

13. The system of claim 12,

wherein the collaboration server is constructed to control the at least one display device to update display of at least the displayed first object of the first collaboration session based on a change in at least one of position and orientation of one of:

the first wand that is aimed at the at least one display device, and

the second wand that is aimed at the at least one display device.

14. The system of claim 12,

wherein the collaboration server is constructed to control the at least one display device to update display of the first displayed object of the first collaboration session based on a change in at least one of position and orientation of the first wand that is aimed at the at least one display device, and

wherein the collaboration server is constructed to control the at least one display device to update display of a second displayed object of the first collaboration session based on a change in at least one of position and orientation of the second wand that is aimed at the at least one display device.

15. The system of claim 11,

wherein the collaboration server is constructed to simultaneously control display of the first object and a second object of the first collaboration session at the at least one display device, and

wherein the collaboration server is constructed to simultaneously:

update display of the first object based on a change in at least one of position and orientation of the first wand that is aimed at the displayed first object, as determined by the tracking system, and

update display of the second object based on a change in at least one of position and orientation of a second wand that is aimed at the displayed second object, as determined by the tracking system.

16. The system of claim 15, wherein the first object is an object of a first content stream received from a first remote client device and the second object is an object of a second content stream received from a second remote client device.

17. The system of claim 16, wherein the collaboration server is constructed to simultaneously:

update display of the first object based on a change in at least one of position and orientation of the second wand that is aimed at the displayed first object, as determined by the tracking system, and

update display of the second object based on a change in at least one of position and orientation of the first wand that is aimed at the displayed second object, as determined by the tracking system.

18. The system of claim 16, wherein the collaboration server is constructed to update display of the first object based on a change in at least one of position and orientation of the second wand that is aimed at the displayed first object, as determined by the tracking system.

19. The system of claim 16, wherein the collaboration server is constructed to update display of the second object based on a change in at least one of position and orientation of the first wand that is aimed at the displayed second object, as determined by the tracking system.