Operating environment with gestural control and multiple client devices, displays, and users

• US 10,739,865 B2
• Filed: 06/04/2019
• Issued: 08/11/2020
• Est. Priority Date: 04/24/2008
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Kramer, Kwindla Hultman, Underkoffler, John, Sparrell, Carlton, Singh, Navjot, Hollenbach, Kate, Yarin, Paul
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US16/430,913
• Publication Number:
  US 20190286243A1
• Time in Patent Office: 434 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/02 Input arrangements using manually operated switches, e.g. using keyboards or dials
• CPC Class Codes: G06F 3/0236 using selection techniques to select from displayed items
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0325 using a plurality of light emitters or reflectors or a plurality of detectors forming a reference frame from which to derive the orientation of the object, e.g. by triangulation or on the basis of reference deformation in the picked up image
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: H04L 67/025 for remote control or remote monitoring of applications
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Embodiments described herein includes a system comprising a processor coupled to display devices, sensors, remote client devices, and computer applications. The computer applications orchestrate content of the remote client devices simultaneously across the display devices and the remote client devices, and allow simultaneous control of the display devices. The simultaneous control includes automatically detecting a gesture of at least one object from gesture data received via the sensors. The detecting comprises identifying the gesture using only the gesture data. The computer applications translate the gesture to a gesture signal, and control the display devices in response to the gesture signal.

Claims

1. A system comprising:

a multi-user collaboration server constructed to control at least one display device to display content of at least a first collaboration session that includes content streams received from a plurality of remote client devices;

a plurality of wands; and

a tracking system;

wherein the collaboration server is constructed to receive three-dimensional position and orientation information for each of the plurality of wands by using the tracking system, and

wherein the collaboration server is constructed to control the at least one display device to update display of at least one displayed first object of the first collaboration session based on a change in at least one of position and orientation of a first wand that is aimed at the displayed first object, as determined by the tracking system.

2. The system of claim 1, wherein the collaboration server is constructed to receive a plurality of content streams from a first remote client device of the plurality of remote client devices.

3. The system of claim 1,

wherein the collaboration server is constructed to:

control at least one display device to display content of a second collaboration session that includes content streams received from a second plurality of remote client devices,

receive three-dimensional position and orientation information for each of a second plurality of wands, and

control the at least one display device to update display of at least one displayed second object of the second collaboration session based on a change in at least one of position and orientation of a wand of the second plurality of wands that is aimed at the displayed second object.

4. The system of claim 1, wherein the first object is a video stream object.

5. The system of claim 2, wherein the collaboration server is constructed to update at least one of a display size of the video stream object and a display location of the video stream object.

6. The system of claim 1, wherein the first object is one of a plurality of objects included in a content stream received from one of the plurality of remote client devices.

7. The system of claim 6, wherein the collaboration server is constructed to update at least one of a display size of the first object, a display location of the first object, and an appearance of the first object.

8. The system of claim 1, wherein the at least one display device includes a display device of the collaboration server.

9. The system of claim 8,

wherein the at least one display device further includes a display device of a first remote client device of the plurality of remote client devices that provides content streams included in the first collaboration session, and

wherein the collaboration server is constructed to simultaneously update display of the at least one displayed first object at the display device of the collaboration server and the display device of the first remote client device.

10. A method comprising:

with a multi-user collaboration server;

controlling at least one display device to display content of at least a first collaboration session that includes content streams received from a plurality of remote client devices;

receiving three-dimensional position and orientation information for each of a plurality of wands by using a tracking system;

controlling the at least one display device to update display of at least one displayed first object of the first collaboration session based on a change in at least one of position and orientation of a first wand that is aimed at the displayed first object, as determined by a tracking system.

11. The method of claim 10, further comprising:

with the collaboration server, receiving a plurality of content streams from a first remote client device of the plurality of remote client devices.

12. The method of claim 10, further comprising with the collaboration server:

controlling at least one display device to display content of a second collaboration session that includes content streams received from a second plurality of remote client devices;

receiving three-dimensional position and orientation information for each of a second plurality of wands; and

controlling the at least one display device to update display of at least one displayed second object of the second collaboration session based on a change in at least one of position and orientation of a wand of the second plurality of wands that is aimed at the displayed second object.

13. The method of claim 10, wherein the first object is a video stream object.

14. The method of claim 11, wherein controlling the at least one display device to update display of at least one displayed first object comprises at least one of updating a display size of the video stream object and updating a display location of the video stream object.

15. The method of claim 10, wherein the first object is one of a plurality of objects included in a content stream received from one of the plurality of remote client devices.

16. The method of claim 15, wherein controlling the at least one display device to update display of at least one displayed first object comprises at least one of updating a display size of the first object, updating a display location of the first object, and updating an appearance of the first object.

17. The method of claim 10, wherein the at least one display device includes a display device of the collaboration server.

18. The method of claim 17,

wherein the at least one display device further includes a display device of a first remote client device of the plurality of remote client devices that provides content streams included in the first collaboration session, and

wherein updating display of the at least one displayed first object comprises simultaneously updating display of the at least one displayed first object at the display device of the collaboration server and the display device of the first remote client device.

19. The method of claim 10, further comprising with the collaboration server:

controlling at least one display device to display content of a second collaboration session that includes content streams received from a second plurality of remote client devices;

receiving three-dimensional position and orientation information for each of a second plurality of wands; and

controlling the at least one display device to update display of at least one displayed second object of the second collaboration session based on a change in at least one of position and orientation of a wand of the second plurality of wands that is aimed at the displayed second object,

wherein the collaboration server receives a first content stream for the first collaboration session and a second content stream for the second collaboration session from a remote client device included in the second plurality of remote client devices.

20. The method of claim 10, further comprising:

with the collaboration server, controlling a display device of the collaboration server to display a unique pointer for at least one wand.