Operating environment with gestural control and multiple client devices, displays, and users

• US 10,824,238 B2
• Filed: 03/28/2019
• Issued: 11/03/2020
• Est. Priority Date: 04/02/2009
• Status: Active Grant

• Current Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Original Assignee: Oblong Industries, Inc. (Oblong, Inc.)
• Inventors: Kramer, Kwindla Hultman, Underkoffler, John
• Primary Examiner(s): Shankar, Vijay
• Application Number:
  US16/368,692
• Publication Number:
  US 20190220100A1
• Time in Patent Office: 586 Days
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: G06V 40/113 Recognition of static hand signs
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 3/002 to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT
• CPC Class Codes: G09G 3/003 to produce spatial visual effects
• CPC Class Codes: H04L 67/025 for remote control or remote monitoring of applications

Abstract

Embodiments described herein includes a system comprising a processor coupled to display devices, sensors, remote client devices, and computer applications. The computer applications orchestrate content of the remote client devices simultaneously across the display devices and the remote client devices, and allow simultaneous control of the display devices. The simultaneous control includes automatically detecting a gesture of at least one object from gesture data received via the sensors. The detecting comprises identifying the gesture using only the gesture data. The computer applications translate the gesture to a gesture signal, and control the display devices in response to the gesture signal.

Claims

1. A system comprising:

a three-space tracking system configured to generate absolute three-space location data of an instantaneous state of an object within a three-space coordinate system; and

at least one processing device configured to display a multi-user virtual space of at least a first session across a plurality of display devices;

wherein the at least one processing device is configured to;

integrate content of at least one remote client device in the multi-user virtual space of the first session and update display of the multi-user virtual space to include the integrated content,

for each user of the first session, receive from the three-space tracking system absolute three-space location data of an instantaneous state of an object of the user within a three-space coordinate system, and

update display of the multi-user virtual space across the plurality of display devices simultaneously based on the received absolute three-space location data.

2. The system of claim 1,

wherein the at least one processing device is configured to integrate content of at least two remote client devices in the multi-user virtual space of the first session and update display of the multi-user virtual space to include the integrated content, and

wherein the at least one processing device is configured to update display of the integrated content of at least one remote client device based on the received absolute three-space location data of at least one user.

3. The system of claim 2, wherein the integrated content of each remote client device is video data of at least one application of the remote client device.

4. The system of claim 2,

wherein the at least one processing device is configured to update the first application of the first remote client device.

5. The system of claim 4, wherein the at least one processing device is configured to update the first application at the first remote client device based on absolute three-space location data received by the at least one processing device.

6. The system of claim 5, wherein the absolute three-space location data received by the at least one processing device includes at least one three-space location that corresponds to a physical display location at which one of the plurality of display devices displays the integrated content of the first application, within the three-space coordinate system.

7. The system of claim 1, wherein each remote client device is remote to the at least one processing device.

8. The system of claim 1, wherein absolute three-space location data for each user is user pointer data.

9. The system of claim 1, wherein the absolute three-space location data includes at least one three-space location that corresponds to a physical display location at which one of the plurality of display devices displays integrated content an application of a remote client device, within the three-space coordinate system.

10. The system of claim 1, wherein the at least one processing device is configured to:

in response to updating of display of integrated content of an application of a remote client device, update the application of the remote client device.

11. A method comprising:

displaying a multi-user virtual space of a first session across a plurality of display devices by using at least one processing device;

integrating content of at least one remote client device in the multi-user virtual space of the first session and updating display of the multi-user virtual space to include the integrated content;

for each user of the first session, receiving absolute three-space location data of an instantaneous state of an object of the user within a three-space coordinate system; and

updating display of the multi-user virtual space across the plurality of display devices simultaneously based on the received absolute three-space location data.

12. The method of claim 11,

wherein integrating content comprises:

the at least one processing device integrating content of at least two remote client devices in the multi-user virtual space of the first session and updating display of the multi-user virtual space to include the integrated content, and

wherein updating display of the multi-user virtual space comprises;

the at least one processing device updating display of the integrated content of at least one remote client device based on the received absolute three-space location data of at least one user.

13. The method of claim 12, wherein the integrated content of each remote client device is video data of at least one application of the remote client device.

14. The method of claim 13,

wherein updating display of integrated content of a first client application of a first remote client device comprises:

updating the first application of the first remote client device.

15. The method of claim 14, wherein the at least one processing device updates the first application at the first remote client device based on absolute three-space location data received by the at least one processing device.

16. The method of claim 15, wherein the absolute three-space location data received by the at least one processing device includes at least one three-space location that corresponds to a physical display location at which one of the plurality of display devices displays the integrated content of the first application, within the three-space coordinate system.

17. The method of claim 11, wherein each remote client device is remote to the at least one processing device.

18. The method of claim 11, wherein absolute three-space location data for each user is user pointer data.

19. The method of claim 11, wherein the absolute three-space location data is received by the at least one processing device and includes at least one three-space location that corresponds to a physical display location at which one of the plurality of display devices displays integrated content an application of a remote client device, within the three-space coordinate system.

20. The method of claim 11, wherein updating display of the multi-user virtual space comprises:

in response to updating of display of integrated content of an application of a remote client device, the at least one processing device updating the application of the remote client device.