Natural user interface having video conference controls

• US 9,787,943 B2
• Filed: 02/18/2016
• Issued: 10/10/2017
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Yee, Dawson
• Primary Examiner(s): Nguyen, Joseph J
• Application Number:
  US15/047,019
• Publication Number:
  US 20160165186A1
• Time in Patent Office: 600 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A system and method providing gesture controlled video conferencing includes a local capture device detecting movements of a user in a local environment and an audio/visual display. A processor is coupled to the capture device and a remote capture device and a remote processor at a remote environment via a network. The local processor includes instructions to render a representation of the remote environment on the display responsive to the remote processor and remote capture device. The processor also tracks movements of a local user in a space proximate to the local capture device. Responsive to a user gesture detected at the local capture device, the audio or visual signals provided by the remote capture device are altered to change the representation of the remote location is altered locally.

Claims

1. A method of providing signals by way of a network from a remote site to a local site where the provided signals are provided in accordance with controls generated at the local site, the method comprising:

causing acquisition from sensors at the remote site of first signals representing relatively rich information about real aspects of the remote site;

causing processing at the remote site of the first signals representing the relatively rich information, the caused processing utilizing relatively rich signal processing resources present at the remote site, the caused processing producing second signals representing relatively less rich information about real aspects of the remote site where the less rich information can be transmitted over the network using less bandwidth than needed for transmitting the first signals representing the relatively rich information;

causing transmission of the less rich information over the network from the remote site to the local site; and

causing transmission of control signals generated at the local site to the remote site, the transmitted control signals controlling said processing at the remote site of the first signals.

2. The method of claim 1 wherein the locally generated control signals include first and second local control signals, the first local control signals being configured to select an aspect of the remote processing that is to be controlled by the second local control signals.

3. The method of claim 2 wherein the first local control signals are configured to select at least one of an audio processing aspect and a video processing aspect of the remote processing.

4. The method of claim 3 wherein the audio processing aspect of the remote processing that is selectable by the first local control signals includes processing of respective audio signals acquired from respective ones of plural audio sensors among the sensors at the remote site.

5. The method of claim 4 wherein the audio processing aspect of the remote processing that is selectable by the first local control signals includes audio array processing whereby an area of audio pickup focus may be provided at the remote site.

6. The method of claim 3 wherein the first local control signals that are locally generated are derived from first user gestures detected at the local site.

7. The method of claim 6 wherein the detected first user gestures include gestures pointing to at least one of visual representations of sound sources present at the remote site and visual representations of image sources present at the remote site.

8. The method of claim 3 wherein the video processing aspect of the remote processing that is selectable by the first local control signals includes processing of respective image signals acquired from respective ones of plural image sensors among the sensors at the remote site.

9. The method of claim 8 wherein the remotely acquired image signals include signals indicative of three dimensional (3D) aspects of imagery present at the remote site and the controlled processing includes remote processing of the 3D aspects of the imagery present at the remote site.

10. The method of claim 9 wherein the remote processing of the 3D aspects of the imagery present at the remote site includes control of image brightness based on depth of the remote image sources.

11. A machine system configured to provide signals by way of a network from a remote site to a local site where the provided signals are provided in accordance with controls generated at the local site, the machine system comprising:

remote sensors disposed at the remote site and configured to produce first signals representing relatively rich information about real aspects of the remote site;

one or more processors disposed at the remote site and configured to process at the remote site the first signals representing the relatively rich information, wherein the processing by remotely disposed one or more processors includes relatively rich signal processing and production therefrom of second signals representing relatively less rich information about real aspects of the remote site where the less rich information can be transmitted over the network using less bandwidth than needed for transmitting the first signals representing the relatively rich information;

a network interface transmitter disposed at the remote site and configured to transmit the less rich information over the network from the remote site to the local site; and

a network interface receiver disposed at the remote site and configured to receive control signals generated at the local site, the received control signals controlling said processing by the one or more processors at the remote site of the first signals.

12. The machine system of claim 11 wherein:

the received control signals include first and second control signals generated at the local site, the first received control signals being configured to select an aspect of the processing at the remote site that is to be controlled by the second received control signals.

13. The machine system of claim 12 wherein the first received control signals are configured to select at least one of an audio processing aspect and a video processing aspect of the processing at the remote site.

14. The machine system of claim 13 wherein the audio processing aspect that is selectable by the first received control signals includes processing of respective audio signals acquired from respective ones of plural audio sensors among the sensors disposed at the remote site.

15. The machine system of claim 14 wherein the audio processing aspect of the remote processing that is selectable by the first received control signals includes audio array processing whereby an area of audio pickup focus may be provided at the remote site.

16. The machine system of claim 13 and further comprising:

one or more gesture detectors disposed at the local site and configured to detect user gestures made at the local site;

wherein the first received control signals that are locally generated are derived from first user gestures detected at the local site by the one or more gesture detectors.

17. The machine system of claim 16 and further comprising:

a display disposed at the local site and configured to display at least one of visual representations of sound sources present at the remote site and visual representations of image sources present at the remote site;

wherein the detected first user gestures include gestures pointing to at least one of displayed visual representations of the sound sources present at the remote site and of the visual representations of image sources present at the remote site.

18. The machine system of claim 13 wherein:

the video processing aspect of the processing at the remote site that is selectable by the first received control signals includes processing of respective image signals acquired from respective ones of plural image sensors among the sensors at the remote site; and

the image signals acquired at the remote site include signals indicative of three dimensional (3D) aspects of imagery present at the remote site and the controlled processing includes remote processing of the 3D aspects of the imagery present at the remote site.

19. One or more processor readable storage devices having instructions encoded thereon which when executed cause one or more processors to perform a method of providing signals by way of a network from a remote site to a local site where the provided signals are provided in accordance with controls generated at the local site, the method comprising:

causing acquisition from sensors at the remote site of first signals representing relatively rich information about real aspects of the remote site;

causing processing at the remote site of the first signals representing the relatively rich information, the caused processing utilizing relatively rich signal processing resources present at the remote site, the caused processing producing second signals representing relatively less rich information about real aspects of the remote site where the less rich information can be transmitted over the network using less bandwidth than needed for transmitting the first signals representing the relatively rich information;

causing transmission of the less rich information over the network from the remote site to the local site; and

causing transmission of control signals generated at the local site to the remote site, the transmitted control signals controlling said processing at the remote site of the first signals.

20. The processor readable storage devices of claim 19 wherein the locally generated control signals include first and second local control signals, the first local control signals being configured to select an aspect of the remote processing that is to be controlled by the second local control signals.