NUI video conference controls

• US 9,274,606 B2
• Filed: 03/14/2013
• Issued: 03/01/2016
• Est. Priority Date: 03/14/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Yee, Dawson
• Primary Examiner(s): Nguyen, Joseph J
• Application Number:
  US13/828,558
• Publication Number:
  US 20140267548A1
• Time in Patent Office: 1,083 Days
• Field of Search: 348/14.03, 348/14.07, 704/272
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A system and method providing gesture controlled video conferencing includes a local capture device detecting movements of a user in a local environment and an audio/visual display. A processor is coupled to the capture device and a remote capture device and a remote processor at a remote environment via a network. The local processor includes instructions to render a representation of the remote environment on the display responsive to the remote processor and remote capture device. The processor also tracks movements of a local user in a space proximate to the local capture device. Responsive to a user gesture detected at the local capture device, the audio or visual signals provided by the remote capture device are altered to change the representation of the remote location is altered locally.

Claims

1. A method of providing a video conference between a local location and a remote location, the method comprising:

receiving at the local location, video and audio signal obtained from a remote capture device disposed at the remote location, the obtained signals including those that are representative of real sights and real sound captured at the remote location by the remote capture device and are thus representative of real aspects of the remote location, the remote capture device being part of a remote system that is operatively coupled via a network to a local system having a local capture device disposed at the local location, the remote capture device including a respective remote depth camera;

presenting at the local location, a respective local representation of the remote location including that derived from the obtained video and audio signal that are respectively representative of real sights and real sound as captured by the remote capture device, the local representation including a displayed representation presented on a local display proximate to the local capture device;

tracking movements of a local user in a space proximate to the local capture device; and

responsive to a user'"'"'s local gesture detected by the local capture device, controlling the remote system to alter how it captures and processes reality representing signals at the remote location including altering processing of audio or visual signals by the remote system such that the respective local representation of real aspects of the remote location is correspondingly altered, said altering including altering of processing of depth based visual signals having a depth attribute provided by the remote depth camera.

2. The method of claim 1 wherein:

the remote capture device includes a respective remote array of audio sensors;

the local gesture includes an audio control gesture configured to alter an audio focus of the remote capture device by altering of processing at the remote location of sensed audio signals provided by the remote array of audio sensors such that the respective local representation of real audio aspects of the remote location is correspondingly altered.

3. The method of claim 2 wherein the local audio control gesture causes an audio input at the remote location to increase or decrease in volume for a focused-upon remote region proximate to the remote capture device.

4. The method of claim 1 wherein the local gesture includes a video control gesture configured to alter a predetermined video processing at the remote location of a video signal that is representative of real sights obtained from beyond a gesture indicated distance away from and captured by the remote capture device.

5. The method of claim 4 wherein the local video control gesture causes a video processing carried out by the remote system to change such that the respective local representation of the remote location beyond the indicated distance is correspondingly altered.

6. The method of claim 1 further including the step of presenting a user interface responsive to the user gesture, the user interface presenting additional input options to the local user to alter presentation of the video conference.

7. The method of claim 1 wherein the gesture comprises a human motion interaction with a projection of the remote environment on the display.

8. The method of claim 1 wherein:

the local capture device includes a respective local depth camera;

the method further including receiving a control signal from the remote system and responsively altering at the local location video signals output from the local system to the remote location, said altering at the local location including altering of processing of depth based visual signals having a depth attribute provided by the local depth camera.

9. The method of claim 8 wherein the remote capture device includes one or more permissions determining whether to alter remote capture device processing responsive to the control input from the local capture device.

10. A gesture controlled video conferencing apparatus configured for providing a video conference between a local environment and a remote environment, the apparatus comprising:

a local capture device configured to detect movements of a local user in the local environment;

an audio/visual display disposed in the local environment;

a processor coupled to the local capture device, the processor being further coupled via a network to a combination of a remote capture device and a remote processor disposed at the remote environment, the remote capture device including a respective remote depth camera, the processor including code instructing the processor to;

render a representation of the remote environment on the audio/visual display, the rendered representation including at least one of reality representing audio and video portions corresponding to real sights and real sounds captured and processed at the remote environment;

track movements of the local user in a space proximate to the local capture device; and

responsive to a predetermined user gesture detected at the local capture device, alter a corresponding visual portion of the rendered representation of real sights and real sounds captured and processed at the remote environment, the altering at the remote environment including altering of processing of depth based visual signals having a depth attribute provided by the remote depth camera, whereby due to the altering at the remote environment, the reality representing representation of the real sights of the remote environment is altered locally on the audio/visual display.

11. The video conferencing apparatus of claim 10 wherein:

the remote capture device includes a respective remote array of audio sensors;

the local processing device is configured to output a control signal to the remote environment that causes an altering of processing at the remote environment of sensed audio signals provided by the remote array of audio sensors so as to thereby cause an alteration of an audio focus provided by the remote capture device.

12. The video conferencing apparatus of claim 11 wherein the user gesture is configured to cause an audio input at the remote environment to correspondingly increase or decrease in volume for a focused-upon remote region proximate to the remote capture device.

13. The video conferencing apparatus of claim 10 wherein the gesture is configured to alter a predetermined video processing of an obtained video signal that is representative of real sights obtained from beyond a gesture indicated distance away from and captured by the remote capture device so as to thereby change the video representation at the local display.

14. The video conferencing apparatus of claim 10 wherein the local processing device outputs a control signal to the remote location the step of presenting a user interface responsive to the user gesture, the user interface presenting additional input options to the local user to alter presentation of the video conference.

15. The video conferencing apparatus of claim 10 wherein the processor includes instructions for receiving a control signal from the remote processing device and altering video and audio signals output to the remote location.

16. A gesture controlled video conferencing apparatus configured for providing a video conference between a local first physical environment and a remote second physical environment, the apparatus comprising:

a depth and image capture device configured to detect movements of a user in the first physical environment;

an audio/visual display disposed in the first physical environment;

a first processor coupled to the capture device and the audio/visual display, the processor being further coupled via a network to a second depth and image capture device and a second processor disposed in the remote second physical environment, the second depth and image capture device being configured to capture real aspects of the second physical environment and to provide corresponding audio visual signals, the first processor including code instructing the first processor to;

render a representation of real aspects of the second physical environment on the display responsive to the corresponding audio visual signals provided by the second depth and image capture device;

present a user interface on the audio/visual display, the user interface responsive to gestures from the user;

track movements of the user in a space proximate to the first capture device and responsive to a predetermined user gesture by the user, presenting a user interface allowing the user a set of one or more controls for altering how the second depth and image capture device is used for capturing and processing reality representing signals at the second physical environment to thereby alter the representation of the real aspects of the remote second physical environment as rendered on the audio/visual display.

17. The gesture controlled video conferencing apparatus of claim 16 wherein the predetermined gesture includes a region identifying selection of a region of the display proximate to a remote user represented in the display.

18. The gesture controlled video conferencing apparatus of claim 17 wherein the predetermined gesture includes a manipulation type identification indicative of a desired type of manipulating of the presentation of the region selected in the display.

19. The gesture controlled video conferencing apparatus of claim 18 wherein the processor includes instructions for receiving a control signal from the remote processing device and for responsively altering video and audio signals output to the remote location, the responsively altered video and audio signals including those representing real aspects of the local first physical environment.

20. The gesture controlled video conferencing apparatus of claim 19 wherein the processor includes one or more permissions determining whether to alter local processing in response to the received control signal from the remote location.