Methods and systems for mobile telepresence

• US 8,471,888 B2
• Filed: 12/11/2009
• Issued: 06/25/2013
• Est. Priority Date: 08/07/2009
• Status: Active Grant

• Current Assignee: Malikie Innovations Limited (Key Patent Innovations Limited)
• Original Assignee: Blackberry Limited
• Inventors: George, Richard John, Oliver, Brian Alexander
• Primary Examiner(s): Gauthier, Gerald
• Application Number:
  US12/636,270
• Publication Number:
  US 20110032324A1
• Time in Patent Office: 1,292 Days
• Field of Search: 348/14.02, 348/14.03, 348/14.08, 348/14.12, 348/121, 348/14.01, 348/14.09, 348/14.16, 348/742, 370/401, 370/260, 455/508, 455/416, 704/3, 706/11, 709/204, 709/227, 709/206, 709/221, 715/1.1, 715/701, 715/753, 715/757, 345/161, 379/93.21, 381/92
• US Class Current: 348/14.02
• CPC Class Codes: G06F 1/3203 Power management, i.e. event-based initiation of a power-saving mode
• CPC Class Codes: H04L 65/1069 Session establishment or de-establishment
• CPC Class Codes: H04L 65/1104 Session initiation protocol [SIP]
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 21/2343 involving reformatting operations of video signals for distribution or compliance with end-user requests or end-user device requirements video transcoding H04N19/40; media packet handling at the source H04L65/762
• CPC Class Codes: H04N 21/2662 Controlling the complexity of the video stream, e.g. by scaling the resolution or bitrate of the video stream based on the client capabilities
• CPC Class Codes: H04N 21/41407 embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44209 Monitoring of downstream path of the transmission network originating from a server, e.g. bandwidth variations of a wireless network arrangements for maintenance or administration in data switching networks involving bandwidth and capacity management H04L41/0896
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4424 Monitoring of the internal components or processes of the client device, e.g. CPU or memory load, processing speed, timer, counter or percentage of the hard disk space used error monitoring in general G06F11/30; arrangements for monitoring conditions of receiving stations in broadcast systems H04H60/32; diagnosis, testing or measuring for television receivers H04N17/04
• CPC Class Codes: H04N 21/44245 Monitoring the upstream path of the transmission network, e.g. its availability, bandwidth
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4858 for modifying screen layout parameters, e.g. fonts, size of the windows
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04W 4/00 Services specially adapted for wireless communication networks; Facilities therefor
• CPC Class Codes: H04W 48/08 Access restriction or access information delivery, e.g. discovery data delivery signalling during connection H04W76/00
• CPC Class Codes: H04W 52/0261 managing power supply demand, e.g. depending on battery level
• CPC Class Codes: H04W 88/02 : Terminal devices
• CPC Class Codes: Y02D 30/70 : in wireless communication networks

Abstract

Exemplary methods and systems relate to telepresence at mobile devices. Aspects comprise dynamic configuration of video provided from a server that intermediates between a mobile device and a telepresence system during a telepresence session. The server receives high definition video from the telepresence system and transcodes into a format that can indicated by the mobile device. The mobile device can dynamically update the format in response to real-time battery life and network interface status. A progressive reduction in bandwidth and power consumptive activities can be implemented. The device can implement face detection and face centering, front/back mobile device camera switching, and automatic contrast control and other video quality controls in an outgoing video stream.

Claims

1. A method for telepresence at a mobile device, comprising:

determining a desired composition of telepresence session data to be provided to the mobile device from a remote server, based on a plurality of high-definition video sources provided to the server, the determining at least in part based on a real-time availability of resources to the mobile device, the resources comprising current status of a battery powering the mobile device and bandwidth available over a wireless network;

transmitting information representative of the desired composition of telepresence session data to the remote server;

receiving the telepresence session data through an interface to the wireless network; and

performing portions of the telepresence session data on a display and through an audio speaker at the mobile device.

2. The method of claim 1, wherein the determining of the desired composition of the telepresence session data comprises determining a desired composition of a video stream, based on one or more of the plurality of high-definition video sources, to be provided by the remote server to the mobile device.

3. The method of claim 1, wherein the determining further comprises accepting an input at the mobile device and interpreting the input as a selection of one or more of the plurality of high-definition video sources, and using that selection in determining the desired composition of telepresence session data.

4. The method of claim 1, wherein the determining of the desired composition of telepresence session data comprises available power and network bandwidth resources of the mobile device.

5. The method of claim 1, wherein the determining comprises selecting a video stream composition requiring higher bandwidth upon detection of availability of a higher bandwidth wireless network connection.

6. The method of claim 5, wherein the determining comprises selecting a video stream composition requiring lower bandwidth upon detection of lack of availability of the higher bandwidth wireless network connection.

7. The method of claim 1, wherein the determining comprises selecting a format for the desired composition from a list of available formats ordered according to resource consumption rate to receive video in that format.

8. The method of claim 1, wherein the determining comprises selecting a video stream composition from video stream compositions comprising a roster view, an active speaker view, and a thumbnail matrix view.

9. The method of claim 1, wherein the determining is further based on one or more of a resolution of the display at the remote device, a size of the display, and availability of video and audio codecs at the remote device.

10. A server-based method for providing telepresence-like capabilities to mobile devices, comprising:

making a connection to a telepresence system that receives high-definition video from cameras during a telepresence session;

receiving telepresence session configuration information updates from the mobile device during the telepresence session;

receiving a plurality of video streams from the telepresence system;

determining characteristics of telepresence session information to be provided to the mobile device based on accessed mobile device telepresence session configuration information, representative of accepted configuration information updates;

forming the telepresence session information to be provided to the mobile device according to the determined characteristics; and

transmitting the telepresence session information for reception by the mobile device.

11. The method of claim 10, wherein the forming of the telepresence session information to be provided to the mobile device comprises selecting a video composition format from available formats comprising a roster view, an active speaker view, and a thumbnail matrix view.

12. The method of claim 10, the determined characteristics comprising characteristics for a video stream comprising one or more of a selection from the received video streams, a frame rate, and a resolution.

13. A mobile telepresence-enabled device, comprising:

a display;

a network interface;

a memory resource;

processing resources coupled with the memory resource, the network interface, the display, for operation under battery power, and comprising one or more modules configured for

monitoring status of a battery providing the battery power and status of a network to which the network interface communicates,

controlling elements of a telepresence session involving the device by using one or more of the status of the battery and of the network in formulating control commands to a network-based server, the control commands indicating changes in telepresence session data that is to be transmitted from the server to the device, and

controlling the display based on the data that is received from the server.

14. The device of claim 13, further comprising a video camera, and wherein the one or more modules of the processing resources further are for encoding video from the video camera and transmitting the encoded video to the server.

15. The device of claim 13, wherein the one or more modules of the processing resources further are for detecting a face captured on video taken from a video camera provided with the device and maintaining an approximately fixed position of the face in a video stream outputted from the device.

16. The device of claim 13, wherein the one or more modules of the processing resources further are for adjusting contrast of video taken from a video camera provided with the device.

17. The device of claim 13, further comprising a first video camera provided to capture video from a side of the device on which the display is provided, and a second video camera provided to point from an opposite side of the device, wherein the one or more modules further comprise an input module for detecting a selection indicative of an active camera selection of either the first video camera or the second video camera.

18. A system for telepresence at mobile devices, comprising:

a telepresence system for connecting over a comparatively high-speed local area network connection with a telepresence studio, the telepresence system communicating through a firewall with one or more wide area networks;

a server, capable of communication with the telepresence system through the firewall and

over the one or more wide area networks; and

a mobile device capable of communication with the server, wherein the server and the mobile device are configured for mutual interoperation to dynamically adjust video streamed from the server to the mobile device, responsive to control commands provided by the mobile device to the server.

19. The system of claim 18, wherein the control commands are formed based on real-time status sensed by the mobile device of one or more of battery life and network capability.

20. The system of claim 18, wherein the control commands are formed based on real-time status sensed by the mobile device of one or more of battery life and network capability, and on one or more of mobile device screen resolution, available video codecs and available audio codecs.

21. The system of claim 18, wherein the control commands are formed based on inputs to the mobile device.

22. The system of claim 18, wherein the mobile device comprises two or more video cameras, and wherein the mobile device is further configured for transmitting, to the server, video captured by a selected video camera from the two or more video cameras.

23. The system of claim 18, wherein the mobile device comprises a plurality of video cameras, and wherein the mobile device is further configured for transmitting, to the server, video captured by one of the plurality of video cameras that is selected by a control command sent from the server to the mobile device.

24. A computer readable medium storing computer executable instructions for performing a telepresence method at a mobile device, comprising:

determining a desired composition of telepresence session data to be provided to the mobile device from a remote server, based on a plurality of high-definition video sources provided to the server, the determining at least in part based on a real-time availability of resources to the mobile device, the resources comprising current status of a battery powering the mobile device and bandwidth available over a wireless network;

transmitting information representative of the desired composition of telepresence session data to the remote server;

receiving the telepresence session data through an interface to the wireless network; and

performing portions of the telepresence session data on a display and through an audio speaker at the mobile device.