Methods and apparatuses for controlling video content displayed to a viewer

• US 9,930,270 B2
• Filed: 10/15/2015
• Issued: 03/27/2018
• Est. Priority Date: 10/15/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Soni, Shahil, Kankaanp, Esa
• Primary Examiner(s): Elahi, Shan E
• Application Number:
  US14/883,952
• Publication Number:
  US 20170111595A1
• Time in Patent Office: 894 Days
• Field of Search: 348 36
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04L 65/1089 by adding media; by removing media
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/613 for the control of the source by the destination control signals issued by the client directed to the server or network components specially adapted for selective content distribution H04N21/637
• CPC Class Codes: H04L 65/75 Media network packet handling
• CPC Class Codes: H04L 65/756 adapting media to device capabilities
• CPC Class Codes: H04L 65/80 Responding to QoS
• CPC Class Codes: H04N 1/387 Composing, repositioning or otherwise geometrically modifying originals
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 5/2628 Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
• CPC Class Codes: H04N 7/147 : Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/152 : Multipoint control units therefor

Abstract

A method and an apparatus for controlling video content displayed to a viewer are presented. The method includes receiving video stream of a video. Two or more areas of interest are identified in a segment of the video stream and a display priority for each area of interest is determined. The method includes facilitating display of concurrently occurring portions of the segment if the display priority for the each area of interest is equal and the areas of interest can be simultaneously presented in a single view on a display screen. The concurrently occurring portions are displayed in a split screen arrangement if the areas of interest cannot be simultaneously presented in the single view. The portions of the segment related to one or more areas of interest are displayed based on an order of the display priority if the display priority for the each area of interest is unequal.

Claims

1. A method for controlling a video display, the method comprising:

capturing a video stream of a video, the video captured with at least one camera, using an image-capture field of view of at least 180 degrees;

identifying, by a processor, two or more areas of interest in the video stream comprising at least one image frame from among a series of image frames of the video stream, the two or more areas of interest identified based at least in part on one or more active objects detected in the video stream;

determining, by a processor, a display priority for each area of interest from among the two or more areas of interest based on predefined rules for assigning the display priority to the each area of interest based on an at least one of sound, motion, object posture or change of scene in the each area of interest;

determining, by a processor, whether the display priority for the each area of interest is equal or not;

in response to determining, by a processor, the display priority for the each area of interest to be equal by comparing the display priority assigned to the each area of interest, determining, by a processor, whether the two or more areas of interest are capable of being simultaneously presented in a single view on a display screen, and, performing;

facilitating, by a processor, display of concurrently occurring portions of the video stream related to the two or more areas of interest in the single view on the display screen if the two or more areas of interest are capable of being simultaneously presented in the single view on the display screen; and

facilitating, by a processor, display of the concurrently occurring portions of the video stream related to the two or more areas of interest using a split screen arrangement on the display screen if the two or more areas of interest are not capable of being simultaneously presented in the single view on the display screen; and

in response to determining the display priority for the each area of interest to be unequal, facilitating display of portions of the video stream related to one or more areas of interest from among the two or more areas of interest on the display screen based on an order of the display priority for the each area of interest,

wherein the display on the display screen is transitioned based on a change detected in the display priority of at least one area of interest from among the two or more areas of interest, the display transitioned using a predefined pace of display transition.

2. The method of claim 1, wherein the display is transitioned from an area of interest associated with a lower display priority to an area of interest associated with a higher display priority.

3. The method of claim 1, wherein the display is transitioned from an area of interest associated with a higher display priority to an area of interest associated with a lower display priority.

4. The method of claim 1, wherein the video is captured using a camera associated with the image-capture field of view of 360 degrees in at least one of a horizontal direction and a vertical direction.

5. The method of claim 1, wherein an active object from among the one or more active objects corresponds to an entity associated with at least one of motion and sound.

6. The method of claim 1, wherein the two or more areas of interest in the video stream are identified based on detecting the one or more active objects using at least one of face detection, sound detection, motion detection, thermal detection, gaze detection, posture detection, background scene detection and whiteboard detection.

7. The method of claim 1, wherein the plurality of rules for assigning the display priority further comprise one or more rules for assigning the display priority to the each area of interest based on whether the video stream is to be displayed on the display screen as a streaming video stream or a recorded video stream.

8. The method of claim 1, further comprising:

computing distances from the each area of interest to remaining areas of interest from among the two or more areas of interest;

comparing the computed distances with a predetermined distance measure; and

determining one of;

the two or more areas of interest are capable of being simultaneously presented in the single view on the display screen if the computed distances are less than or equal to the predetermined distance measure; and

the two or more areas of interest are not capable of being simultaneously presented in the single view on the display screen if the computed distances are greater than the predetermined distance measure.

9. The method of claim 8, wherein the predetermined distance measure is computed based at least in part on an aspect ratio of the display screen configured to display the video stream.

10. The method of claim 1, wherein determining whether the two or more areas of interest are capable of being simultaneously presented in the single view on the display screen is performed based on a number of active objects associated with the two or more areas of interest.

11. The method of claim 1, wherein the predefined pace of display transition is configured to afford to a viewer of the display screen, a gradual display experience from the one area of interest to the another area of interest, and wherein the predefined pace of display transition is configured to be adjusted to a rate of motion activity or sound activity associated with the one or more active objects within the two or more areas of interest.

12. An apparatus comprising:

at least one processor;

at least one camera; and

at least one memory comprising computer program code, the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to at least perform;

capture a video stream of a video by the at least one camera using an image-capture field of view of at least 180 degrees;

identify two or more areas of interest in the video stream comprising at least one image frame from among the series of image frames of the video stream, the two or more areas of interest identified based at least in part on one or more active objects detected in the video stream;

determine a display priority for each area of interest from among the two or more areas of interest based on predefined rules for assigning the display priority to the each area of interest based on an at least one of sound, motion, object posture or change of scene in the each area of interest;

determine whether the display priority for the each area of interest is equal or not;

in response to determining the display priority for the each area of interest to be equal by comparing the display priority assigned to the each area of interest, determine whether the two or more areas of interest are capable of being simultaneously presented in a single view on a display screen, and, perform;

facilitate display of concurrently occurring portions of the video stream related to the two or more areas of interest in the single view on the display screen if the two or more areas of interest are capable of being simultaneously presented in the single view on the display screen; and

facilitate display of the concurrently occurring portions of the video stream related to the two or more areas of interest using a split screen arrangement on the display screen if the two or more areas of interest are not capable of being simultaneously presented in the single view on the display screen; and

in response to determining the display priority for the each area of interest to be unequal, facilitate display of portions of the video stream related to one or more areas of interest from among the two or more areas of interest on the display screen based on an order of the display priority for the each area of interest,

wherein the display on the display screen is transitioned based on a change detected in the display priority of at least one area of interest from among the two or more areas of interest, the display transitioned using a predefined pace of display transition.

13. The apparatus of claim 12, further comprising:

a camera configured to capture the video, the camera associated with the image-capture field of view of 360 degrees in at least one of a horizontal direction and a vertical direction.

14. The apparatus of claim 12, wherein the apparatus is further caused, at least in part to perform one of:

transition the display from an area of interest associated with a lower display priority to an area of interest associated with a higher display priority; and

transition the display from an area of interest associated with a higher display priority to an area of interest associated with a lower display priority.

15. The apparatus of claim 12, wherein the apparatus is further caused, at least in part to:

compute distances from the each area of interest to remaining areas of interest from among the two or more areas of interest;

compare the computed distances with a predetermined distance measure, the predetermined distance measure computed based at least in part on an aspect ratio of the display screen configured to display the video stream; and

determine one of;

the two or more areas of interest are capable of being simultaneously presented in the single view on the display screen if the computed distances are less than or equal to the predetermined distance measure; and

the two or more areas of interest are not capable of being simultaneously presented in the single view on the display screen if the computed distances are greater than the predetermined distance measure.

16. The apparatus of claim 12, wherein the predefined pace of display transition is configured to afford to a viewer of the display screen, a gradual display experience from the one area of interest to the another area of interest, and, wherein the predefined pace of display transition is configured to be adjusted to a rate of motion activity or sound activity associated with the one or more active objects within the two or more areas of interest.

17. A computer-readable storage device, the computer-readable storage device comprising a set of instructions, which, when executed by one or more processors, cause an apparatus to at least perform:

capture a video stream of a video by at least one camera using an image-capture field of view of at least 180 degrees;

identify two or more areas of interest in the video stream comprising at least one image frame from among the series of image frames of the video stream, the two or more areas of interest identified based at least in part on one or more active objects detected in the video stream;

determine a display priority for each area of interest from among the two or more areas of interest based on predefined rules for assigning the display priority to the each area of interest based on an at least one of sound, motion, object posture or change of scene in the each area of interest;

determine whether the display priority for the each area of interest is equal or not by comparing the display priority assigned to the each area of interest;

in response to determining the display priority for the each area of interest to be equal, determine whether the two or more areas of interest are capable of being simultaneously presented in a single view on a display screen, and, perform;

facilitate display of concurrently occurring portions of the video stream related to the two or more areas of interest in the single view on the display screen if the two or more areas of interest are capable of being simultaneously presented in the single view on the display screen; and

facilitate display of the concurrently occurring portions of the video stream related to the two or more areas of interest using a split screen arrangement on the display screen if the two or more areas of interest are not capable of being simultaneously presented in the single view on the display screen; and

in response to determining the display priority for the each area of interest to be unequal, facilitate display of portions of the video stream related to one or more areas of interest from among the two or more areas of interest on the display screen based on an order of the display priority for the each area of interest,

wherein the display on the display screen is transitioned based on a change detected in the display priority of at least one area of interest from among the two or more areas of interest, the display transitioned using a predefined pace of display transition.

18. The computer-readable storage device of claim 17, wherein the apparatus is further caused, at least in part to:

compute distances from the each area of interest to remaining areas of interest from among the two or more areas of interest;

compare the computed distances with a predetermined distance measure, the predetermined distance measure computed based at least in part on an aspect ratio of the display screen configured to display the video stream; and

determine one of;

the two or more areas of interest are capable of being simultaneously presented in the single view on the display screen if the computed distances are less than or equal to the predetermined distance measure; and

the two or more areas of interest are not capable of being simultaneously presented in the single view on the display screen if the computed distances are greater than the predetermined distance measure.

19. The computer-readable storage device of claim 17 wherein the display is transitioned from an area of interest associated with a lower display priority to an area of interest associated with a higher display priority.

20. The computer-readable storage device of claim 17 wherein the display is transitioned from an area of interest associated with a higher display priority to an area of interest associated with a lower display priority.