Selection and tracking of objects for display partitioning and clustering of video frames

• US 9,589,595 B2
• Filed: 12/18/2014
• Issued: 03/07/2017
• Est. Priority Date: 12/20/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Gao, Dashan, Zhong, Xin, Kandhadai, Ananthapadmanabhan Arasanipalai, Beith, Scott, Laver, Steven Douglas
• Primary Examiner(s): Zhao, Daquan
• Application Number:
  US14/575,945
• Publication Number:
  US 20150179219A1
• Time in Patent Office: 810 Days
• Field of Search: 386/280, 386/278, 386/283, 386/239, 386/248, 348/169, 348/142, 348/25, 348/208.14
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/269 using gradient-based methods
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G08B 13/19 using infrared-radiation detection systems G08B13/194 takes precedence
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253
• CPC Class Codes: H04N 9/87 Regeneration of colour television signals H04N9/80 takes precedence

Abstract

A method includes selecting at least two objects within a primary video stream, generating a first video stream from the primary video stream that includes a first of the selected objects, and generating a second video stream from the primary video stream that includes a second of the selected objects. The primary video stream has a primary field of view, and the first and second video streams have respective first and second fields of view that are more narrow than the primary field of view. The first field of view includes a portion of the primary field of view that is not within the second field of view, and the second field of view includes a portion of the primary field of view that is not within the first field of view.

Claims

1. A method of video processing, the method comprising:

receiving a first selection of a first object represented within a primary video stream that has a primary field of view;

in response to receiving the first selection, generating, from the primary video stream, a first video stream including the first object and having a first field of view that is narrower than the primary field of view;

receiving a second selection of a second object represented within the primary video stream; and

in response to receiving the second selection, generating, from the primary video stream, a second video stream including the second object and having a second field of view that is narrower than the primary field of view,

wherein the first video stream is generated prior to receiving the second selection and video corresponding to the second video stream is displayed on a screen while video corresponding to the first video stream is displayed on the screen, and wherein the first field of view includes a portion of the primary field of view that is not within the second field of view and the second field of view includes a portion of the primary field of view that is not within the first field of view.

2. The method of video processing according to claim 1, further comprising:

detecting, within at least one of a first plurality of frames of the primary video stream or a second plurality of frames of the first video stream, a movement of the first object; and

changing a direction of the first field of view in response to detecting the movement.

3. The method of video processing according to claim 2, wherein changing the direction of the first field of view comprises changing the direction to follow the movement of the first object.

4. The method of video processing according to claim 1, wherein generating the first video stream comprises generating each frame of the first video stream from a portion of at least one corresponding frame of the primary video stream.

5. The method of video processing according to claim 1, wherein, for each frame of the first video stream, a first distance between a first coordinate position of a centroid of a first representation of the first object within the frame and a center coordinate of the frame is less than a second distance between a second coordinate position of a centroid of a second representation of the first object within a corresponding frame of the primary video stream and a center coordinate of the corresponding frame.

6. The method of video processing according to claim 1, wherein generating the first video stream comprises normalizing a size of a representation of the first object within each of a first plurality of frames of the first video stream.

7. The method of video processing according to claim 1, wherein generating the first video stream comprises normalizing at least one of brightness, contrast, or white balance within each of a first plurality of frames of the first video stream.

8. The method of video processing according to claim 1, further comprising:

displaying, on the screen, the first video stream within a first window; and

displaying, on the screen, the second video stream within a second window.

9. The method of video processing according to claim 1, further comprising displaying on the screen an indication of a geometric relationship between the first object and the second object.

10. The method of video processing according to claim 8, further comprising:

determining that, in a first frame of the primary video stream, the first object is closer to a first frame edge of the first frame than the second object is to the first frame edge;

subsequent to determining that the first object is closer to the first frame edge, determining that, in a second frame of the primary video stream, the second object is closer to a second frame edge of the second frame than the first object is to the second frame edge, the second frame edge corresponding to the first frame edge; and

in response to determining that the second object is closer to the second frame edge, displaying the second video stream within the first window and displaying the first video stream within the second window.

11. The method of video processing according to claim 8, further comprising:

subsequent to displaying the first video stream within the first window, determining that the second object overlaps the first object in at least one of a frame of the primary video stream, a frame of the first video stream, or a frame of the second video stream; and

in response to determining that the second object overlaps the first object, displaying the second video stream within the first window and displaying the first video stream within the second window.

12. The method of video processing according to claim 8, further comprising:

generating, from the primary video stream, a display video stream having a secondary field of view that is narrower than the primary field of view;

displaying, within a third window of the screen, the display video stream; and

in response to detecting that at least one of the first object or the second object is near an edge of a frame of the display video stream, changing the display video stream to have a tertiary field of view, wherein the tertiary field of view has a wider angle than the secondary field of view.

13. The method of video processing according to claim 1, wherein the first object is a robotic object, and wherein the method further comprises controlling a movement of the robotic object.

14. A non-transitory computer-readable medium storing instructions that when executed by a processor, cause the processor to:

receive a first selection of a first object represented within a primary video stream that has a primary field of view;

generate from the primary video stream, in response to receiving the first g selection, a first video stream including the first object and having a first field of view that is narrower than the primary field of view;

receive a second selection of a second object represented within the primary video stream; and

generate from the primary video stream, in response to receiving the second selection, a second video stream including the second object and having a second field of view that is narrower than the primary field of view,

wherein the first video stream is generated prior to receiving the second selection and video corresponding to the second video stream is displayed on a screen while video corresponding to the first video stream is displayed on the screen, and wherein the first field of view includes a portion of the primary field of view that is not within the second field of view and the second field of view includes a portion of the primary field of view that is not within the first field of view.

15. The non-transitory computer-readable medium according to claim 14, the medium storing instructions that when executed by the processor, cause the processor to:

detect a movement of the first object based on at least one of a first plurality of frames of the primary video stream or a second plurality of frames of the first video stream; and

change a direction of the first field of view, in response to said detecting the movement of the first object, to follow the movement of the first object.

16. The non-transitory computer-readable medium according to claim 14, wherein the instructions to generate the first video stream comprise instructions that, when executed by the processor, cause the processor to normalize a size of a representation of the first object within each of a first plurality of frames of the first video stream.

17. The non-transitory computer-readable medium according to claim 14, the medium storing instructions that, when executed by the processor, cause the processor to:

display, on the screen, the first video stream within a first window and the second video stream within a second window;

determine that, in a first frame of the primary video stream, the first object is closer to a first frame edge of the first frame than the second object is to the first frame edge;

subsequent to determining that the first object is closer to the first frame edge, determine that, in a second frame of the primary video stream, the second object is closer to a second frame edge of the second frame than the first object is to the second frame edge, the second frame edge corresponding to the first frame edge; and

in response to determining that the second object is closer to the second frame edge, display the second video stream within the first window and display the first video stream within the second window.

18. The non-transitory computer-readable medium according to claim 14, the medium storing instructions that when executed by the processor, cause the processor to:

display, on the screen, the first video stream within a first window and the second video stream within a second window;

subsequent to displaying the first video stream within the first window, determine that the second object overlaps the first object in at least one of a frame of the primary video stream, a frame of the first video stream, or a frame of the second video stream; and

in response to determining that the second object overlaps the first object, display the second video stream within the first window and the first video stream within the second window.

19. The non-transitory computer-readable medium according to claim 14, the medium storing instructions that when executed by the processor, cause the processor to:

display, on the screen, the first video stream within a first window and the second video stream within a second window;

generate, from the primary video stream, a display video stream having a secondary field of view that is narrower than the primary field of view;

display, within a third window of the screen, the display video stream; and

in response to detecting that at least one of the first object or the second object is near an edge of a frame of the display video stream change the display video stream to have a tertiary field of view, wherein the tertiary field of view has a wider angle than the secondary field of view.

20. An apparatus for video processing, said apparatus comprising:

means for receiving a first selection of a first object represented within a primary video stream that has a primary field of view;

means for generating from the primary video stream, in response to receiving the first selection, a first video stream including the first object and having a first field of view that is narrower than the primary field of view;

means for receiving a second selection of a second object represented within the primary video stream; and

means for generating from the primary video stream, in response to receiving the second selection, a second video stream including the second object and having a second field of view that is narrower than the primary field of view,

wherein the first video stream is generated prior to receiving the second selection and video corresponding to the second video stream is displayed on a screen while video corresponding to the first video stream is displayed on the screen, and wherein after generating the first video stream and generating the second video stream, the first field of view includes a portion of the primary field of view that is not within the second field of view and the second field of view includes a portion of the primary field of view that is not within the first field of view.

21. The apparatus according to claim 20, wherein the means for generating the first video stream is configured to normalize a size of a representation of the first object within each of a first plurality of frames of the first video stream.

22. The apparatus according to claim 20, further comprising:

means for displaying, on the screen, the first video stream within a first window and the second video stream within a second window;

means for determining that, in a first frame of the primary video stream, the first object is closer to a first frame edge of the first frame than the second object is to the first frame edge;

means for determining, subsequent to determining that the first object is closer to the first frame edge, that in a second frame of the primary video stream, the second object is closer to a second frame edge of the second frame than the first object is to the second frame edge, the second frame edge corresponding to the first frame edge; and

means for displaying, in response to determining that the second object is closer to the second frame edge, the second video stream within the first window and the first video stream within the second window.

23. The apparatus according to claim 20, further comprising:

means for displaying, on the screen, the first video stream within a first window and the second video stream within a second window;

means for determining, subsequent to displaying the first video stream within the first window, that the second object overlaps the first object in at least one of a frame of the primary video stream, a frame of said first video stream, or a frame of the second video stream; and

means for displaying, in response to determining the second object overlaps the first object, the second video stream within the first window and the first video stream within the second window.

24. The apparatus according to claim 20, further comprising:

means for displaying, on the screen, the first video stream within a first window and the second video stream within a second window;

means for generating, from the primary video stream, a display video stream having a secondary field of view that is narrower than the primary field of view;

means for displaying, within a third window of the screen, the display video stream;

means for, in response to detecting that at least one of the first object or the second object is near an edge of a frame of the display video stream, changing the display video stream to have a tertiary field of view, wherein the tertiary field of view has a wider angle than the secondary field of view.

25. An apparatus for video processing, the apparatus comprising:

a discriminator configured receive a first selection of a first object represented within a primary video stream that has a primary field of view and to receive a second selection of a second object represented within the primary video stream; and

a video stream generator configured to generate from the primary video stream, in response to receiving the first selection, a first video stream including the first object and having a first field of view that is narrower than the primary field of view, and to generate from the primary video stream, in response to receiving the second selection, a second video stream including the second object and having a second field of view that is narrower than the primary field of view,

wherein the first video stream is generated prior to receiving the second selection and video corresponding to the second video stream is displayed on a screen while video corresponding to the first video stream is displayed on the screen, and wherein after generating the first video stream and generating the second video stream, the first field of view includes a portion of the primary field of view that is not within the second field of view and the second field of view includes a portion of the primary field of view that is not within the first field of view.

26. The apparatus according to claim 25, further comprising an object tracker configured to detect a movement of the first object based on at least one of a first plurality of frames of the primary video stream or a second plurality of frames of the first video stream, wherein the video stream generator is configured to, in response to detecting the movement of the first object, change a direction of the first field of view to follow the movement of the first object.

27. The apparatus according to claim 25, wherein the video stream generator is configured to normalize a size of a representation of the first object within each of a first plurality of frames of the first video stream.

28. The apparatus according to claim 25, further comprising:

a display signal generator configured to display, on the screen, the first video stream within a first window and the second video stream within a second window; and

an object tracker configured to;

determine that, in a first frame of the primary video stream, the first object is closer to a first frame edge of the first frame than the second object is to the first frame edge, and

determine, subsequent to said determining that the first object is closer to the first frame edge, that in a second frame of the primary video stream, the second object is closer to a second frame edge of the second frame than the first object is to the second frame edge, the second frame edge corresponding to the first frame edge,

wherein the display signal generator is configured to display, subsequent to displaying the first video stream within the first window, and in response to determining that the second object is closer to the second frame edge, the second video stream within the first window and the first video stream within the second window.

29. The apparatus according to claim 25, further comprising:

a display signal generator configured to display, on a screen, the first video stream within a first window and the second video stream within a second window; and

an object tracker configured to determine that the second object overlaps the first object in at least one of a frame of the primary video stream, a frame of the first video stream, or a frame of the second video stream,

wherein the display signal generator is configured to display, subsequent to displaying the first video stream within the first window, and in response to determining that the second object overlaps the first object, the second video stream within the first window and the first video stream within the second window.

30. The apparatus according to claim 25, wherein the video stream generator is configured to generate, from the primary video stream, a display video stream having a secondary field of view that is narrower than the primary field of view, the apparatus further comprising:

a display signal generator configured to display, on a screen, the first video stream within a first window, the second video stream within a second window, and the display video stream within a third window; and

an object tracker configured to detect that, in a frame of the display video stream, at least one of the first object or the second object is near an edge of the frame, wherein the video stream generator is configured to change the display video stream to have a tertiary field of view in response to the object tracker detecting that the first object or the second object is near the edge, and wherein the tertiary field of view has a wider angle than the secondary field of view.