Methods of modifying search areas

• US 10,708,617 B2
• Filed: 03/07/2017
• Issued: 07/07/2020
• Est. Priority Date: 07/31/2015
• Status: Active Grant

• Current Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Original Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Inventors: Wu, Yannan, Tang, Xiaozheng, Chen, Wei, Cao, Zisheng, Wang, Mingyu
• Primary Examiner(s): Vo, Tung T
• Application Number:
  US15/452,680
• Publication Number:
  US 20170180754A1
• Time in Patent Office: 1,218 Days
• Field of Search: 37524016
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/56 Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search
• CPC Class Codes: H04N 19/57 Motion estimation characterised by a search window with variable size or shape
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

Methods and systems for evaluating a search area for encoding video are provided. The method comprises receiving video captured by an image capture device, the video comprising video frame components. Additionally, the method comprises receiving optical flow field data associated with the video frame component, wherein at least a portion of the optical flow field data is captured by sensors. The method also comprises determining a search area based on the optical flow field data.

Claims

1. A method of evaluating a search area for encoding video, said method comprising:

receiving video captured by an image capture device, the video comprising video frame components;

receiving sensor data from a plurality of sensors, the sensor data including information about a spatial disposition of the image capture device, a movement of the image capture device, a zoom state of the image capture device, and a field of view (FOV) of the image capture device;

generating, using the sensor data, an optical flow field associated with a video frame component of the video frame components;

obtaining a first initial search area within the video frame component;

determining, based on the optical flow filed, a bias of the first initial search area;

adjusting, based on the optical flow field, the first initial search area to determine a first search area within the video frame component, wherein adjusting the first initial search area includes, based on the bias of the first initial search area;

adjusting a number of sides of a boundary of the first initial search area from a first number to a second number that is different from the first number;

or

adjusting the boundary of the first initial search area by a first ratio in one direction and adjusting the boundary of the first initial search area by a second ratio in another direction, the first ratio being different from the second ratio;

providing a second initial search area of the video frame component; and

adjusting the second initial search area based on the optical flow field to yield a second search area.

2. The method of claim 1, further comprising evaluating the first search area within the video frame component to be encoded using motion information of the video captured by the image capture device.

3. The method of claim 2, wherein the step of evaluating the first search area occurs while a movable object carrying the image capture device is in flight, and wherein the movable object is an unmanned aerial vehicle (UAV).

4. The method of claim 1, wherein adjusting the first initial search area further includes varying a dimension of the first initial search area.

5. The method of claim 1, wherein adjusting the first initial search area further includes varying a location of the first initial search area.

6. The method of claim 5, wherein the optical flow field bias the location of an initial point of the first initial search area based on a motion vector of a previous frame.

7. The method of claim 1, wherein adjusting the first initial search area further includes vertically elongating the first initial search area when the optical flow field indicates a vertical bias.

8. The method of claim 1, wherein adjusting the first initial search area further includes moving a position of the first initial search area on a video frame component along a circular arc when the optical flow field indicates a rotation.

9. The method of claim 2, wherein the first search area is evaluated to identify a motion vector within the video frame component.

10. The method of claim 1, wherein the second search area is smaller than the first search area.

11. The method of claim 1, wherein the video frame component is a block within a video frame, wherein the video frame comprises a plurality of blocks.

12. The method of claim 11, wherein the block is an encoding video block.

13. The method of claim 1, wherein the image capture device is installed on an unmanned aerial vehicle (UAV).

16. The method of claim 1, wherein the optical flow field includes a plurality of arrows that indicate a plurality of optical flow vectors, and adjusting the first initial search area further includes adjusting the first initial search area based on a density of the plurality of arrows.

17. The method of claim 1,

wherein the video includes one or more objects;

the method further comprising;

determining, based on the optical flow field, whether the image capture device moves with respect to the one or more objects faster in the one direction than in the another direction,

in response to determining the image capture device moves with respect to the one or more objects faster in the one direction than in the another direction, adjusting the boundary of the first initial search area by the first ratio in the one direction and adjusting the boundary of the first initial search area by the second ratio in the another direction, the first ratio being greater than the second ratio.

14. A non-transitory computer readable medium containing program instructions for evaluating a search area for encoding video, said computer readable medium comprising:

program instructions for receiving video captured by an image capture device, the video comprising video frame components;

program instructions for receiving sensor data from a plurality of sensors, the sensor data including information about a spatial disposition of the image capture device, a movement of the image capture device, a zoom state of the image capture device, and a field of view (FOV) of the image capture device;

program instructions for generating, using the sensor data, an optical flow field associated with a video frame component of the video frame components;

program instructions for obtaining a first initial search area within the video frame component;

program instructions for determining a bias of the first initial search area based on the optical flow field;

program instructions for adjusting, based on the optical flow field, the first initial search area to determine a first search area within the video frame component, wherein adjusting the first initial search area includes, based on the bias of the first initial search area;

adjusting a number of sides of a boundary of the first initial search area from a first number to a second number that is different from the first number;

or

adjusting the boundary of the first initial search area by a first ratio in one direction and adjusting the boundary of the first initial search area by a second ratio in another direction, the first ratio being different from the second ratio;

program instructions for providing a second initial search area of the video frame component; and

program instructions for adjusting the second initial search area based on the optical flow field to yield a second search area.

15. A system for evaluating a search area for encoding video, said system comprising:

an image capture device configured to capture a video;

one or more sensors configured to obtain sensor data; and

one or more processors, individually or collectively configured to;

receive the video captured by the image capture device, the video comprising video frame components;

receive the sensor data from the one or more sensors, the sensor data including information about a spatial disposition of the image capture device, a movement of the image capture device, a zoom state of the image capture device, and a field of view (FOV) of the image capture device;

generate, using the sensor data, an optical flow field associated with a video frame component of the video frame components;

obtain a first initial search area within the video frame component;

determine, according to the optical flow field, a bias of the first initial search area; and

adjust, based on the optical flow field, the first initial search area to determine a first search area within the video frame component, wherein adjusting the first initial search area includes, based on the bias of the first initial search area;

adjusting a number of sides of a boundary of the first initial search area from a first number to a second number that is different from the first number;

or

adjusting the boundary of the first initial search area by a first ratio in one direction and adjusting the boundary of the first initial search area by a second ratio in another direction, the first ratio being different from the second ratio;

provide a second initial search area of the video frame component; and

adjust the second initial search area based on the optical flow field to yield a second search area.