SYSTEM AND METHOD FOR CONSTRUCTING OPTICAL FLOW FIELDS

• US 20170188046A1
• Filed: 03/14/2017
• Published: 06/29/2017
• Est. Priority Date: 07/31/2015
• Status: Active Grant

• Current Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Original Assignee: SZ DJI Technology Co., Ltd. (d/b/a DJI) (iFlight Technology Company Limited)
• Inventors: WU, Yannan
• Granted Patent: US 10,321,153 B2
• Time in Patent Office: Days
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/182 the unit being a pixel
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/527 Global motion vector estimation

Abstract

A system constructs an optical flow field that corresponds with a selected video frame. The optical flow field is constructed based on a first motion of a mobile platform having an imaging device and a status of the imaging device. The first motion and the status are determined with measurements of sensors installed on the mobile platform and/or the imaging device installed on the mobile platform. The first motion includes at least one of a first rotation, a horizontal movement, or a vertical movement of the mobile platform. The status includes a rotation of the imaging device and/or an orientation of the imaging device relative to the mobile platform.

Claims

1. A method for constructing an optical flow field, comprising:

acquiring a motion of a mobile platform carrying an imaging device;

obtaining a status of the imaging device relative to the mobile platform; and

constructing a corresponding optical flow field for a frame based upon the motion of mobile platform and the status of the imaging device.

2. The method of claim 1, wherein the acquiring the motion comprises determining at least one of a rotation of the mobile platform, a first movement of the mobile platform in a horizontal direction, or a second movement of the mobile platform in a vertical direction.

3. The method of claim 1, wherein the obtaining the status comprises determining at least one of a motion of the imaging device or a position of the imaging device.

4. The method of claim 3, wherein the determining the at least one of the motion of the imaging device or the position of the imaging device comprises determining at least one of a zooming of the imaging device, a first rotation of the imaging device in a horizontal direction, a second rotation of the imaging device in a vertical direction, an orientation of the imaging device, or a tilt angle of the imaging device.

5. The method of claim 1, wherein the constructing comprises associating a pre-categorized optical flow field with an elementary scenario of at least one of the motion of the mobile platform or the status of the imaging device.

6. The method of claim 5, wherein:

the associating comprises associating a rotating view with the elementary scenario,

the rotating view comprises a predetermined view in which a motion at an outer portion of the frame is significantly faster than a motion at an inner portion of the frame, and

the elementary scenario comprises the imaging device shooting in a direction that is orthogonal to a horizontal plane while the mobile platform is rotating in a plane that is parallel to the horizontal plane.

7. The method of claim 5, wherein:

the associating comprises associating a side moving view with the elementary scenario,

the side moving view comprises a predetermined view in which all objects of the frame are moving along curve paths, and

the elementary scenario comprises the imaging device shooting in a direction parallel to a horizontal plane while the mobile platform or the imaging device is rotating in a plane parallel to the horizontal plane.

8. The method of claim 5, wherein:

the associating comprises associating a zooming-in view with the elementary scenario,

the zooming-in view comprises a predetermined view in which points away from a center of the frame are moving faster than points in the center and getting out of the frame, and

the elementary scenario comprises the imaging device shooting in a direction that is orthogonal to a horizontal plane while zooming in or while the mobile platform is moving toward a ground.

9. The method of claim 5, wherein:

the associating comprises associating a zooming-out view with the elementary scenario,

the zooming-out view comprises a predetermined view in which points away from a center of the frame are moving faster than points in the center and getting into the frame, and

the elementary scenario comprises the imaging device shooting in a direction that is orthogonal to a horizontal plane while zooming out or while the mobile platform is moving away from a ground.

10. The method of claim 5, wherein:

the associating comprises associating a straight moving view with the elementary scenario,

the straight moving view comprises a predetermined view in which the motion is global for all pixels within the frame, the pixels at an upper portion of the frame are moving slower than the pixels of near-by objects at a lower portion of the frame, and

the elementary scenario comprises the imaging device shooting at a shooting direction with a certain angle to a horizontal plane while the mobile platform is moving at a certain speed in the shooting direction.

11. The method of claim 5, wherein:

the associating comprises associating a global motion view with the elementary view,

the global motion view comprises a predetermined view in which the imaging device moves at ultra-fast global constant speed, and

the elementary scenario comprises the mobile platform holding still while the imaging device is rotating in a plane perpendicular to a horizontal plane.

12. The method of claim 5, wherein the associating comprises combining a plurality of elementary scenarios to generate a combined scenario, and combining the corresponding optical flow fields to generate a corresponding combined optical flow field.

13. The method of claim 12, wherein the associating comprises associating the corresponding combined optical flow field with the combined scenario.

14. The method of claim 5, further comprising:

defining a plurality of elementary scenarios through at least one learning process to generate learned scenarios; and

defining corresponding learned optical flow fields.

15. The method of claim 14, wherein the associating comprises associating one of the corresponding learned optical flow fields with one of the learned scenarios.

16. A method for constructing an optical flow field, comprising:

acquiring at least one of a first motion of a mobile platform or a second motion of a imaging device attached to the mobile platform;

obtaining a position of the imaging device relative to the mobile platform; and

constructing a corresponding optical flow field for a frame based upon at least one of the first motion, the second motion, or the position.

17. An apparatus for constructing an optical flow field, comprising a processor configured to:

acquire a motion of a mobile platform carrying an imaging device;

obtain a status of the imaging device relative to the mobile platform; and

construct a corresponding optical flow field for a frame, based upon the motion of mobile platform and the status of the imaging device.

18. The apparatus of claim 17, wherein the processor is further configured to determine at least one of a rotation of the mobile platform, a first movement of the mobile platform in a horizontal direction, or a second movement of the mobile platform in a vertical direction.

19. The apparatus of claim 17, wherein:

the processor is further configured to determine at least one of a motion of the imaging device or a position of the imaging device,

the motion of the imaging device comprises at least one of a zooming of the imaging device, a first rotation of the imaging device in a horizontal direction, or a second rotation of the imaging device in a vertical direction, and

the position comprises at least one of an orientation or a tilt angle of the imaging device.

20. The apparatus of claim 17, wherein the processor is further configured to associate a pre-categorized optical flow field with a classified scenario of at least one of the motion of the mobile platform or the status of the imaging device.