Subject tracking systems for a movable imaging system

• US 10,636,150 B2
• Filed: 07/21/2017
• Issued: 04/28/2020
• Est. Priority Date: 07/21/2016
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Omari, Sammy, Enke, Joseph A.
• Primary Examiner(s): Sun, Yulin
• Application Number:
  US15/656,559
• Publication Number:
  US 20180025498A1
• Time in Patent Office: 1,012 Days
• Field of Search: 348144
• CPC Class Codes: B64C 39/024 of the remote controlled vehicle type, i.e. RPV
• CPC Class Codes: B64D 47/08 Arrangements of cameras
• CPC Class Codes: B64U 10/13 Flying platforms
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: B64U 30/20 Rotors; Rotor supports
• CPC Class Codes: G01C 19/00 Gyroscopes; Turn-sensitive devices using vibrating masses; Turn-sensitive devices without moving masses; Measuring angular rate using gyroscopic effects
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 20/13 Satellite images
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A method for tracking a subject in successive image frames includes obtaining previous image frames with an imaging device, processing the previous image frames, obtaining motion information of the imaging device and a subject, determining a region of interest, obtaining a subsequent image frame, and processing the region of interest. The processing includes determining previous frame positions of the subject therein. The motion information is obtained with sensors physically associated with one or more of the imaging device and the subject. The region of interest is located in a predetermined spatial relationship relative to a predicted frame position of the subject.

Claims

1. A method for tracking a subject in successive image frames comprising:

obtaining previous image frames with an imaging device at previous times;

processing the previous image frames to determine previous frame positions of a subject therein;

obtaining motion information of the imaging device;

obtaining motion information of the subject from one or more sensors physically associated with the subject;

determining a region of interest for a subsequent image frame to be obtained at a subsequent time after the previous times, including determining a predicted frame position of the subject in the subsequent image frame from the motion information of the imaging device, the motion information of the subject, and the previous frame positions, and locating the region of interest in a predetermined spatial relationship relative to the predicted frame position;

obtaining the subsequent image frame at the subsequent time; and

processing the region of interest of the subsequent image frame to determine a frame position of the subject.

2. The method of claim 1, wherein the determining the predicted frame position is determined from the previous frame positions, motion information of the imaging device, and motion information of the subject.

3. The method of claim 1, further comprising determining another region of interest for another subsequent image frame to be obtained at another subsequent time after the subsequent time, including determining another predicted frame position of the subject in the other subsequent image frame from at least one of the motion information or a combination of the subsequent frame position and one or more of the previous frame positions;

obtaining the other subsequent image frame at the other subsequent time; and

processing the other region of interest of the other subsequent image frame to determine another frame position of the subject.

4. A method for tracking a subject in successive image frames comprising:

determining a predicted frame location at which a subject is estimated to appear in a subsequent image frame to be obtained at a subsequent time;

determining a region of interest corresponding to the subsequent image frame, including determining a location of the region of interest to be in a predetermined spatial relationship relative to the predicted frame location;

obtaining the subsequent image frame at the subsequent time with an imaging device; and

processing the region of interest of the subsequent image frame to locate the subject,

wherein the predicted frame location is determined according to visual information from previous image frames, motion information of the imaging device, and motion information of the subject, the motion information of the subject being obtained from one or more sensors physically associated with the subject.

5. The method of claim 4, wherein the predetermined spatial relationship is the predicted frame location centered in the region of interest.

6. The method of claim 4, wherein the determining the region of interest includes determining a size of the region of interest.

7. The method of claim 6, wherein the determining the size of the region of interest includes increasing or decreasing the size of the region of interest relative to a previous region of interest corresponding to a previous image frame.

8. The method of claim 6, wherein the size of the region of interest is determined according to at least one of a predicted size of the subject or a predicted distance between the subject and the imaging device in real space.

9. The method of claim 4, wherein the method further comprises obtaining the previous image frames with the imaging device, wherein the predicted frame location is determined according to the visual information derived from the previous image frames, the previous image frames were obtained by the imaging device, and the visual information includes previous frame positions of the subject within the previous image frames.

10. The method of claim 9, wherein predicted frame location is determined according to a visual motion estimate of the subject, the visual motion estimate being determined by applying a motion model to the previous frame positions.

11. The method of claim 4, wherein the predicted frame location is determined according to the motion information of the imaging device obtained from one or more sensors physically associated with the imaging device.

12. The method of claim 11, wherein the motion information of the imaging device includes one or more of position information and orientation information of the imaging device relative to real space.

13. The method of claim 12, wherein predicted frame location is determined according to a imaging device motion estimate of the imaging device, and the one or more sensors include a gyroscope.

14. The method of claim 4, wherein the motion information of the subject includes position information of the subject relative to real space.

15. The method of claim 14, wherein the predicted frame location is determined according to a subject motion estimate of the subject.

16. The method of claim 4, wherein the predicted frame location is determined according to the image data from the previous image frames, the motion information of the imaging device, and the motion information of the subject.

17. The method of claim 16, wherein the predicted frame location is determined according to a visual motion estimate determined from frame positions of the subject in the previous image frames, an imaging device motion estimate determined from the motion information of the imaging device obtained from a sensor physically associated with the imaging device, and a subject motion estimate determined from the motion information of the subject obtained from another sensor physically associated with the subject.

18. A movable imaging system comprising:

a movable platform movable in real space;

an imaging device for capturing successive image frames that form a video and being connected to the movable platform; and

a tracking system for maintaining a subject in the successive image frames, wherein the tracking system locates a region of interest for a subsequent image frame at a predicted frame location of the subject in a future image frame based on previous frame positions of the subject in the successive images, motion information of the imaging device, and motion information of the subject obtained from one or more sensors physically associated with the subject, and thereafter processes the region of interest of the future image frame to locate the subject in the future image frame.