Combined digital and mechanical tracking of a person or object using a single video camera

• US 20070120979A1
• Filed: 11/21/2005
• Published: 05/31/2007
• Est. Priority Date: 11/21/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rui, Yong, Zhang, Cha, He, Li-wei
• Granted Patent: US 8,085,302 B2
• Time in Patent Office: Days
• US Class Current: 348/154
• CPC Class Codes: G08B 13/19667 Details realated to data compression, encryption or encoding, e.g. resolution modes for reducing data volume to lower transmission bandwidth or memory requirements
• CPC Class Codes: H04N 7/185 from a mobile camera, e.g. for remote control
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253

Abstract

A combined digital and mechanical tracking system and process for generating a video using a single digital video camera that tracks a person or object of interest moving in a scene is presented. This generally involves operating the camera at a higher resolution than is needed for the application, and cropping a sub-region out of the image captured that is output as the output video. The person or object being tracked is at least partially contained within the cropped sub-region. As the person or object moves within the field of view of the camera, the location of the cropped sub-region is also moved so as to keep the subject of interest within its boundaries. When the subject of interest moves to the boundary of the FOV of the camera, the camera is mechanically panned to keep the person or object inside its FOV.

Claims

1. A computer-implemented process for generating a video using a single digital video camera that tracks a person or object of interest moving in a scene, comprising using a computer to perform the following process actions:

on a periodic basis,

detecting movement of the person or object being tracked,

digitally tracking the person or object of interest within the last frame captured by the video camera by identifying a cropping region defined as a prescribed-sized sub-region of the last frame captured by the video camera that shows at least part of the person or object of interest whenever the detected motion indicates the person or object being tracked is shown completely within a prescribed-sized portion of the last frame captured by the video camera, and

mechanically tracking the person or object of interest by mechanically panning the video camera in some circumstances where the detected motion indicates the person or object being tracked is not shown completely within the prescribed-sized portion of the last frame captured by the video camera so as to show at least part of the person or object of interest in an identified cropping region of the last frame captured by the video camera after the mechanical panning is complete; and

generating a video that shows the person or object of interest as that person or object moves through the scene by making each consecutive one of said identified cropping regions a consecutive frame of the video.

2. The process of claim 1, wherein detecting movement of the person or object being tracked comprises an action of establishing the location in the last frame captured by the video camera of a motion segment representing the approximate lateral extent of the person or object being tracked.

3. The process of claim 2, wherein the process action of establishing the location of the motion segment in the last frame captured by the video camera, comprises the actions of:

searching for a motion segment using a motion histogram-based detection technique;

whenever a segment is found, designating it as the motion segment for a current time instance; and

whenever no motion segment can be found,

designating the last previously computed motion segment as the motion segment for the current time instance if such a previously computed motion segment exists, and

designating the motion segment for the current time instance to be empty if no previously computed motion segment exists.

4. The process of claim 3, wherein the process action of digitally tracking the person or object of interest within the last frame captured by the video camera, comprises the actions of:

(a) determining if any part of the motion segment for the current time instance comes within a prescribed distance of either one of the lateral side boundaries of the last frame captured by the video camera;

(b) whenever the motion segment for the current time instance does not come within the prescribed distance of either one of the lateral side boundaries of the last frame captured by the video camera, computing a location of a safety segment, said safety segment being a lateral segment contained completely within and set-off from the lateral side boundaries of the cropping region computed for the last previous time instance or in the case of the first time instance, lateral boundaries based on an initial motion segment;

(c) determining if the motion segment for the current time instance is contained completely within the extent of the safety segment;

(d) whenever the motion segment for the current time instance is contained completely within the extent of the safety segment, or is empty, setting the location of a current cropping segment defined as a lateral segment extending between the lateral side boundaries of a cropping region, to the location of the last previously computed cropping segment or in the case of the first time instance, to a location based on the initial motion segment;

(e) whenever the motion segment for the current time instance is not contained completely within the extent of the safety segment, determining if the motion segment is completely outside of the safety segment or if it is partially overlapping the extent of the safety segment;

(f) whenever the motion segment for the current time instance is partially overlapping the extent of the safety segment, determining if the period of time that the overlap condition has existed exceeds a prescribed overlap period;

(g) whenever the period of time that the overlap condition has existed does not exceed the prescribed overlap period, setting the location of a current cropping segment to the location of the last previously computed cropping segment or in the case of the first time instance, to a location based on the initial motion segment, and waiting for the next time instance, and when the next time instance is reached, repeating process action (a) through (f);

(h) whenever the motion segment for the current time instance is completely outside of the safety segment, or the period of time that the overlap condition has existed does exceed the prescribed overlap period, identifying the side of the safety segment that the current motion segment is adjacent to or straddling, and computing the separation distance between the corresponding side of the motion segment and the identified side of the safety segment;

(i) computing the current cropping segment location as the location of the last previously computed cropping segment, or in the case of the first time instance, a location of a cropping segment based on the initial motion segment, shifted in the direction of the identified side of the safety segment based on said separation distance; and

(j) establishing the location of the cropping region using the current cropping segment and a user-prescribed vertical height of the region.

5. The process of claim 4, wherein said safety segment is defined as the lateral segment contained completely within the lateral side boundaries of the cropping region computed for the last previous time instance or in the case of the first time instance, lateral boundaries based on the initial motion segment, but set-in from the lateral side boundaries by a prescribed safety gap.

6. The process of claim 4, wherein the process action of computing the current cropping segment location as the location of the last previously computed cropping segment, or in the case of the first time instance, the location of a cropping segment based on the initial motion segment, shifted in the direction of the identified side of the safety segment based on said separation distance, comprises an action of shifting the previous cropping segment by said separation distance.

7. The process of claim 4, wherein the process action of computing the current cropping segment location as the location of the last previously computed cropping segment, or in the case of the first time instance, the location of a cropping segment based on the initial motion segment, shifted in the direction of the identified side of the safety segment based on said separation distance, comprises the actions of:

computing a moving speed of the cropping region at the current time instance as the lesser of a prescribed maximum velocity, or the velocity computed for the last time instance, increased by the product of a prescribed acceleration and the difference in time between the current and last time instances and given a positive or negative sign to match that of the separation distance; and

shifting the previous cropping segment by a shifting distance, wherein the shifting distance is computed as the moving speed of the cropping region at the current time instance multiplied by the difference in time between the current and last time instances.

8. The process of claim 3, wherein the process action of mechanically tracking the person or object of interest by mechanically panning the video camera, comprises the actions of:

(a) determining if any part of the motion segment for the current time instance comes within the prescribed distance of either one of the lateral side boundaries of the last frame captured by the video camera;

(b) whenever any part of the motion segment for the current time instance falls within the prescribed distance of either one of the lateral side boundaries of the last frame captured by the video camera, determining if a mechanical panning operation has been performed in the past within a prescribed minimum time interval;

(c) whenever a mechanical panning operation has not been performed in the past within the prescribed minimum time interval, computing a mechanical panning distance for the current camera zoom level and mechanically panning the video camera over the computed mechanical panning distance in the direction that will bring the center of the next video frame to be captured by the video camera closer to the person of object being tracked; and

(d) whenever a mechanical panning operation has not been performed in the past within the prescribed minimum time interval, at each time instance prior to reaching the prescribed minimum time interval, the frame of the video being generated is created using the cropping region location associated with the last previous time instance, and when the prescribed minimum time interval is reached, computing a mechanical panning distance for the current camera zoom level and mechanically panning the video camera over the computed mechanical panning distance in the direction that will bring the center of the next video frame to be captured by the video camera closer to the person of object being tracked.

9. The process of claim 4, further comprising process actions performed prior to performing the process action of establishing the location of the cropping region, said process actions comprising:

determining if a secondary area segment, defined as a lateral segment extending between the lateral side boundaries of a secondary region of interest in the scene specified by the user, is completely contained within the last frame captured by the video camera;

whenever the secondary area segment is not completely contained within the last frame captured by the video camera, determining if the last-computed cropping segment location is closer to the secondary area segment than the cropping segment location computed for the last previous time instance;

whenever the last-computed cropping segment location is closer to the secondary area segment than the cropping segment location computed for the last previous time instance, determining if the last-computed motion segment will likely remain within the FOV of the frame of the video captured after a mechanical pan operation is performed; and

whenever the last-computed motion segment will likely remain within the FOV of the frame of the video captured after a mechanical pan operation is performed, computing a mechanical panning distance for the current camera zoom level and mechanically panning the video camera over the computed mechanical panning distance in a direction toward the secondary region of interest.

10. The process of claim 4, further comprising a process action of controlling the zoom level of the video camera on a periodic basis using the amount of movement of the person or object being tracked to decide if the zoom level should be changed and by how much.

11. The process of claim 10, wherein the process action of controlling the zoom level comprises the actions of:

setting the video camera to a prescribed initial zoom level at the beginning of a tracking session; and

at the end of each prescribed-length zoom period during the tracking session,

compute the total horizontal distance that the cropping region moved during the last zoom period,

compute the zoom level that is to be used for the next zoom period based on the total distance computed for the last zoom period, and

zoom the video camera to the computed zoom level if it is different from the level used in the last zoom period.

12. The process of claim 1, further comprising the process actions of:

determining if the movement of the person or object being tracked has not been detected of a prescribed period of time; and

whenever, the movement has not been detected for the prescribed period of time, zoom the video camera out to a prescribed minimum level so as to maximize the field of view.

13. A computer-readable medium having computer-executable instructions for performing the process actions recited in claim 1.

14. A system for generating a video that tracks a person or object of interest moving in a scene, comprising:

a digital video camera disposed so as to view a part of the scene and which is capable of mechanically panning so as to view other parts of the scene;

a general purpose computing device; and

a computer program comprising program modules executable by the computing device, wherein the computing device is directed by the program modules of the computer program to,

detect movement of the person or object being tracked,

produce frames of the video being generated using prescribed-sized sub-regions of frames captured by the video camera wherein each sub-region shows at least part of the person or object of interest, and wherein said sub-region in each video camera frame is identified by tracking the person or object of interest via digital or mechanical panning based on the detected motion, wherein digital panning is used whenever the detected motion indicates the person or object being tracked is shown completely within a prescribed-sized portion a frame captured by the video camera and mechanical panning is used when the detected motion indicates the person or object being tracked is not shown completely within the prescribed-sized portion the frame captured by the video camera.

15. The system of claim 14, wherein each prescribed-sized sub-region is a cropping region have a user-specified height and width, and a fixed user-specified vertical position within the video frames captured by the camera, and wherein the program module for producing frames of the video using digital panning comprises, for each frame produced, sub-modules for:

determining if the detected motion indicates the person or object being tracked is shown completely within, partial within or completely outside a safety region in the last frame captured by the video camera, wherein said safety region is defined as a sub-region in the last-captured video frame corresponding to a sub-region of a previously captured frame used to produce the last previous frame of the video being generated, that has lateral side boundaries which are offset in from the lateral side boundaries of the cropping region associated with said previously captured frame by a prescribed distance;

whenever the detected motion indicates the person or object being tracked is shown completely within a safety region in the last frame captured by the video camera, establishing the location of the cropping region associated with the last-captured frame as the same as the cropping region associated with said previously captured frame;

whenever the detected motion indicates the person or object being tracked is partially within the safety region in the last frame captured by the video camera but has not been for a prescribed period of time, establishing the location of the cropping region associated with the last-captured frame as the same as the cropping region associated with said previously captured frame; and

whenever the detected motion indicates the person or object being tracked is partially within the safety region in the last frame captured by the video camera and has been for the prescribed period of time, or is completely outside the safety region, establishing the location of the cropping region in the last-captured frame by,

identifying the side of the safety region in the last-captured frame that the detected motion indicates the person or object being tracked is adjacent to or straddling, and computing the separation distance between the corresponding side of a lateral segment representing the width of the person or object being tracked as indicated by the detected motion and the identified side of the safety segment, and

establishing the cropping region location as the location of the cropping region established for said previously captured frame shifted in the direction of the identified side of the safety segment by said separation distance.

16. The system of claim 14, wherein each prescribed-sized sub-region is a cropping region have a user-specified height and width, and a fixed user-specified vertical position within the video frames captured by the camera, and wherein the program module for producing frames of the video using digital panning comprises, for each frame produced, sub-modules for:

determining if the detected motion indicates the person or object being tracked is shown completely within, partial within or completely outside a safety region in the last frame captured by the video camera, wherein said safety region is defined as a sub-region in the last-captured video frame corresponding to a sub-region of a previously captured frame used to produce the last previous frame of the video being generated, that has lateral side boundaries which are offset in from the lateral side boundaries of the cropping region associated with said previously captured frame by a prescribed distance;

whenever the detected motion indicates the person or object being tracked is shown completely within a safety region in the last frame captured by the video camera, establishing the location of the cropping region associated with the last-captured frame as the same as the cropping region associated with said previously captured frame;

whenever the detected motion indicates the person or object being tracked is partially within the safety region in the last frame captured by the video camera but has not been for a prescribed period of time, establishing the location of the cropping region associated with the last-captured frame as the same as the cropping region associated with said previously captured frame; and

whenever the detected motion indicates the person or object being tracked is partially within the safety region in the last frame captured by the video camera and has been for the prescribed period of time, or is completely outside the safety region, establishing the location of the cropping region in the last-captured frame by,

identifying the side of the safety region in the last-captured frame that the detected motion indicates the person or object being tracked is adjacent to or straddling, and computing the separation distance between the corresponding side of a lateral segment representing the width of the person or object being tracked as indicated by the detected motion and the identified side of the safety segment, and

establishing the cropping region location as the location of the cropping region established for said previously captured frame shifted in the direction of the identified side of the safety segment by a shifting distance, wherein the shifting distance is defined as a moving speed of the cropping region associated with the last-captured frame multiplied by the difference in time between the capture of last-captured frame and said previously captured frame.

17. The system of claim 16, wherein the moving speed of the cropping region associated with the last-captured frame comprises the lesser of a prescribed maximum velocity, and the moving speed of the cropping region associated with said previously captured frame, increased by the product of a prescribed acceleration and the difference in time between the capture of last-captured frame and said previously captured frame and given a positive or negative sign to match that of said separation distance.

18. The system of claim 14, wherein the program module for producing frames of the video using mechanical panning comprises, for each frame produced, sub-modules for:

determining if a mechanical panning operation has been performed in the past within a prescribed minimum time interval;

whenever a mechanical panning operation has not been performed in the past within the prescribed minimum time interval, computing a mechanical panning distance for the current camera zoom level and mechanically panning the video camera over the computed mechanical panning distance in the direction that will bring the center of the next video frame to be captured by the video camera closer to the person of object being tracked; and

whenever a mechanical panning operation has been performed in the past within the prescribed minimum time interval, waiting until the prescribed minimum time interval has been reached and then computing a mechanical panning distance for the current camera zoom level and mechanically panning the video camera over the computed mechanical panning distance in the direction that will bring the center of the next video frame to be captured by the video camera closer to the person of object being tracked.

19. The system of claim 18, wherein the sub-module for waiting until the prescribed minimum time interval has been reached, comprises, at each time instance when a frame of the video being generated is created prior to reaching the prescribed minimum time interval, creating the frame of the video being generated using the cropping region location associated with the last previous time instance when a frame of the video was generated.

20. The system of claim 14, wherein the program module for producing frames of the video using mechanical panning comprises, for each frame produced, sub-modules for:

computing a mechanical panning distance for the current camera zoom level; and

mechanically panning the video camera over the computed mechanical panning distance in the direction that will bring the center of the next video frame to be captured by the video camera closer to the person of object being tracked.