Apparatus and methods for real time estimation of differential motion in live video

• US 10,194,163 B2
• Filed: 05/22/2014
• Issued: 01/29/2019
• Est. Priority Date: 05/22/2014
• Status: Active Grant

• Current Assignee: Brain Corporation
• Original Assignee: Brain Corporation
• Inventors: Richert, Micah
• Primary Examiner(s): Kelley, Christopher S
• Assistant Examiner(s): Walsh, Kathleen M
• Application Number:
  US14/285,385
• Publication Number:
  US 20150341633A1
• Time in Patent Office: 1,713 Days
• Field of Search: 37524016, 348169
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 19/50 using predictive coding H04N19/61 takes precedence
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/52 by predictive encoding

Abstract

A data processing apparatus may use a video encoder in order to extract motion information from streaming video in real time. Output of the video encoder may be parsed in order to extract motion information associated with one or more objects within the video stream. Motion information may be utilized by e.g., an adaptive controller in order to detect one or more objects salient to a given task. The controller may be configured to determine a control signal associated with the given task. The control signal determination may be configured based on a characteristic of an object detected using motion information extracted from the encoded output. The control signal may be provided to a robotic device causing the device to execute the task. The use of dedicated hardware video encoder output may reduce energy consumption associated with execution of the task and/or extend autonomy of the robotic device.

Claims

1. A video processing device of a robotic apparatus, comprising:

a camera configured to generate a compressed video frame;

an input interface coupled to the camera and configured to receive the compressed video frame;

a video encoder configured to;

determine information associated with a portion of the compressed frame, the information corresponding to motion of one or more objects within the compressed frame;

evaluate the information for a pattern, the pattern comprising a temporal sequence of a first group of motion vectors and a subsequent second group of motion vectors; and

identify the one or more objects based at least in part on a salient feature in the pattern within the information and based on a determination that a residual motion vector associated with the portion of the compressed frame breaches a velocity threshold, the residual motion vector being derived relative to a prevailing motion vector within the portion of the compressed frame;

an output interface configured to communicate an indication based on the identification of the one or more objects;

wherein;

the one or more objects identified in the compressed frame correspond to a physical task of the robotic apparatus; and

the indication is configured to modify a trajectory of the robotic apparatus based on the identified one or more objects.

2. The device of claim 1, wherein:

the compressed frame is produced by the video encoder comprising a motion estimator configured to provide the motion information;

the compressed frame comprises a plurality of blocks;

the information for a given block is determined by the video encoder based on an evaluation of the block and one other block associated with a preceding compressed frame; and

the information for the given block comprises first and second components.

3. The device of claim 2, wherein:

the preceding compressed frame and the compressed frame comprise representations of a feature;

the indication is configured based on identifying that the pattern is present in the information;

the portion of the compressed frame comprises the given block;

the first and the second components comprise horizontal displacement and vertical displacement of the feature between the preceding compressed frame and the compressed frame.

4. The device of claim 3, wherein:

the video encoder is selected from the group consisting of MPEG-4, H.262, H.263, H.264, and H.265 encoder; and

the video encoder is configured to compress live video in real-time.

5. The device of claim 4, wherein:

the compressed frame comprises a rectangular array of pixels; and

horizontal dimension and vertical dimension of the block are each set by the encoder at 8×

8 pixels or 16×

16 pixels.

6. The device of claim 3, wherein:

the determination of the information comprises;

extraction of a motion vector for individual blocks of the compressed frame, the motion vector characterized by horizontal displacement and vertical displacement components, the horizontal displacement and the vertical displacement components stored as the chromaticity channels; and

transformation of the motion vector into a three channel image frame representation comprised of a luminosity channel, a first chromaticity channel and a second chromaticity channel;

wherein the identification of the pattern comprises access of the three channel image frame representation to obtain the horizontal and the vertical displacement components for one or more individual blocks within the portion.

7. The device of claim 1, wherein:

the pattern is characterized one or more of area, shape, and velocity; and

the indication is configured based on the evaluation indicating an absence of the pattern in the information.

8. The device of claim 1, wherein:

the indication is configured based on a presence of the pattern;

individual block of the plurality of blocks is characterized by a block motion vector;

the motion estimation operation comprises determination of a plurality of motion vectors associated with individual blocks of the plurality of blocks; and

an occurrence of a feature is determined based on an evaluation of one or more motion vector associated with individual blocks within the portion of the plurality of blocks.

9. The device of claim 1, wherein the video encoder is further configured to:

characterize an individual block of the plurality of blocks by a block motion vector; and

determine an occurrence of a feature based on an evaluation of one or more motion vector associated with individual blocks within the portion of the plurality of blocks;

wherein;

the compressed frame is encoded using an encoder comprising a motion estimation operation performed on individual frames of the sequence of frames; and

the motion estimation operation comprises determination of a plurality of motion vectors associated with individual blocks of the plurality of blocks.

10. The device of claim 1, wherein:

the portion of the compressed frame comprises a representation of a person'"'"'s body,

the logic component is further adapted to determine motion information associated with another portion of the compressed frame conveying information amount surroundings of the person; and

the identification of the pattern comprises evaluation of the first motion characteristic associated with the portion and a second motion characteristic associated with the another portion.

11. The device of claim 10, wherein:

the pattern is characterized by a velocity threshold;

the portion comprises a first plurality of blocks characterized by a first plurality of motion vectors;

the another portion comprises a second plurality of blocks characterized by a second plurality of motion vectors;

the first motion characteristic is configured based on an operation on the first plurality of motion vectors;

the second ‘

motion characteristic is configured based on an operation on the second plurality of motion vectors;

the evaluation comprises determination of residual motion associated with the first plurality of blocks configured based on the first motion and the second motion characteristic; and

the pattern identification is configured based on the residual motion breaching the velocity threshold.

12. The device of claim 11, wherein:

the first plurality of motion vectors are configured to convey motion of the body;

the second plurality of motion vectors are configured to convey motion of the surroundings; and

the pattern comprises a gesture by the person.

13. A method of operating a video encoder apparatus, the apparatus configured to compress a raw frame of real-time video into a compressed frame, the method comprising:

causing the encoder apparatus to produce the compressed frame;

parsing the compressed frame, the parsing configured to provide information associated with the compressed frame, the information corresponding to motion of an object in the raw frame, the information comprising a plurality of components;

evaluating the information to determine a presence of the object within the raw frame, the determination of the presence of the object being based on a determination, based on at least one statistical parameter, of at least one residual motion vector derived from at least one dominant motion vector within the compressed frame, the determination of the presence of the object being further based on a determination that the derived at least one residual motion vector breaches a velocity threshold; and

providing an indication of the presence of the object based on the evaluation.

14. The method claim 13, wherein:

the real time video is provided by a video sensor disposed on the robotic device; and

the indication is configured to cause the robotic device to execute an action.

15. The method claim 13, wherein:

the encoder apparatus comprises a hardware component;

the real time video is provided by a video sensor disposed on a robotic device and configured to provide information related to environment of the device; and

the indication is configured to cause the robotic device to execute one of an a object approach or Object avoidance action.

16. The method of claim 15, wherein:

the encoded frame comprises a plurality of blocks, individual ones of the plurality of blocks being characterized by a motion vector;

the evaluation of the information comprises determination of the motion vector; and

the occurrence of a feature is determined based on an evaluation of the motion vector and one other motion vector associated with another block of the plurality of blocks.

17. The method of claim 15, wherein:

the encoded frame comprises a plurality of blocks, individual ones of the plurality of blocks being characterized by a motion vector;

the plurality of blocks comprises a first portion of blocks characterized by a first motion vector, and a second portion characterized by a second motion vector; and

the evaluation of the information comprises a statistical operation configured based on the first motion vector and the second motion vector.

18. The method of claim 15, wherein:

the encoded frame comprises a plurality of blocks, individual ones of the plurality of blocks being characterized by a motion vector;

the plurality of blocks comprises a first portion of blocks characterized by a first motion vector, and a second portion characterized by a second motion vector; and

the evaluation of the information comprises determining a first extent within the frame associated with the first portion, and a second extent within the frame associated with the second portion, using a clustering operation on a plurality of motion vectors associated with the plurality of blocks, the plurality of the motion vectors comprising the first and the second motion vectors.

19. The method of claim 15, wherein:

the robotic device comprises a battery operated vehicle; and

parsing the compressed frame and evaluating the information cooperate to reduce energy used associated with detection of the object compared to object detection configured based on the raw video.

20. The method of claim 15, wherein

the robotic device comprises a computerized appliance comprising a processing component configured to effectuate the parsing the compressed frame and the evaluating the information;

detection of the object based on the information obtained from the compressed frame is configured to reduce computational load of the processing component when compared to object detection configured based on the raw video.

21. The method of claim 18, wherein:

the first motion vector is configured based on self-motion of the robotic device;

the second motion vector is configured based on a motion of the object within the environment;

the first motion vector is greater than the second motion vector; and

the evaluation comprises determining residual motion within the frame based on a removal of motion vectors of the first portion of blocks.

22. A method of processing raw video to efficiently detect objects present therein, the method comprising:

compressing the raw video according to a compression algorithm to produce compressed video;

parsing at least a portion of the compressed video to obtain information corresponding to motion of a moving object, the information comprising a map of pixel groups of the at least the portion of the compressed video, the map of pixel groups comprising a distribution of at least (i) a first plurality of the pixel groups having corresponding first motion vectors and (ii) a second plurality of the pixel groups having corresponding second motion vectors, the first motion vectors and the second motion vectors being different in location and direction; and

evaluating the information associated with the first plurality and the second plurality of pixel groups of the at least the portion of the parsed compressed video to identify the moving object therein, the evaluating the information associated with the first and second plurality of pixel groups to identify the moving object further comprising,

evaluating a respective rate of change of motion within the first plurality and the second plurality of pixel groups, and

determining that at least one residual motion vector within the map of pixel groups breaches a velocity threshold.

23. A non-transitory computer-readable storage medium having computer readable instructions stored thereon, the instructions being executable by at least one processor to cause a video encoder apparatus to:

produce a compressed frame;

parse the compressed frame, the parse being configured to provide information associated with the compressed frame and corresponding to motion of one or more objects in the compressed frame, the information comprising a plurality of components;

evaluate the information to determine a presence of the one or more objects in the compressed frame, the determination of the presence of the one or more objects being based on (i) a determination, based on at least one statistical parameter, of a residual motion vector derived from a dominant vector in the compressed frame, and (ii) a determination that the derived residual motion vector breaches a velocity threshold; and

provide an indication of the presence of the one or more objects based on the evaluation.