Apparatus and methods for distance estimation using stereo imagery

• US 10,057,593 B2
• Filed: 07/08/2014
• Issued: 08/21/2018
• Est. Priority Date: 07/08/2014
• Status: Active Grant

• Current Assignee: Brain Corporation
• Original Assignee: Brain Corporation
• Inventors: Richert, Micah
• Primary Examiner(s): Dastouri, Mehrdad
• Assistant Examiner(s): Cattungal, Rowina
• Application Number:
  US14/326,374
• Publication Number:
  US 20160014426A1
• Time in Patent Office: 1,505 Days
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/285 using a sequence of stereo image pairs
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation

Abstract

Frame sequences from multiple image sensors may be combined in order to form, for example, an interleaved frame sequence. Individual frames of the combined sequence may be configured a by combination (e.g., concatenation) of frames from one or more source sequences. The interleaved/concatenated frame sequence may be encoded using a motion estimation encoder. Output of the video encoder may be processed (e.g., parsed) in order to extract motion information present in the encoded video. The motion information may be utilized in order to determine a depth of visual scene, such as by using binocular disparity between two or more images by an adaptive controller in order to detect one or more objects salient to a given task. In one variant, depth information is utilized during control and operation of mobile robotic devices.

Claims

1. A non-transitory computer-readable storage medium having computer readable instructions stored thereon, that when executed by at least one processor causes the at least one processor to,

produce a sequence of composite images, each one of a respective sequence of composite images comprising a first image from a first sequence of images and a second image from a second sequence of images, the first image and the second image being joined adjacent to each other;

and evaluate the sequence of composite images to determine a depth parameter of a scene, the evaluating of the sequence of composite images comprising encoding the sequence of composite images into an encoded frame sequence, the encoded frame sequence comprising disparity estimates and motion estimates obtained based on (i) a first set of images from the first sequence of images occurring at one or more time frames and (ii) a second set of images from the second sequence of images occurring at the one or more time frames,

wherein the sequence of composite images includes a first composite image and a second composite image, the first composite image is based on a combination of the second image from the second sequence of images and a plurality of replicas of the first image from the first sequence of images, the second composite image subsequent to the first composite image within the sequence of composite images is based on a combination of a third image from the second sequence of images and a plurality of replicas of a fourth image from the first sequence of images, the third image from the second sequence of images is acquired contemporaneously with the first image from the first sequence of images and subsequent to the second image from the second sequence of images, and the fourth image from the first sequence of images is acquired subsequent to the first image from the first sequence of images.

2. The computer readable storage medium of claim 1, wherein the at least one processor is further configured to execute the computer readable instructions to,

generate the first sequence of images based on input received of the scene by a first image sensor, and

generate the second sequence of images based on input received of the scene by a second image sensor, respectively, the first image sensor is spatially separated from the second image sensor such that a binocular disparity in the first sequence of images and the second sequence of images is achieved, the binocular disparity corresponds to the depth parameter.

3. The computer readable storage medium of claim 2, wherein, the first image from the first sequence of images comprises a first set of plurality of pixels, the second image from the second sequence of images comprises a second set of plurality of pixels, and the at least one processor is further configured to execute the computer readable instructions to encode the sequence of composite images using a motion estimation encoder.

4. The computer readable storage medium of claim 3, wherein, the first composite image of the sequence of composite images comprises the first image from the first sequence of images;

and the second composite image occurring subsequent to the first composite image within the sequence of composite images configured, the second composite image based on a combination of a fifth image from the first sequence of images with a sixth image from the second sequence of images, the first image from the first sequence of images is acquired contemporaneously with the second image from the second sequence of images and prior to the fifth image from the first sequence of images.

5. The computer readable storage medium of claim 3, wherein:

the motion estimation encoder is configured to, based on encoding the first composite image and the second composite image of the sequence of composite images, determine;

a first displacement of at least one pixel in the first set of plurality of pixels within the first image from the first sequence of images relative to the second set of plurality of pixels within the second image from the second sequence of images; and

a second displacement of at least one pixel in the first set of plurality of pixels within the fourth image from the first sequence of images relative to the second set of plurality of pixels within the first image from the first sequence of images;

the first displacement of pixels corresponds to determination of the disparity; and

the second displacement of pixels corresponds to determination of motion associated with one or more objects in the scene.

6. The computer readable storage medium of claim 3, wherein

the first composite image of the sequence of composite images is based on a combination of the first image from the first sequence of images and a fifth transformed image from the first sequence of images, the fifth transformed image includes a reduced pixel count compared to a pixel count in the first image from the first sequence of images.

7. The computer readable storage medium of claim 4, wherein, the motion estimation encoder is configured to determine a first displacement of at least one pixel in the first set of plurality of pixels within the image from the first sequence of images relative to the second set of plurality of pixels within the second image from the second sequence of images;

and the binocular disparity is determined based on the first displacement.

8. The computer readable storage medium of claim 4, wherein, the first set of plurality pixels within the first image from the first sequence of images and the second set of plurality of pixels within the second image from the second sequence of images correspond to at least one object within the scene;

and the at least one processor is further configured to execute the computer readable instructions to determine distance to the object based on the depth parameter.

9. The computer readable storage medium of claim 8,

wherein, the motion estimation encoder is further configured to determine a second displacement of a first pixel in the first set of plurality of pixels within the first image from the first sequence of images and a second pixel in the first set of plurality of pixels within the fourth image from the first sequence of images;

and

the second displacement corresponds to information related to motion of the at least one object.

10. An image processing system, comprising:

an input interface configured to receive a stereo representation of a visual scene, the stereo representation comprising a first portion and a second portion;

a logic component in communication with the input interface and configured to;

arrange the first portion of the stereo representation with the second portion of the stereo representation into a concatenated frame; and

form a sequence of concatenated frames by arranging first portions of the stereo representation and second portions of the stereo representation within a first concatenated frame in an alternate order relative to a preceding concatenated frame within the sequence, the first concatenated frame comprising a different size from either the first portion or the second portion of the stereo representation;

wherein the sequence of concatenated frames further includes a first composite image and a second composite image;

the first composite image of the sequence of concatenated frames is based on a combination of an image from the second sequence of images and a plurality of replicas of an image from the first sequence of images;

the second composite image subsequent to the first composite image within the sequence of concatenated frames, the second composite image is based on a combination of another image from the second sequence of images and a plurality of replicas of another image from the first sequence of images;

the another image from the second sequence of images is acquired contemporaneously with the image from the first sequence of images, and subsequent to the image from the second sequence of images; and

the another image from the first sequence of images is acquired subsequent to the image from the first sequence of images;

a video encoder in data communication with the logic component and configured to encode the sequence of concatenated frames to produce a sequence of compressed frames; and

a processor in data communication with the video encoder and configured to execute computer readable instructions to obtain motion information based on an evaluation of the compressed frames.

11. The system of claim 10, wherein,

the sequence of compressed frames comprises a keyframe characterized by an absence of the motion information;

and the processor is further configured to execute the computer readable instructions to not utilize the keyframe during the evaluation of the compressed frames.

12. The system of claim 10, wherein,

the processor is further configured to execute computer readable instructions to determine, a depth parameter associated with the visual scene based on the motion information;

and the video encoder is configurable to produce the sequence of compressed frames at least at a frame rate selected between 1 and 100 frames per second inclusive; and

the image processing apparatus is embedded in an integrated circuit.

13. The system of claim 10, wherein, the video encoder is further configured to:

estimate a displacement of a block of pixels within the first concatenated frame relative a block of pixels within the preceding frame; and

wherein the motion information based on the displacement.

14. The system of claim 10, wherein,

individual ones of the first and the second portion each comprise a frame of pixels that are characterized by a first number of pixels;

and

the arranging of the first portion with the second portion into the concatenated frame further comprises converting at least one of the first and the second portion into a converted frame that is characterized by a second number of pixels smaller than the first number of pixels.

15. An image processing apparatus system, comprising:

computerized logic configured to;

receive a plurality of stereo representations, each of the plurality being representative of a corresponding visual scene and comprising a first portion and a second portion;

combine a first portion of a given first stereo representation at a first time point with a second portion of the given first stereo representation into a first frame in a first relative arrangement;

combine a first portion of another stereo representation at a second time point with a second portion of the other stereo representation into a second frame in a second relative arrangement different from the first relative arrangement; and

form a sequence comprising at least the first and second frames;

produce a sequence of composite image, the sequence of composite images including a first composite image and a second composite image, wherein, the first composite image of the sequence of composite images is based on a combination of an image from the second sequence of images and a plurality of replicas of an image from the first sequence of images, the second composite image subsequent to the first composite image within the sequence of composite images, the second composite image is based on a combination of another image from the second sequence of images and a plurality of replicas of another image from the first sequence of images, the another image from the second sequence of images is acquired contemporaneously with the image from the first sequence of images, and subsequent to the image from the second sequence of images, and the another image from the first sequence of images is acquired subsequent to the image from the first sequence of images; and

a video encoder in data communication with the computerized logic and configured to encode the sequence of frames to produce a sequence of encoded frames; and

processing logic in data communication with the video encoder and configured to evaluate the sequence of encoded frames to determine motion information.

16. A method of determining motion information within a visual scene, the method comprising:

producing a first composite frame and a second composite frame by combining images from a first plurality of images and a second plurality of images of the visual scene;

producing an interleaved sequence of composite frames comprising the first and the second composite frames; and

evaluating the interleaved sequence of composite frames to determine a stream of encoded frames comprising the motion information, the motion information comprising (i) information associated with a comparison of an image from the first plurality of images and an image from the second plurality of images, (ii) information associated a comparison of two images from the first plurality of images, and (iii) information associated with a comparison of two images from the second plurality of images;

wherein individual images of the first and second pluralities of images are provided by first and second sensing apparatus, respectively, the second sensing apparatus being separated spatially from the first sensing apparatus, and wherein the sequence of composite images includes a first composite image and a second composite image, the first composite image of the sequence of composite images is based on a combination of an image from the second sequence of images and a plurality of replicas of an image from the first sequence of images, the second composite image subsequent to the first composite image within the sequence of composite images, the second composite image is based on a combination of another image from the second sequence of images and a plurality of replicas of another image from the first sequence of images, the another image from the second sequence of images is acquired contemporaneously with the image from the first sequence of images, and subsequent to the image from the second sequence of images, and the another image from the first sequence of images is acquired subsequent to the image from the first sequence of images.

17. The method of claim 16, wherein, the first composite frame is characterized by a first placement configuration of (i) the image from the first plurality of images, and (ii) the image from the second plurality of images;

and the second composite frame is characterized by a second placement configuration of (i) the image from the first plurality of images and (ii) the image from the second plurality of images;

wherein the second placement configuration is different from the first placement configuration.

18. The method of claim 17, further comprising evaluating the motion information to determine a distance to an object disposed within the visual scene;

wherein the evaluating of the interleaved sequence of frames comprises encoding the interleaved sequence of frames using an encoder process comprising a motion estimation process configured to provide data related to motion of a pixel group within the second composite frame relative to a pixel group within the first composite frame.

19. The method of claim 18, wherein, the pixel group within the second composite frame corresponds to a first representation of the object;

the pixel group within the first composite frame corresponds to a second representation of the object; and

the data related to motion are based at least on a displacement of first representation of the object relative to the second representation of the object.

20. The method of claim 18, wherein, the first composite frame comprises a first image from the first plurality of images, and a second image from the second plurality of images acquired contemporaneously with the first image from the first plurality of images;

and the second composite frame comprises the first image from the first plurality of images and a third image of from the first plurality of images, the third image being acquired subsequent to the first image.

21. The method of claim 18, wherein, the first composite frame comprises a first image from the first plurality of images;

the first composite frame further comprises a second image from the first plurality of images and a third image from the second plurality of images;

the second image is acquired subsequent to the first image; and

the third image is acquired contemporaneously with the first image.

22. The method of claim 19, wherein, the evaluating of the motion information comprises determining a binocular disparity between the image from the second plurality of images and the image from the first plurality of images, the binocular disparity being related to the distance of the object and the spatial separation;

and the determination of the distance to the object is based at least on the determining of the binocular disparity.

23. The method of claim 22 wherein, individual images of the first and second pluralities of images comprise a plurality of pixels;

the encoder process is selected from the group consisting of MPEG-4, H.262, H.263 H.264, and H.265 encoders;

the first representation of the object corresponds to a first block of pixels within the image of the first plurality of images;

the second representation of the object corresponds to a second block of pixels within the image of the second plurality of images; and

the displacement is configured to relate a displacement of the first block of pixels relative the second block of pixels.