Integrated image processor

• US 7,664,315 B2
• Filed: 10/31/2005
• Issued: 02/16/2010
• Est. Priority Date: 11/03/2004
• Status: Expired due to Fees

• Current Assignee: Intel Corporation
• Original Assignee: Tyzx, Inc. (Intel Corporation)
• Inventors: Buck, Ronald John, Brown, Terrence Lee, Jurasek, David Walter, Woodfill, John Iselin, Gordon, Gaile Gibson
• Primary Examiner(s): Carter; Aaron W
• Application Number:
  US11/264,760
• Publication Number:
  US 20060153450A1
• Time in Patent Office: 1,569 Days
• Field of Search: 382/154, 382/285, 382/303, 382/304, 345419-427, 356 12- 14, 348 42- 60, 359462-477
• US Class Current: 382/154
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 10/955 using specific electronic processors
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

An integrated image processor implemented on a substrate is disclosed. An input interface is configured to receive pixel data from two or more images. A pixel handling processor disposed on the substrate is configured to convert the pixel data into depth and intensity pixel data. In some embodiments, a foreground detector processor disposed on the substrate is configured to classify pixels as background or not background. In some embodiments, a projection generator disposed on the substrate is configured to generate a projection in space of the depth and intensity pixel data.

Claims

1. An integrated image processor implemented on a substrate comprising:

an input interface configured to receive pixel data from one or more images; and

a pixel handling processor disposed on the substrate configured to convert the pixel data from the input interface into depth pixel data; and

a foreground detector processor disposed on the substrate configured to classify pixels from the pixel handling processor as background or not background, wherein the output of the foreground detector processor comprises one bit per pixel indicating whether the pixel is classified as background or not background, and wherein a background pixel comprises a pixel that is determined to be relatively stable using a model of the pixel'"'"'s behavior over time, and wherein the pixel handling processor and the foreground detector are arranged in a low latency pipelined architecture such that each are operating on pixel data from the one or more input images at the same time, and wherein the foreground detector processor includes storage for the model of each pixel'"'"'s behavior over time, wherein the pixel handling processor only requires a subset of its input image to produce an output depth pixel data, and wherein the foreground detector only requires a subset of an input range image to classify a pixel as foreground or background.

2. An integrated image processor implemented on a substrate as in claim 1, further including a segmentation processor, wherein the segmentation processor determines a list of zero or more object descriptions, wherein one description of the list comprises a 3D location and a 3D physical extent.

3. An integrated image processor implemented on a substrate as in claim 1, further including an application processor.

4. An integrated image processor implemented on a substrate as in claim 1, further including a projection generator disposed on the substrate configured to generate a projection in space of the depth pixel data received from the pixel handling processor, wherein the projection applies a 3D rotation and translation to the depth data and maps the transformed depth pixel data into a plurality of cells, and wherein one or more statistics for each cell of the plurality of cells are computed based at least in part on the transformed depth pixel data mapped into each cell, and wherein the projection generator also receives the output of the foreground detector and determines, based at least in part on the one bit per pixel output from the foreground detector, whether a depth pixel is not background, and in the event that the depth pixel is not background, the projection generator processes the depth pixel, and in the event that the depth pixel is background, the projection generator does not process the depth pixel, and wherein the pixel handling processor only requires a subset of its input data to produce an output depth pixel data, and wherein the projection generator only requires a subset of its input data to update a first cell in an output array of the projection generator, and wherein the projection generator processor includes storage for the one or more statistics for each cell of the plurality of cells.

5. An integrated image processor implemented on a substrate as in claim 1, wherein the pixel handling processor includes an image grabbing processor.

6. An integrated image processor implemented on a substrate as in claim 1, wherein the pixel handling processor includes a rectify processor.

7. An integrated image processor implemented on a substrate as in claim 1, wherein the pixel handling processor includes a depth processor.

8. An integrated image processor implemented on a substrate as in claim 1, wherein depth comprises disparity.

25. An integrated image processor implemented on a substrate as in claim 1, wherein the pixel handling processor disposed on the substrate is further configured to convert the pixel data from the input interface into intensity pixel data, and wherein the foreground detector takes as input both depth and intensity data from the pixel handling processor.

9. An integrated image processor implemented on a substrate comprising:

an input interface configured to receive pixel data from one or more images;

a pixel handling processor disposed on the substrate configured to convert the pixel data from the input interface into depth pixel data; and

a projection generator disposed on the substrate configured to generate a projection in space of the depth pixel data received from the pixel handling processor, wherein the projection applies a 3D rotation and translation to the depth data and maps the transformed depth pixel data into a plurality of cells, and wherein one or more statistics for each cell of the plurality of cells are computed based at least in part on the depth pixel data mapped into each cell, and wherein the pixel handling processor and the projection generator are arranged in a low latency pipelined architecture such that each are operating on pixel data from the one or more input images at the same time, and wherein the pixel handling processor only requires a subset of its input data to produce an output depth pixel data, and wherein the projection generator only requires a subset of its input data to update a first cell in an output array of the projection generator, and wherein the projection generator processor includes storage for the one or more statistics for each cell of the plurality of cells.

10. An integrated image processor implemented on a substrate as in claim 9, further including a segmentation processor, wherein the segmentation processor determines a list of zero or more object descriptions, wherein one description of the list comprises a 3D location and a 3D physical extent.

11. An integrated image processor implemented on a substrate as in claim 9, further including an application processor.

12. An integrated image processor implemented on a substrate as in claim 9, wherein the pixel handling processor includes an image grabbing processor.

13. An integrated image processor implemented on a substrate as in claim 9 wherein the pixel handling processor includes a rectify processor.

14. An integrated image processor implemented on a substrate as in claim 9, wherein the pixel handling processor includes a depth processor.

15. An integrated image processor implemented on a substrate as in claim 9, wherein depth comprises disparity.

26. An integrated image processor implemented on a substrate as in claim 9, wherein the pixel handling processor disposed on the substrate is further configured to convert the pixel data from the input interface into intensity pixel data, and wherein the projection generator takes as input both depth and intensity data from the pixel handling processor, and wherein the projection generator may include statistics related to intensity for each cell.

16. An integrated image processor implemented on a substrate comprising:

an input interface configured to receive pixel data from one or more images; and

a pixel handling processor disposed on the substrate configured to convert the pixel data from the input interface into depth pixel data; and

a foreground detector processor disposed on the substrate configured to classify depth pixels from the pixel handling processor as background or not background, wherein a background pixel comprises a pixel that is determined to be relatively stable using a model of the pixel'"'"'s behavior over time;

a projection generator disposed on the substrate configured to generate a projection in space of the depth pixel data determined by the pixel handling processor, wherein the projection applies a 3D rotation and translation to the depth pixel data and maps the transformed depth pixel data into a plurality of cells, and wherein one or more statistics for each cell of the plurality of cells are computed based at least in part on the depth pixel data mapped into each cell, and wherein the pixel handling processor, the foreground detector, and the projection generator are arranged in a low latency pipelined architecture such that each are operating on pixel data from the one or more input images at the same time, and wherein the projection generator also receives the output of the foreground detector and determines, based at least in part on the one bit per pixel output from the foreground detector, whether a depth pixel is not background, and in the event that the depth pixel is not background, the projection generator processes the depth pixel, and in the event that the depth pixel is background, the projection generator does not process the depth pixel, and wherein the pixel handling processor only requires a subset of its input data to produce an output depth pixel, and wherein the projection generator only requires a subset of its input data to update a first cell in an output array of the projection generator, and wherein the projection generator processor includes storage for the one or more statistics for each cell of the plurality of cells, and wherein the foreground detector processor includes storage for the model of the pixel'"'"'s behavior over time, and wherein the foreground detector only requires a subset of its input image data to produce an output pixel;

a segmentation processor disposed on the substrate configured to compute from the output of the projection generator a list of zero or more individual object descriptions, wherein one description of the list comprises a 3D location and a 3D physical extent; and

an application processor disposed on the substrate configured to receive the list of zero or more individual object descriptions output by the segmentation processor.

17. An integrated image processor implemented on a substrate as in claim 16, wherein the application processor produces a count of people output.

18. An integrated image processor implemented on a substrate as in claim 16, wherein the application processor produces a count of cars output.

19. An integrated image processor implemented on a substrate as in claim 16, wherein the application processor produces a count of objects output.

20. An integrated image processor implemented on a substrate as in claim 16, wherein the application processor produces a detection of objects output.

21. An integrated image processor implemented on a substrate as in claim 16, wherein the application processor produces a tracking of objects output.

22. An integrated image processor implemented on a substrate as in claim 16, wherein the application processor produces an identification of obstacles output.

23. An integrated image processor implemented on a substrate as in claim 16, wherein the application processor produces a detection of a navigable path output.

24. An integrated image processor implemented on a substrate as in claim 16, wherein the application processor produces an identification of a tailgater output.

27. An integrated image processor implemented on a substrate as in claim 16, wherein the pixel handling processor disposed on the substrate is further configured to convert the pixel data from the input interface into intensity pixel data, and wherein the foreground detector and the projection generator take as input both depth and intensity data from the pixel handling processor, and wherein the projection generator may include statistics related to intensity for each cell.