Method and system for inserting software processing in a hardware image sensor pipeline

• US 9,058,668 B2
• Filed: 11/15/2007
• Issued: 06/16/2015
• Est. Priority Date: 05/24/2007
• Status: Active Grant

• Current Assignee: Avago Technologies International Sales Pte Limited (Broadcom, Inc.)
• Original Assignee: Broadcom Corporation (Broadcom, Inc.)
• Inventors: Plowman, David, Keall, Gary, Walker, Clive
• Primary Examiner(s): Chu, Randolph I
• Assistant Examiner(s): BLOOM, NATHAN J
• Application Number:
  US11/940,788
• Publication Number:
  US 20080292132A1
• Time in Patent Office: 2,770 Days
• Field of Search: 712 10- 22, 708/3, 708/5, 708/100, 708/200, 708230-233, 345501-506, 345/519
• US Class Current: 1/1
• CPC Class Codes: G06F 15/7807 System on chip, i.e. computer system on a single chip; System in package, i.e. computer system on one or more chips in a single package
• CPC Class Codes: G06F 9/3867 using instruction pipelines
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 2201/0096 Portable devices
• CPC Class Codes: H04N 23/76 by influencing the image signals
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof

Abstract

Image data may be processed via one or more stages by a hardware image sensor pipeline (ISP) wherein one or more software processing steps may be inserted at any point within the hardware ISP. Output from any stage of the hardware ISP may be stored in memory. Stored hardware ISP output may be retrieved from memory and processed via one or more software processes. Results from the one or more software processes may be stored in memory and communicated to any stage of the hardware ISP for additional processing. In this regard, the hardware ISP and one or more processors may simultaneously process portions of image data. In addition, the hardware ISP and the one or more processors may be integrated within a chip.

Claims

1. A method for processing images, the method comprising:

performing by a hardware image sensor pipeline (ISP) comprising a plurality of hardware processing stages;

processing image data via one or more steps in a first stage of said plurality of hardware processing stages to generate a first processed image data, the first stage being communicatively coupled to a memory, a second stage of the plurality of hardware processing stages, and a processor, the second stage being subsequent to the first stage;

receiving a signal from the processor indicating whether to store the first processed image data in the memory;

in response to the signal having a first signal status;

outputting said first processed image data from a first output of the first stage of said plurality of hardware processing stages to the memory for access by the processor which performs one or more software processing steps or stages on said first processed image data to generate a second processed image data;

in response to the signal having a second signal status;

outputting said first processed image data from a first output of the first stage of said plurality of hardware processing stages to an input of the second stage without storing the image data in the memory to operate on said first processed image data to generate a second processed image data.

2. The method according to claim 1, comprising storing said first processed image data output from said one or more of said plurality of hardware processing stages of said hardware ISP.

3. The method according to claim 1, wherein said processor controls one or more of:

said processing said image data via said one or more of said plurality of hardware processing stages of said hardware image sensor pipeline;

said outputting of said first processed image data from said output of said one or more of said plurality of hardware processing stages,

said storing of said first processed image data,

said retrieving of said first processed image data by said processor;

said storing of said second processed image data by said processor; and

said receiving of said second processed image data at an input of one or more of said plurality of hardware processing stages of said hardware image sensor pipeline; and

said processing of said second processed image data by said hardware image sensor pipeline.

4. The method according to claim 1, wherein said hardware ISP and said processor are integrated within a chip.

15. The method according to claim 1, further comprising processing said received second processed image data by any of said plurality of hardware processing stages.

18. The method according to claim 1, further comprising outputting a third processed image data from a second output of the second stage to the processor, the second stage communicatively coupled to the memory and the processor.

19. The method according to claim 18, wherein the processor performs one or more software processing steps or stages on said third processed image data to generate a fourth processed image data.

20. The method according to claim 19, further comprising receiving a fourth processed image data at a third input of a third stage of said plurality of hardware processing stages of said hardware image sensor pipeline, the third stage communicatively coupled to the second stage and occurring subsequent to the second stage.

21. The method according to claim 20, wherein the first stage, the second stage, and the third stage simultaneously process portions of the image data.

5. A system for processing images, the system comprising:

one or more circuits comprising a hardware image sensor pipeline (ISP) comprising a plurality of processing stages, said one or more circuits being operable to;

process image data via a first stage of a plurality of processing stages to generate a first processed image data, the first stage being communicatively coupled to a memory, a second stage, and a processor, the second stage being subsequent to the first stage;

receive a first signal from the processor;

in response to the first signal having a first signal status;

receive said first processed image data at an input of the second stage of said plurality of processing stages of said hardware image sensor pipeline from the memory;

in response to the first signal having a second signal status;

receive said first processed image data at an input of the second stage of the plurality of processing stages of the hardware image sensor pipeline from the first stage of the plurality of processing stages of the hardware image sensor pipeline, without storing the image data in the memory.

6. The system according to claim 5, wherein said one or more circuits enables storage of said first processed image data output from one or more of said plurality of processing stages of said hardware ISP.

7. The system according to claim 6, wherein said processor retrieves said stored said first processed image data for handling via said one or more software processing steps or stages.

8. The system according to claim 7, wherein said processor stores said generated second processed image data.

9. The system according to claim 8, wherein said one or more circuits enables receiving as input said stored said generated second processed image data, by any of said one or more of said plurality of processing stages of said hardware ISP for processing.

10. The system according to claim 9, wherein said processor controls one or more of:

said processing said image data via said one or more of said plurality of processing stages of said hardware image sensor pipeline;

said outputting of said first processed image data from said output of said one or more of said plurality of processing stages,

said storing of said first processed image data,

said retrieving of said first processed image data by said processor;

said storing of said second processed image data by said processor; and

said receiving of said second processed image data at an input of one or more of said plurality of processing stages of said hardware image sensor pipeline; and

said processing of said second processed image data by said hardware image sensor pipeline.

16. The system according to claim 5, wherein said one or more circuits are operable to process said received second processed image data by any of said plurality of hardware processing stages.

22. The system according to claim 5, further comprising outputting a third processed image data from a second output of the second stage to the processor, the second stage communicatively coupled to the memory and the processor.

23. The system according to claim 22, wherein the processor comprises plurality of processing cores operable to process a plurality of software processing steps concurrently within the plurality of processing cores, the plurality of processing steps corresponding to the first processed image data and the third processed image data.

24. The system according to claim 5, wherein the one or more circuits are further operable to:

process the first processed image data in the second stage of the plurality of processing stages to generate a second processed image data;

receive a second signal from the processor indicating whether to store the second processed image data in the memory;

in response to the second signal having the first signal status;

outputting the second processed image data from a first output of the second stage of said plurality of processing stages to the memory for access by the processor which performs one or more software processing steps or stages on said second processed image data;

in response to the second signal having the second signal status;

outputting said second processed image data from a first output of the second stage of said plurality of processing stages to a third stage of the plurality of processing stages to operate on said second processed image data.

11. A non-transitory machine-readable storage having stored thereon, a computer program having at least one code section for processing images, the at least one code section being executable by a machine for causing the machine to perform steps comprising:

performing by a hardware image sensor pipeline (ISP) comprising a plurality of hardware processing stages;

processing image data in a tiled format via one or more steps or of said plurality of hardware processing stages to generate a first processed image data;

outputting said first processed image data from an output of one or more of said plurality of hardware processing stages to an input of one or more of said plurality of hardware processing stages of said hardware image sensor pipeline, without storing the image data in a common memory.

12. The non-transitory machine-readable storage according to claim 11, wherein said at least one code section comprises code that enables storing of said first processed image data output from said one or more of said plurality of hardware processing stages of said hardware ISP.

13. The non-transitory machine-readable storage according to claim 12, wherein said processor retrieves said stored said first processed image data for handling via said one or more software processing steps or stages.

14. The non-transitory machine-readable storage according to claim 11, wherein said at least one code section comprises code that enables receiving as input said stored said generated second processed image data, by any of said one or more of said plurality of hardware processing stages of said hardware ISP for processing.

17. The non-transitory machine-readable storage according to claim 11, wherein the steps further comprise processing by said received second processed image data by any of said plurality of hardware processing stages.