Method and apparatus for tightly coupled, low power image processing

• US 9,001,267 B2
• Filed: 11/11/2011
• Issued: 04/07/2015
• Est. Priority Date: 11/11/2011
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Shen, Jian, Chua-Eoan, Lew G., Hsiao, Yi-Pin
• Primary Examiner(s): SELBY, GEVELL V
• Application Number:
  US13/294,681
• Publication Number:
  US 20130120593A1
• Time in Patent Office: 1,243 Days
• Field of Search: 348/222.1, 348/374
• US Class Current: 348/374
• CPC Class Codes: G06F 13/00 Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units interface circuits for specific input/output devices G06F3/00 ; multiprogram control therefor G06F9/46; multiprocessor systems G06F15/16
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: H04N 1/2112 using still video cameras
• CPC Class Codes: H04N 19/423 characterised by memory arrangements H04N19/433 takes precedence
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 25/443 by reading pixels from selected 2D regions of the array, e.g. for windowing or digital zooming
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure

Abstract

An image divided into N pixel blocks, stored block wise in a camera core and transferred block wise from the camera core to a downstream processing engine local to the local memory. A direct handshaking is communicated, between the camera core and the downstream processing engine, in the block wise transfers. Optionally an optical sensor scanner divides the image with a scan rate N times a frame rate, each scan providing a block of the frame. Optionally, the block wise transfer includes a transfer through a local memory, local to the camera core, controlled by the direct handshaking.

Claims

1. A method of processing an image, comprising:

receiving the image at a camera core; and

transferring the image on a block basis from the camera core to a downstream processing engine, wherein the transferring includes

sending the image from the camera core over a local interconnect to a local memory coupled to the local interconnect,

transferring the image from the local memory, over the local interconnect, to the downstream processing engine, and

in association with transferring the image from the local memory, over the local interconnect, to the downstream processing engine, communicating a handshaking signal between the camera core and the downstream processing engine,

wherein the communicating a handshaking signal between the camera core and the downstream processing engine is over a direct handshaking path outside of the local interconnect and coupled to the camera core and to the downstream processing engine.

2. The method of claim 1, wherein the transferring transfers the image as N image blocks, wherein N is at least two, over a local interconnect coupled to the camera core and to the downstream processing engine.

3. The method of claim 1, further comprising an image processing at the downstream processing engine according to a given order, wherein the transferring transfers the image as a sequence of image blocks according to the given order.

4. The method of claim 1, wherein the downstream processing engine is video coder/decoder (VCODEC), graphics processing unit (GPU) or display.

5. The method of claim 1, wherein the transferring the image over the local interconnect to the downstream processing engine includes transferring the image in blocks.

6. The method of claim 1, wherein the transferring the image over the local interconnect from the local memory to the downstream processing engine over the local interconnect includes transforming the image in N blocks, wherein N is at least two.

7. The method of claim 1, wherein communicating the handshake signals between the camera core and the downstream processing engine includes communicating handshake signals indicating a data availability.

8. The method of claim 1, wherein communicating the handshake signals between the camera core and the downstream processing engine is based at least in part on preventing an overflow in a buffer in the camera core.

9. The method of claim 8, wherein communicating the handshake signals between the camera core and the downstream processing engine is based at least in part on preventing an overflow in a buffer in the local memory.

10. The method of claim 1, wherein receiving the image at the camera core comprises:

generating the image at an optical sensor;

dividing the image into N image blocks, wherein N is at least two;

transferring the N image blocks sequentially to the camera core; and

concurrently storing a number of the N image blocks in the camera core.

11. The method of claim 10, wherein N is in the range of 2-20.

12. The method of claim 10 wherein the number of the image blocks in the camera core is less than N.

13. The method of claim 1, wherein receiving the image at the camera core comprises:

generating the image at an optical sensor;

dividing the image into a plurality of N image blocks, wherein N is at least two;

transferring the image blocks sequentially from the optical sensor to the camera core; and

concurrently storing a quantity of the N image blocks in the camera core.

14. The method of claim 1, wherein receiving the image at the camera core comprises:

scanning the image N times at an optical sensor to generate N scans, wherein N is at least two;

transferring, for each of the N scans, at least one image block of the image to the camera core; and

concurrently storing a quantity of the N image blocks in the camera core.

15. The method of claim 1, wherein sending the image from the camera core to the local memory includes storing at least a portion of the image in a latency buffer associated with the local memory, wherein the method further comprises

detecting a fill state of the latency buffer associated with the local memory;

and generating the handshaking signal based, at least in part, on the detected fill state.

16. A method of transferring an image to a processing engine, comprising:

scanning an optical sensor to obtain one scan of the image;

extracting an image block of a given N-block division of the one scan of the image, wherein N is at least two;

storing the extracted image block in a camera core;

transferring the extracted image block from the camera core, over a local interconnect to a local memory connected to the local interconnect;

transferring the extracted image block from the local memory, over the local interconnect, to a downstream processing engine connected to the local interconnect; and

repeating, N times, the scanning, the extracting, the storing, and the transferring to provide the image to the downstream processing engine,

wherein the transferring the extracted image block from the local memory to the downstream processing engine includes communicating a handshaking signal directly between the camera core and the downstream processing engine, over a direct handshaking path outside of the local interconnect and coupled to the camera core and to the downstream processing engine.

17. The method of claim 16, wherein transferring the extracted image from the camera core to the local memory includes storing at least a portion of the image in a latency buffer associated with the local memory, wherein the method further comprises

detecting a fill state of the latency buffer associated with the local memory;

and generating the handshaking signal for communicating to the downstream processing engine based, at least in part, on the detected fill state.

18. A method of transferring an image to a processing engine, comprising:

scanning an optical sensor to obtain one scan of the image;

extracting an image block of a given N-block division of the one scan of the image;

storing the extracted image block in at least one of a camera core or a local memory associated with the camera core;

transferring the extracted image block from at least one of the camera core or the local memory, to a downstream processing engine; and

repeating, N times, the scanning, the extracting, the storing, and the transferring to provide the image to the downstream processing engine,

wherein the image is a video frame, and wherein the method further comprises providing a sequence of the video frames to the processing engine by the scanning being at a scan rate of N times FS and, for each of the video frames, performing the repeating N times of the scanning, the extracting, the storing, and the transferring, at rate to provide the sequence of the video frames at a frame rate of FS.

19. An image processor, comprising:

a local data interconnect;

a camera core coupled to the local data interconnect;

a downstream processor coupled to the local data interconnect;

a local memory coupled to the data interconnect; and

a direct handshaking path coupled to the camera core and to the downstream processor,

wherein the camera core is configured to

receive a video frame,

transfer the video frame, over the local data interconnect to the local memory,

and to control transfer of the video frame from the local memory to the downstream processor by block transfers over the local data interconnect and to control the block transfers by communicating handshaking signals with the downstream processor over the direct handshaking path.

20. The image processor of claim 19, wherein the camera core is configured to receive the video frame as a succession of N scan frames, wherein N is at least two.

21. The image processor of claim 20, wherein the camera core is configured to extract from each of the N scan frames a corresponding one of the N pixel blocks.

22. The image processor of claim 19, wherein the downstream processor is any from among a video coder/decoder (VCODEC), a graphics processing unit (GPU) or a display.

23. The image processor of claim 19, wherein the camera core includes a buffer to store at least a portion of the video frame, wherein the camera core is configured to detect a fill level of the buffer and to control the block transfers to the downstream processor based at least in part on the detected fill level.

24. The image processor of claim 23, wherein the camera core is configured to control the block transfers by generating at least a portion of said direct handshake signals based, at least in part, on the detected fill level.

25. The image processor of claim 23, wherein the camera core is further configured to control of the block transfers based on the detected fill level to prevent an overflow of the buffer.

26. The image processor of claim 19, wherein the image processor is integrated in at least one semiconductor die.

27. The image processor of claim 19, further comprising a device, selected from a group consisting of a set top box, music player, video player, entertainment unit, navigation device, communications device, personal digital assistant (PDA), fixed location data unit, and a computer, into which the image processor is integrated.

28. The image processor of claim 19, wherein the local memory includes a latency buffer to store at least a portion of the video frame received from the camera core, wherein the camera core is configured to detect a fill level of the buffer and to control the block transfers to the downstream processor based at least in part on the detected fill level.

29. An image processor, comprising:

a local interconnect;

a camera core connected to the local interconnect;

a local memory coupled to the local interconnect;

a direct handshaking path coupled at one end to the camera core, and configured at another end to couple to a downstream processing engine;

means for scanning an optical sensor to obtain one scan of the image,

means for extracting a block of a given N-block division of the one scan of the image, wherein N is at least two;

means for storing an extracted block, resulting from the extracting, in the camera core;

means for transferring the extracted block, through the local interconnect, from the camera core to the local memory; and

means for transferring the extracted block from the local memory, through the local interconnect, to the downstream processing engine, wherein means for transferring the extracted block from the local memory to the downstream processing engine is configured to control the transferring by communicating handshaking signals over the direct handshaking path between the camera core and the downstream processing engine.

30. The image processor of claim 29, wherein the image processor is integrated in at least one semiconductor die.

31. The image processor of claim 29, further comprising a device, selected from a group consisting of a set top box, music player, video player, entertainment unit, navigation device, communications device, personal digital assistant (PDA), fixed location data unit, and a computer, into which the image processor is integrated.

32. A computer program product, having a computer-readable medium comprising:

code for causing at least one computer to receive at a camera core an image; and

code for causing at least one computer to transfer the image on a block basis, to a downstream processing engine, and to communicate, in association with the transfer on a block basis, a handshaking signal between the camera core and the downstream processing engine, over a direct handshaking path coupled to the camera core and to the downstream processing engine,

wherein the code for causing at least one computer to transfer the image on a block basis causes the at least one computer to transfer the image to a local memory over a local interconnect coupled to the camera core and coupled to the local memory and to the downstream processing engine, and to transfer the image from the local memory to the downstream processing engine over the local interconnect,

wherein transfer of the image blocks from the local memory to the downstream processing engine includes communicating direct handshake signals between the camera core and the downstream processing engine over said direct handshaking path coupled to the camera core and to the downstream processing engine,

wherein communicating the direct handshake signals between the camera core and the downstream processing engine is based on preventing an overflow in a buffer associated with the camera core.

33. The computer program product of claim 32, wherein the code for causing at least one computer to receive the image comprises:

code for causing at least one computer to scan the image N times at an optical sensor to generate N scans, wherein N is at least two;

code for causing at least one computer to transfer, for each of the N scans, at least one block of the image to at least one of the camera core and a local memory associated with the camera core; and

code for causing at least one computer to concurrently store a quantity of N of the blocks of the image in at least one of the camera core and the local memory.

34. A computer program product, having a computer-readable medium comprising:

code for causing at least one computer to scan an optical sensor to obtain one scan of an image;

code for causing at least one computer to extract an image block of the one scan of the image, according to a given N-block division, wherein N is at least two;

code for causing at least one computer to store the extracted image block in a camera core;

code for causing at least one computer to transfer the extracted image block from the camera core, over a local interconnect, to a local memory connected to the local interconnect;

code for causing at least one computer to transfer the extracted image block from the local memory, over the local interconnect, to a downstream processing engine connected to the local interconnect; and

code for causing at least one computer to repeat the scan, extract, store, and transfer N times to provide the image to the downstream processing engine,

wherein the code for causing at least one computer to transfer the extracted image block from the local memory to the downstream processing engine includes code for causing at least one computer to communicate a handshaking signal directly between the camera core and the downstream processing engine, over a direct handshaking path outside of the local interconnect and coupled to the camera core and the downstream processing engine.

35. A method of image processing, comprising:

step of receiving an image at a camera core; and

step of transferring the image on a block basis from the camera core to a downstream processing engine, wherein the transferring includes

communicating a handshaking signal between the camera core and the downstream processing engine over a direct handshaking path coupled to the camera core and to the downstream processing engine and, in association with the communicating a handshaking signal,

sending the image from the camera core over a local interconnect to a local memory, and

transferring the image from the local memory, over the local interconnect, to the downstream processing engine.

36. A method of transferring an image to a processing engine, comprising:

step of scanning an optical sensor to obtain one scan of the image,

step of extracting an image block of the one scan of the image according to a given N-block division of the one scan;

step of storing the extracted image block in a camera core;

step of transferring the extracted image block from the camera core, over a local interconnect to a local memory connected to the local interconnect;

step of transferring the extracted image block from the local memory, over the local interconnect, to a downstream processing engine; and

repeating, N times, wherein N is at least two, the step of scanning, the step of extracting, the step of storing, and the step of transferring to provide the image to the downstream processing engine,

wherein the step of transferring the extracted image block from the local memory to the downstream processing engine includes communicating a handshaking signal over a direct handshaking path coupled directly between the camera core and the downstream processing engine.

37. A method of processing an image, comprising:

receiving the image at a camera core; and

transferring the image on a block basis from the camera core to a downstream processing engine, over a local interconnect coupled to the camera core and to the downstream processing engine,

wherein the transferring includes communicating a handshaking signal between the camera core and the downstream processing engine, over a direct handshaking path between the camera core and the downstream processing engine,

wherein receiving the image at the camera core comprises;

generating the image at an optical sensor;

dividing the image into N image blocks, wherein N is at least two;

transferring the N image blocks sequentially to the camera core; and

concurrently storing a number of the N image blocks in the camera core, and

wherein the number of the image blocks in the camera core is less than N.

38. An image processor, comprising:

a camera core;

a downstream processor coupled over a data interconnect to the camera core; and

a direct handshaking path coupled to the camera core and to the downstream processor,

wherein the camera core is configured to

receive a video frame,

and to transfer the video frame by N block transfers over the data interconnect to the downstream processor, each of the block transfers transferring a corresponding one of N pixel blocks of the video frame, and to control the N block transfers over the direct handshaking path,

wherein the camera core includes a buffer to store at least a portion of the video frame, and

wherein the camera core is configured to detect a fill level of the buffer and to control the block transfers to the downstream processor based at least in part on the detected fill level.

39. The image processor of claim 38, wherein said direct handshake signals are based, at least in part, on the detected fill level.

40. The image processor of claim 38, wherein said control of the block transfers based on the detected fill level controls the block transfers to prevent an overflow of the buffer.

41. The image processor of claim 38, wherein the image processor is integrated in at least one semiconductor die.

42. The image processor of claim 38, further comprising a device, selected from a group consisting of a set top box, music player, video player, entertainment unit, navigation device, communications device, personal digital assistant (PDA), fixed location data unit, and a computer, into which the image processor is integrated.

43. The image processor of claim 38, wherein N is at least two.