Method and/or architecture for motion estimation using integrated information from camera ISP

• US 8,135,068 B1
• Filed: 06/27/2007
• Issued: 03/13/2012
• Est. Priority Date: 07/19/2005
• Status: Active Grant

• Current Assignee: Geo Semiconductor Incorporated
• Original Assignee: Maxim Integrated Products, Inc. (Analog Devices, Inc.)
• Inventors: Cote, Guy, Alvarez, José R.
• Primary Examiner(s): Chai, Longbit
• Application Number:
  US11/769,023
• Time in Patent Office: 1,721 Days
• Field of Search: 375/240.16
• US Class Current: 375/240.16
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression

Abstract

A camera comprising a first circuit and a second circuit. The first circuit may be configured to perform image signal processing using encoding related information. The second circuit may be configured to encode image data using image signal processing related information. The first circuit may be further configured to pass the image signal processing related information to the second circuit. The second circuit may be further configured to pass the encoding related information to the first circuit. The second circuit may be further configured to modify one or more motion estimation processes based upon the information from the first circuit.

Claims

1. A camera comprising:

a first circuit configured to receive an input signal and perform image signal processing on said input signal using camera setting information and encoding related information;

a second circuit configured to encode image data using image signal processing related information and said camera setting information, the second circuit comprising a still picture encoder-decoder block and a video encoder-decoder block for processing still pictures and picture sequences, respectively; and

wherein (i) said first circuit is further configured to pass said image data and said image signal processing related information to said second circuit, and (ii) said second circuit is further configured to pass said encoding related information to said first circuit and modify one or more motion estimation processes based upon said image signal processing related information from said first circuit.

2. The camera according to claim 1, wherein said first and said second circuits are integrated in a single integrated circuit.

3. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes to favor small motion vectors as long as distortion is below a predetermined threshold when a zoom process is active.

4. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes to favor co-located motion prediction when a zoom process is active.

5. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes to favor small blocks in areas containing edges for medium zoom speeds.

6. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes to favor intra coding modes to reduce distortion for zoom speeds ranging from medium to high.

7. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes to enable reference picture resampling with a direction of the resampling being determined by a direction of zoom.

8. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes to enable reference picture resampling with a resampling ratio being determined by a zoom ratio.

9. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes to favor small motion vectors and intra blocks at edges of a picture while zooming out.

10. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes to favor multiple reference pictures when zoom is varied between zoom-in and zoom-out.

11. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes to favor co-located motion prediction and small motion vectors when an automatic exposure process is active.

12. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes to favor a weighted prediction mode when a substantially linear luminance change is detected.

13. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes to control weighted prediction factors based on exposure times when an auto-exposure process is active.

14. The camera according to claim 13, wherein weighted prediction factors comprise one or more factors selected from the group consisting of offset factors and multiplicative factors.

15. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes such that when small global motion changes are reported in coding statistics the efficiency of Long Term Picture coding is evaluated and pictures designated as Long Term are used for prediction instead of a current picture when global changes between pictures are less than a predetermined percentage of total brightness.

16. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes such that when small global motion changes are reported in coding statistics and automatic exposure is active either a current picture is skipped or skip mode is favored.

17. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes such that small motion co-located motion vectors are emphasized when an automatic focus process is active.

18. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes to favor Skip or Long Term Picture prediction modes when global motion is low and an automatic focus process is in a hunting mode.

19. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes to favor multiple reference pictures using small search ranges when an automatic focus process of said first circuit is hunting for optimal focus.

20. The camera according to claim 1, wherein said second circuit modifies said one or more motion estimation processes such that (i) when a trend in average picture brightness from one picture to another picture exceeds a predetermined threshold the motion estimation processes operate as though automatic exposure is active and (ii) when a sudden increase in brightness or over-exposure is detected the motion estimation processes use either a Long Term Picture prediction mode, zero motion vectors or insertion of an intra frame.

23. The method according to claim 20, wherein:

when zoom is active said one or more motion estimation processes are modified to favor one or more criteria selected from the group consisting of (i) small motion vectors with distortion below a predetermined threshold, (ii) co-located motion vectors, (iii) small blocks in areas containing edges for medium zoom speeds, (iv) intra coding modes to reduce distortion for zoom speeds ranging from medium to high, (v) reference picture resampling with a resampling direction determined by a direction of zoom, (vi) reference picture resampling with a resampling ratio determined by a ratio of zoom, (vii) small motion vectors and intra blocks at edges of a picture while zooming out and (viii) multiple reference pictures when zoom is varied between zooming-in and zooming-out;

when automatic exposure is active said one or more motion estimation processes are modified to favor one or more criteria selected from the group consisting of (i) co-located motion prediction and small motion vectors, (ii) weighted prediction in response to detection of a substantially linear luminance change, (iii) controlling weighted prediction factors based on exposure time, (iv) controlling offset and multiplicative factors based on exposure time, (v) skipping a current picture when small global motion changes are reported in coding statistics, (vi) favoring skip mode when small global motion changes are reported in coding statistics and (vii) using a Long Term Picture for prediction instead of a current picture when global changes between pictures are less than a predetermined percentage of total brightness; and

when automatic focus is active said one or more motion estimation processes are modified to favor one or more criteria selected from the group consisting of (i) small motion co-located motion vectors, (ii) Skip or Long Term Picture prediction when global motion is low and the automatic focus is in a hunting mode and (iii) multiple reference pictures using small search ranges when the automatic focus is hunting for optimal focus.

21. A camera comprising:

means for receiving an input signal and performing image signal processing on said input signal using encoding related information and camera settings information;

means for performing image encoding using image signal processing related information and said camera settings information, and said camera setting information, the second circuit comprising a still picture encoder-decoder block and a video encoder-decoder block for processing still pictures and picture sequences, respectively; and

wherein (i) the image signal processing means is further configured to pass the image signal processing related information to the image encoding means, and (ii) the image encoding means is further configured to pass the encoding related information to the image signal processing means and modify one or more motion estimation processes based upon the information from the image signal processing means.

22. A method for control of motion estimation comprising the steps of:

performing an image signal processing operation using camera settings information and encoding related information within a camera; and

performing an image encoding operation using image signal processing related information and said camera settings information within the camera, and said camera setting information, the second circuit comprising a still picture encoder-decoder block and a video encoder-decoder block for processing still pictures and picture sequences, respectively; and

wherein (i) the image signal processing operation is further configured to pass the image signal processing related information to the image encoding operation, and (ii) the image encoding operation is further configured to pass the encoding related information to the image signal processing operation and modify one or more motion estimation processes based upon the image signal processing related information.