IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD AND PROGRAM, AND IMAGING APPARATUS

• US 20140375856A1
• Filed: 06/18/2014
• Published: 12/25/2014
• Est. Priority Date: 06/25/2013
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Kaneko, Yushi
• Granted Patent: US 9,473,684 B2
• Time in Patent Office: Days
• US Class Current: 348/302
• CPC Class Codes: H04N 13/00 Stereoscopic video systems; Multi-view video systems; Details thereof
• CPC Class Codes: H04N 13/161 Encoding, multiplexing or demultiplexing different image signal components for multi-view video sequence encoding H04N19/597
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/232 using fly-eye lenses, e.g. arrangements of circular lenses
• CPC Class Codes: H04N 19/00 Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
• CPC Class Codes: H04N 19/50 using predictive coding H04N19/61 takes precedence
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding

Abstract

An image processing apparatus has a plurality of optical systems arranged two-dimensionally and a plurality of photoelectric conversion elements which are two-dimensionally arranged for each of the plurality of optical systems and are used to photoelectrically convert an image from each optical system, and inter-frame prediction encodes a plurality of image frames each having a different parallax, which are formed by the plurality of photoelectric conversion elements.

Claims

1. An image processing apparatus comprising:

a plurality of optical systems arranged two-dimensionally;

a plurality of photoelectric conversion elements which are two-dimensionally arranged for each of the plurality of optical systems and are used to photoelectrically convert an image from each of the optical systems; and

an encoding unit configured to inter-frame prediction encode a plurality of image frames each having a different parallax, which are formed by the plurality of photoelectric conversion elements.

2. An apparatus according to claim 1, wherein the encoding unit encodes a predetermined image frame among the plurality of image frames by an intra-frame encoding and encodes the other image frames by an inter-frame prediction encoding, and in the inter-frame prediction encoding, a reference image is determined in accordance with a parallax direction and a motion vector is detected.

3. An apparatus according to claim 1, wherein the encoding unit further encodes the image frames by a prediction encoding based on a prediction in a time direction.

4. An apparatus according to claim 3, wherein the obtained image data is moving image data constructed by a plurality of frames, the plurality of photoelectric conversion elements convert image data of each frame of the moving image data into the plurality of image frames, and the encoding unit encodes the plurality of image frames of each frame of the moving image data by a prediction encoding based on a prediction in a parallax direction and a prediction in a time direction.

5. An apparatus according to claim 4, wherein the encoding unit performs the prediction encoding based on the prediction in the time direction to a predetermined image frame of each frame of the moving image data.

6. An apparatus according to claim 4, wherein the encoding unit performs the prediction encoding based on the prediction in the time direction to all of the image frames of each frame of the moving image data.

7. An apparatus according to claim 1, further comprising a refocus image generation unit configured to generate a refocus image from the plurality of image frames generated by the plurality of photoelectric conversion elements on the basis of an instruction of a user.

8. An apparatus according to claim 7, wherein the refocus image generation unit generates the refocus image by shifting a pixel of each image frame in a direction of the parallax and integrating the shifted pixel.

9. An apparatus according to claim 8, further comprising a decoding unit configured to decode encoding data generated by the encoding unit and generate a plurality of decoded image frames,

wherein the refocus image generation unit generates the refocus image from the plurality of decoded image frames generated by the decoding unit.

10. An apparatus according to claim 9, wherein the refocus image generation unit generates the refocus image by shifting a pixel of each decoded image frame on the basis of information of a motion vector detected in the prediction in the parallax direction by the encoding unit and integrating the shifted pixel.

11. An apparatus according to claim 7, wherein the encoding unit encodes the refocus image generated by the refocus image generation unit.

12. An apparatus according to claim 1, further comprising a multiplex/demultiplex unit configured to generate stream data by multiplexing encoding data generated by the encoding unit and demultiplex the stream data to encoding data.

13. An apparatus according to claim 1, wherein the plurality of image frames are formed by combining a signal from at least one of the plurality of photoelectric conversion elements corresponding to one of the plurality of optical systems and a signal from at least one of the plurality of photoelectric conversion elements corresponding to the other optical systems.

14. An imaging apparatus comprising a recording unit configured to record image data formed by the plurality of photoelectric conversion elements and the image processing apparatus according to claim 1.

15. An apparatus according to claim 14, further comprising a recording/reproducing unit configured to record data generated by the image processing apparatus in a recording medium and reproduce the data recorded in the recording medium, and

wherein the recording/reproducing unit records stream data in the recording medium, demultiplexes the stream data reproduced from the recording medium to encoding data, and decodes the demultiplexed encoding data.

16. An apparatus according to claim 15, further comprising an operation unit configured to receive an operation of a user,

wherein the control unit controls the image processing apparatus and the recording/reproducing unit on the basis of the user'"'"'s operation received by the operation unit.

17. An apparatus according to claim 16, wherein the operation unit receives an operation for designating a focus position where a refocus image is generated or a position of an object, as an instruction of the user.

18. An apparatus according to claim 13, wherein the plurality of optical systems are a plurality of microlenses arranged two-dimensionally on an imaging plane of the imaging element, a predetermined number of the photoelectric conversion elements correspond to each microlens, and the plurality of photoelectric conversion elements generate one of the image frames from image data of the photoelectric conversion elements locating at a same position for the microlens among the predetermined number of the photoelectric conversion elements corresponding to each microlens.

19. An image processing method using an image processing apparatus having a plurality of optical systems arranged two-dimensionally and a plurality of photoelectric conversion elements which are two-dimensionally arranged for each of the plurality of optical systems and are used to photoelectrically convert an image from each of the optical systems, comprising:

an encoding step of inter-frame prediction encoding a plurality of image frames each having a different parallax, which are formed by the plurality of photoelectric conversion elements.

20. A non-transitory computer-readable storage medium storing a program for causing a computer to control an image processing apparatus having a plurality of optical systems arranged two-dimensionally and a plurality of photoelectric conversion elements which are two-dimensionally arranged for each of the plurality of optical systems and are used to photoelectrically convert an image from each of the optical systems,

wherein the program causes the computer to function as an encoding unit configured to inter-frame prediction encode a plurality of image frames each having a different parallax, which are formed by the plurality of photoelectric conversion elements.