Method and apparatus of image processing

• US 10,455,211 B2
• Filed: 05/25/2017
• Issued: 10/22/2019
• Est. Priority Date: 05/25/2017
• Status: Active Grant

• Current Assignee: Advanced Micro Devices, Inc.
• Original Assignee: Advanced Micro Devices, Inc.
• Inventors: Schmit, Michael L., Giduthuri, Radhakrishna, Nagesh Gowda, Kiriti
• Primary Examiner(s): He, Weiming
• Application Number:
  US15/605,474
• Publication Number:
  US 20180343430A1
• Time in Patent Office: 880 Days
• Field of Search: 345418
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/55 Optical parts specially adapted for electronic image sensors; Mounting thereof
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A method and apparatus of precomputing includes capturing a first image by a first image capturing device. An image space for the first image is defined and pixels in the image space are analyzed for validity. Valid pixels are stored as valid pixel groups and the valid pixel groups are processed.

Claims

1. A method of precomputing that reduces a number of computations required by a graphics pipeline, comprising:

capturing a first image of a plurality of images by a first image capturing device of a plurality of image capturing devices;

capturing a second image of the plurality of images by a second image capturing device of the plurality of image capturing devices;

defining a first image space for the first image that is a subset of the first image;

defining a second image space for the second image that is a subset of the second image;

capturing, by a respective image capturing device of the plurality of image capturing devices, a respective remaining image of each of the remaining plurality of images;

defining a respective remaining image space for each of the respective remaining images that is a subset of that image;

determining first valid pixels by analyzing pixels in the first image space for validity for processing of pixels captured by the first image capturing device;

determining second valid pixels by analyzing pixels in the second image space for validity for processing of pixels captured by the second image capturing device;

determining respective remaining valid pixels by analyzing pixels in each of the remaining respective image spaces for validity for processing of pixels captured by the respective image capturing devices;

storing the first valid pixels, the second valid pixels and the respective remaining valid pixels as a valid pixel group; and

once all valid pixels for all images spaces are determined and stored as the valid pixel group, receiving and processing, in a graphics pipeline, the valid pixel group thereby reducing the number of computations required.

2. The method of claim 1 wherein defining the first and second image spaces includes defining the first and second image spaces as a subset of a total image to be projected into a three-hundred sixty degree space.

3. The method of claim 2 wherein the projection is an equirectangular projection.

4. The method of claim 2 wherein the defined first image space includes image space in a view of the first image capturing device and the defined second image space includes image space in a view of the second image capturing device.

5. The method of claim 2, further comprising generating a mask that identifies the defined image spaces and excludes from processing pixels in areas of the total image outside the defined image spaces for the respective first or second image devices.

6. The method of claim 1 wherein a boundary of the defined first image space is located in an overlap area between the first image and one or more adjacent images, and a boundary of the defined second image space is located in an overlap area between the second image and one or more adjacent images.

7. The method of claim 6 wherein the boundary is defined by one or more seams formed in the overlap areas where the first image or second image and respective adjacent images are combined during a stitching process.

8. The method of claim 6 wherein analyzing pixels in the first or second image spaces for validity includes determining if the pixel is within the boundary for that image space.

9. The method of claim 6, further comprising storing a pixel within the boundary in the valid pixel group.

10. The method of claim 9, further comprising storing a mirror pixel at the edge of the boundary in the valid pixel group.

11. The apparatus of claim 10 wherein a boundary of the defined first image space is located in an overlap area between the first image and one or more adjacent images, and a boundary of the defined second image space is located in an overlap area between the second image and one or more adjacent images.

12. The apparatus of claim 11 wherein the boundary is defined by one or more seams formed in the overlap areas where the first image or second image and respective adjacent images are combined during a stitching process.

13. The apparatus of claim 11 wherein the computing device determines if the pixel is within the boundary and stores pixels within the boundary in the valid pixel group.

14. The method of claim 6 wherein analyzing pixels in the first or second image spaces for validity includes determining pixels at the edge of the boundary and generating a mirror pixel for each pixel at the edge of the boundary.

15. An apparatus that reduces a number of computations required by a graphics pipeline, comprising:

a computing device in communication with all image capturing devices of an image capturing system, wherein the computing device receives a first image of a plurality of images captured by a first image capturing device, receives a second image of the plurality of images captured by a second image capturing device, and a respective remaining image of each of the remaining plurality of images captured by each of the remaining respective image capturing devices, the computing device defines a first image space for the first image that is a subset of the first image, defines a second image space for the second image that is a subset of the second image, defines a respective remaining image space for each of the respective remaining images that is a subset of that image, determines first valid pixels by analyzing pixels in the first image space for validity for processing of pixels captured by the first image capturing device, determines second valid pixels by analyzing pixels in the second image space for validity for processing of pixels captured by the second image capturing device, determines respective remaining valid pixels by analyzing pixels in each of the remaining respective image spaces for validity for processing of pixels captured by the respective image capturing devices, stores the first valid pixels, the second valid pixels, and the respective remaining valid pixels as a valid pixel group, and processes, once all valid pixels for all images spaces are determined and stored as the valid pixel group, in a graphics pipeline of the computing device, the valid pixel group thereby reducing the number of computations required.

16. The apparatus of claim 15 wherein the computing device defines the first and second image spaces as a subset of a total image to be projected into a three-hundred sixty degree space.

17. The apparatus of claim 16 wherein the projection is an equirectangular projection.

18. The apparatus of claim 16 wherein the defined first image space includes image space in a view of the first image capturing device and the defined second image space includes image space in a view of the second image capturing device.

19. The apparatus of claim 16, further comprising the computing device generating a mask that identifies the defined image spaces and excludes from processing pixels in areas of the total image outside the defined image spaces for the respective first or second image devices.

20. A non-transitory computer-readable medium that reduces a number of computations required by a graphics pipeline, the non-transitory computer-readable medium having instructions recorded thereon, that when executed by a computing device, cause the computing device to perform operations including:

receiving a first image of a plurality of images, captured by a first image capturing device of a plurality of image capturing devices;

receiving a second image of the plurality of images, captured by a second image capturing device of the plurality of image capturing devices;

receiving a respective remaining image of each of the remaining plurality of images, captured by a respective image capturing device of the plurality of image capturing devices;

defining a first image space for the first image that is a subset of the first image;

defining a second image space for the second image that is a subset of the second image;

defining a respective remaining image space for each of the respective remaining images that is a subset of that image;

determining first valid pixels by analyzing pixels in the first image space for validity for processing of pixels captured by the first image capturing device;

determining second valid pixels by analyzing pixels in the second image space for validity for processing of pixels captured by the second image capturing device;

determining respective remaining valid pixels by analyzing pixels in each of the remaining respective image spaces for validity for processing of pixels captured by the respective image capturing devices;

storing the first valid pixels, the second valid pixels and the respective remaining valid pixels as a valid pixel group; and

once all valid pixels for all images spaces are determined and stored as the valid pixel group, receiving and processing, in a graphics pipeline, the valid pixel group thereby reducing the number of computations required.