MULTIPLE CAMERA VIDEO IMAGE STITCHING BY PLACING SEAMS FOR SCENE OBJECTS

• US 20170140791A1
• Filed: 11/12/2015
• Published: 05/18/2017
• Est. Priority Date: 11/12/2015
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: HICKS, RICHMOND, Das, Barnan, BARAN, STANLEY, PYO, SEAN, LEE, KANGHEE
• Granted Patent: US 9,842,624 B2
• Time in Patent Office: Days
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Image stitching is described for multiple camera video by placing seams for objects in the scene. An object of interest is identified within an overlap between two images each from a different adjacent camera of a multiple camera system. An identified object of interest is placed within an identified one of the two images. A seam is placed between the two images so that the object of interest is within the identified image and not within the other of the two images. The two images are joined at the placed seam and the two joined images are rendered as a single image joined at the placed seam.

Claims

1. A method comprising:

identifying an object of interest within an overlap between two images each from a different adjacent camera of a multiple camera system, the overlap being an image region in which an edge of a field of view of one of the two cameras overlaps an adjacent edge of a field of view of the other of the two cameras so that the overlap is in both of the two images;

determining that the identified object of interest is within an identified one of the two images;

placing a seam between the two images so that the object of interest is within the identified image and not within the other of the two images by rotating a camera support that carries the different adjacent cameras of the multiple camera system;

joining the two images at the placed seam; and

rendering the two images as a single image joined at the placed seam.

2. The method of claim 1, wherein placing the seam comprises searching for a seam position and confining the search to the overlap.

3. The method of claim 1, wherein identifying an object of interest comprises identifying one of a face using a face identification module;

text using a text identification module, or a display screen using a display screen detector module.

4. The method of claim 1, wherein identifying an object of interest comprises tracking a location of a previously identified object of interest.

5. The method of claim 1, wherein placing a seam comprises:

drawing a seam at an initial position;

determining whether the drawn seam passes through the object of interest; and

using a boundary of the object of interest to move the seam to avoid the boundary.

6. The method of claim 1, wherein the boundary is used as a constraint in a seam placement process.

7. The method of claim 6, further comprising identifying a second object of interest within the overlap and using a boundary of the second object of interest as a second constraint in the seam placement process.

8. The method of claim 1, wherein determining that the identified object of interest is within an identified one of the two images comprises determining that the identified object of interest is entirely within an identified one of the two images.

9. A method to minimize or reduce the regions of interest upon which a seam falls, the method comprising:

identifying regions of interest that occur near seams between images each from a different adjacent camera of a multiple camera system;

assigning a weight to each identified region of interest;

determining a plurality of costs, the costs being based on a combination of the weight and a displacement of the corresponding seam required move the corresponding seam from the respective identified region of interest;

minimizing a cost function based on the determined costs to select a seam placement;

placing seams between the two images using the minimized cost;

joining the images at the placed seams; and

rendering the images as a single image joined at the placed seams.

10. The method of claim 9, wherein the weights are assigned based on the importance to a viewer of the respective region of interest.

11. The method of claim 9, wherein the weights are assigned based on a category of an object in each respective region of interest.

12. The method of claim 9, wherein the cost function is a gradient descent.

13. The method of claim 9, wherein minimizing the cost function comprises selecting a series of displacement values and determining a cost until the cost converges.

14. The method of claim 9, wherein placing seams comprises rotating a camera support that carries the different adjacent cameras of the multiple camera system.

15. A video conference node comprising;

a system processor;

a communication interface coupled to the system processor; and

a multiple camera system coupled to the system processor, the multiple camera system capturing at least two images each from a different adjacent camera, the images having an overlap being an image region in which an edge of a field of view of one of the two cameras overlaps an adjacent edge of a field of view of the other of the two cameras so that the overlap is in both of the two images, the system processor identifying an object of interest within the overlap, determining that the identified object of interest is within an identified one of the two images, placing a seam between the two images so that the object of interest is within the identified image and not within the other of the two images by rotating a camera support that carries the different adjacent cameras of the multiple camera system, joining the two images at the placed seam, and rendering the two images as a single image joined at the placed seam.

16. The video conference node of claim 15, wherein the system processor comprises a central processing unit and an imaging chip, the imaging chip joining the two images and rendering the two images.

17. The video conference node of claim 15, wherein the system processor places the seam by using a cost function, the cost function including identifying regions of interest that occur near multiple seams between multiple images each from a different adjacent camera of the multiple camera system, assigning a weight to each identified region of interest, and minimizing a cost function based on a combination of the weight and a displacement of a corresponding seam.

18. The video conference node of claim 15, wherein placing the seam comprises:

drawing a seam at an initial position;

determining whether the drawn seam passes through the object of interest; and

using a boundary of the object of interest to move the seam to avoid the boundary.

19. The video conference node of claim 18, wherein the boundary is used as a constraint in a seam placement process, the system processor further identifying a second object of interest within the overlap and using a boundary of the second object of interest as a second constraint in the seam placement process.

20. The video conference node of claim 15, wherein the multiple camera system comprises a plurality of cameras mounted to a central support, equally spaced around the perimeter of the support and directed outwards from the center of the support to provide a panoramic field of view.