Panoramic image generation from video

• US 10,582,125 B1
• Filed: 06/01/2015
• Issued: 03/03/2020
• Est. Priority Date: 06/01/2015
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Roessler, Ross David, Townsend, Matthew Alan, Yang, Yinfei, Thomas, Jim Oommen, Poncini, Deon, Welbourne, William Evan, Donaldson, Geoff Hunter, Savastinuk, Paul Aksenti, Kuo, Cheng-Hao
• Primary Examiner(s): Patel, Jayanti K
• Assistant Examiner(s): Gadomski, Stefan
• Application Number:
  US14/727,782
• Time in Patent Office: 1,737 Days
• Field of Search: None
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

A video capture device may include multiple cameras that simultaneously capture video data. The video capture device and/or one or more remote computing resources may stitch the video data captured by the multiple cameras to generate stitched video data that corresponds to 360° video. The remote computing resources may apply one or more algorithms to the stitched video data to identify one or more frames that depict content that is likely to be of interest to a user. The video capture device and/or the remote computing resources may generate one or more images from the one or more frames, and may send the one or more images to the user.

Claims

1. A system comprising:

memory;

one or more processors; and

one or more computer-executable instructions stored in the memory and executable by the one or more processors to perform operations comprising;

receiving feedback data related to a first image captured by a video capture device, the feedback data including an indication of an interest level associated with content depicted in the first image;

receiving, from a video capture device, stitched video data, the stitched video data depicting movement of one or more objects and being generated using a video stitching algorithm that combines;

first video data captured by a first camera of the video capture device,

second video data captured by a second camera of the video capture device,

third video data captured by a third camera of the video capture device, and

fourth video data captured by a fourth camera of the video capture device;

determining, using a first set of algorithms including at least one of an object detection algorithm or a motion detection algorithm, a first frame of the stitched video data, the first frame representing a first scene at a first time, the first scene including a representation of a person determined to be of interest based at least partly on the feedback data;

determining, using at least one algorithm from the first set of algorithms, a second frame of the stitched video data that is subsequent in time relative to the first time, the second frame depicting a second scene representing an event that is determined to be of interest based at least partly on the feedback data;

assigning a first interest value to the first frame, the first interest value being determined by determining an identity of the person and based at least partly on the person being determined to be of interest;

assigning a second interest value to the second frame, the second interest value being determined by identifying an object or an additional person associated with the event and based at least partly on the event being determined to be of interest;

determining that the first interest value is greater than the second interest value;

selecting the first frame;

determining a type of scene associated with the first scene represented in the first frame;

determining, based at least partly on the type of scene, a template to be applied to the first frame;

applying the template to the first frame;

generating, using a second set of algorithms, a second image from the first frame, the second set of algorithms including an edge detection algorithm that determines a first location within the first frame that serves as a first endpoint of the second image and a second location within the first frame that serves as a second endpoint of the second image to define which pixel values from the first frame to include in the second image; and

sending, to a mobile telephone, the second image.

2. The system as recited in claim 1, wherein the operations further comprise:

receiving, from the mobile telephone, second feedback data relating to the second image, the second feedback data including a request to modify at least one of the first endpoint or the second endpoint such that the second image depicts different content; and

using the edge detection algorithm to;

determine a third location within the first frame to serve as a third endpoint for a different image; and

determine a fourth location within the first frame to serve as a fourth endpoint for the different image, the third endpoint and the fourth endpoint to define different pixel values from the first frame to include in the different image; and

sending, to the mobile telephone, the different image.

3. The system as recited in claim 1, wherein the operations further comprise:

determining that the first frame depicts content included in a first field of view associated with the first camera; and

determining that the content is included in a second field of view associated with the second camera, the first camera being associated with a first axis of the video capture device that is approximately 90°

from a second axis associated with the second camera.

4. The system of claim 1, wherein the indication of the interest level associated with the content depicted in the first image includes at least one of a first indication that a user likes the content, a second indication that the user dislikes the content, a third indication of a first request from the user to modify the content, or a fourth indication of a second request from the user for a different image.

5. The system as recited in claim 1, wherein the second set of algorithms include:

a horizon detection algorithm that;

determines first pixels from the first frame that represent a sky of the first scene;

determines second pixels from the first frame that represent a ground of the first scene;

determines a horizon of the first scene depicted in the first frame by determining a border between the first pixels and the second pixels; and

determines that a representation of the horizon in the first frame is uneven across the first frame and the second frame; and

a horizon leveling algorithm that rotates the first frame to level the first frame with the horizon.

6. A method comprising:

receiving feedback data related to a first image associated with a video capture device, the feedback data including an indication of an interest level associated with first content depicted the in first image;

determining stitched video data generated from first video data captured by a first camera of the video capture device and second video data captured by a second camera of the video capture device;

identifying, using at least a first algorithm, a first frame of the stitched video data representing second content determined to be of interest based at least partly on the feedback data;

identifying, using at least one of the first algorithm or a second algorithm, a second frame of the stitched video data representing third content determined to be of interest based at least partly on the feedback data;

determining a first interest value associated with the first frame based at least partly on the second content being determined to be of interest;

determining a second interest value associated with the second frame based at least partly on the third content being determined to be of interest;

determining that the first interest value is greater than the second interest value;

determining a type of scene associated with a scene depicted within the first frame;

determining a template to be applied to the first frame based at least partly on the type of scene; and

generating a second image from the first frame.

7. The method as recited in claim 6, further comprising determining a first location within the first frame that serves as a first endpoint of the image and a second location within the first frame that serves as a second endpoint of the image.

8. The method as recited in claim 7, further comprising:

identifying a person depicted in the first frame; and

determining that the person depicted in the first frame is at a third location between the first location and the second location.

9. The method as recited in claim 6, further comprising:

sending, to a user device associated with a user account, the second image;

receiving, from the user device associated with the user account, second feedback data relating to the second image, the second feedback data including a request, from the user device, to send a different image that depicts the third content represented in the second frame; and

determining, based at least partly on the second feedback data, at least one of a variable or a weight associated with at least one of the first algorithm or the second algorithm.

10. The method as recited in claim 6, wherein the indication of the interest level associated with the content depicted in the first image includes at least one of a first indication that a user likes the content, a second indication that the user dislikes the content, a third indication of a first request from the user to modify the content, or a fourth indication of a second request from the user for a different image.

11. The method as recited in claim 6, further comprising:

determining a curve that includes multiple data points that comprise a first data point and a second data point, the first data point representative of a first interest value assigned to the first frame and the second data point representative of a second interest value assigned to the second frame;

determining that the first data point constitutes a local maxima of the curve; and

selecting the first frame based at least partly on the first data point constituting the local maxima of the curve.

12. The method as recited in claim 6, further comprising determining that the at least the first algorithm includes:

a horizon detection algorithm that;

determines first pixels from the first frame that represent a sky of the second content;

determines second pixels from the first frame that represent a ground of the second content;

determines a horizon associated with the second content by determining a border between the first pixels and the second pixels; and

determines that a representation of the horizon is uneven across the first frame and the second frame; and

a horizon leveling algorithm that levels the horizon by rotating the first frame.

13. The method as recited in claim 6, further comprising:

capturing the second video data using the second camera included in the video capture device;

capturing third video data using a third camera included in the video capture device;

capturing fourth video data using a fourth camera included in the video capture device;

determining the stitched video data based at least in part on the first video data, the second video data, the third video data, and the fourth video data; and

identifying the first frame and the second frame from the stitched video data.

14. A system comprising:

memory;

one or more processors; and

one or more computer-executable instructions stored in the memory and executable by the one or more processors to perform operations comprising;

receiving feedback data related to a first image associated with a video capture device, the feedback data including an indication of an interest level associated with first content depicted the in first image;

determining stitched video data generated from first video data captured by a first camera of the video capture device and second video data captured by a second camera of the video capture device;

identifying, using at least a first algorithm, a first frame of the stitched video data representing second content determined to be of interest based at least partly on the feedback data;

identifying, using at least one of the first algorithm or a second algorithm, a second frame of the stitched video data representing third content determined to be of interest based at least partly on the feedback data;

determining a first interest value associated with the first frame based at least partly on the second content being determined to be of interest;

determining a second interest value associated with the second frame based at least partly on the third content being determined to be of interest;

selecting the first frame based at least partly on a determination that the first interest value is greater than a threshold value;

determining a type of scene associated with a scene depicted within the first frame;

determining a template to be applied to the first frame based at least partly on the type of scene; and

generating, a second image from the first frame.

15. The system as recited in claim 14, wherein the first camera is associated with a first axis of the video capture device that is approximately 90°

from a second axis associated with the second camera, and wherein the operations further comprise;

determining a first field of view associated with the first camera;

determining a second field of view associated with the second camera; and

determining that the first frame depicts an object that is included in the first field of view and the second field of view.

16. The system as recited in claim 14, wherein the operations further comprise:

determining, using a human detection algorithm, that an individual is depicted in the first frame;

determining, using a face detection algorithm, that a face of the individual is oriented in a first direction; and

determining a first location within the first frame that serves as a first endpoint of the second image and a second location within the first frame that services as a second endpoint of the second image such that the individual is depicted in the second image.

17. The system as recited in claim 14, wherein the operations further comprise:

determining a first time stamp associated with the first frame;

identifying a third frame of the stitched video data having a second time stamp that is prior to the first time stamp;

identifying a fourth frame of the stitched video data having a third time stamp that is subsequent to the first time stamp; and

generating the second image based at least partly on at least one of fourth content depicted in the third frame or fifth content depicted in the fourth frame.

18. The system as recited in claim 14, wherein the operations further comprise:

sending, to a user device associated with a user account, the second image;

receiving, from the user device associated with the user account, second feedback data relating to the second image, the second feedback data including a request being received from the user device; and

at least one of;

modifying at least one of a first endpoint or a second endpoint associated with the second image based at least partly on the second feedback data;

or

sending one or more additional images to the user device associated with the user account, the one or more additional images being generated from the second frame.

19. The system as recited in claim 14, wherein the operations further comprise:

determining a curve that includes multiple data points that comprise a first data point and a second data point, the first data point representative of a first interest value assigned to the first frame and the second data point representative of a second interest value assigned to the second frame;

determining that the first data point constitutes a local maxima of the curve; and

selecting the first frame based at least partly on the first data point constituting the local maxima of the curve.

20. The system as recited in claim 14, wherein the indication of the interest level associated with the content depicted in the first image includes at least one of a first indication that a user likes the content, a second indication that the user dislikes the content, a third indication of a first request from the user to modify the content, or a fourth indication of a second request from the user for a different image.