Synthetic imaging

• US 10,009,550 B1
• Filed: 12/22/2016
• Issued: 06/26/2018
• Est. Priority Date: 12/22/2016
• Status: Active Grant

• Current Assignee: X Development LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Hunt, Thomas Peter
• Primary Examiner(s): Jerabek, Kelly L
• Application Number:
  US15/388,884
• Time in Patent Office: 551 Days
• Field of Search: 3482181, 348239
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2224 related to virtual studio applications
• CPC Class Codes: H04N 5/265 Mixing

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for synthetic imaging. In one aspect, a method includes receiving from each digital camera respective imaging data, each digital camera having a viewpoint that is different from the viewpoints of each other digital camera and having a field of view that is overlapping with at least one other digital camera; for a synthetic viewpoint that is a viewpoint that is within a geometry defined by the viewpoints of the digital cameras, selecting respective imaging data that each has a field of view that overlaps a field of view of the synthetic viewpoint and generating, from the selected respective imaging data, synthetic imaging data that depicts an image captured from a virtual camera positioned at the synthetic viewpoint.

Claims

1. A system, comprising:

a data processing apparatus;

a non-transitory memory storage in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising;

for a set of two or more digital cameras, receiving from each digital camera in the set respective imaging data from the digital camera, and wherein each digital camera in the set has a viewpoint that is different from the viewpoints of each other digital camera in the set and has a field of view that is overlapping with at least one other digital camera in the set such that the respective imaging data from each digital camera includes overlapping imaging data from at least one other digital camera; and

for each of a plurality of synthetic viewpoints, where each synthetic viewpoint is a viewpoint that is within a geometry defined by the viewpoints of the digital cameras in the set, and each synthetic viewpoint is different from each other synthetic viewpoint;

selecting respective imaging data that each has a field of view that overlaps a field of view of the synthetic viewpoint, comprising;

determining a minimal subset of digital cameras required to generate synthetic imaging data that depicts an image captured from a virtual camera positioned at the synthetic viewpoint, wherein the minimal subset of digital cameras is a proper subset of the set of digital cameras, and

selecting respective imaging data from only digital cameras in the minimal subset of digital cameras; and

generating, from the selected respective imaging data, synthetic imaging data that depicts an image captured from a virtual camera positioned at the synthetic viewpoint.

2. The system of claim 1, wherein generating the synthetic imaging data for a synthetic viewpoint comprises generating synthetic imaging data based on a synthetic zoom level that is different from an actual zoom level of each digital camera.

3. The system of claim 1, wherein the synthetic imaging data is provided to a user device for display by the user device.

4. The system of claim 1, wherein a three-dimensional stereoscopic image is generated from synthetic imaging data depicting two different images captured from two different synthetic viewpoints, respectively.

5. The system of claim 1, wherein the viewpoints for each digital camera in the set of two or more digital cameras defines a line.

6. The system of claim 1, wherein the viewpoints for each digital camera in the set of two or more digital cameras defines a two-dimensional area.

7. The system of claim 1, wherein the viewpoints for each digital camera in the set of two or more digital cameras defines a three-dimensional volume.

8. The system of claim 1, wherein generating the synthetic imaging data comprises generating the synthetic imaging data from portions of the selected respective imaging data from digital cameras that are determined to be unobstructed relative to a subject to be depicted in the synthetic imaging data.

9. The system of claim 1, wherein generating the synthetic imaging data comprises generating the synthetic imaging data from all the selected respective imaging data, including selected respective imaging data from digital cameras that are determined to be obstructed relative to a subject to be depicted in the synthetic imaging data.

10. The system of claim 1, wherein generating the synthetic imaging data further comprises generating an augmented reality virtual object.

11. The system of claim 1, further comprising selecting at least one of the synthetic viewpoints based on a gesture of a subject depicted within the imaging data of at least one of the digital cameras.

12. The system of claim 1, further comprising selecting at least one of the synthetic viewpoints based on a user device selection of a synthetic viewpoint.

13. A system, comprising:

a data processing apparatus;

a non-transitory memory storage in data communication with the data processing apparatus and storing instructions executable by the data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising;

for a set of two or more digital cameras, receiving from each digital camera in the set respective imaging data from the digital camera, and wherein each digital camera in the set has a viewpoint that is different from the viewpoints of each other digital camera in the set and has a field of view that is overlapping with at least one other digital camera in the set such that the respective imaging data from each digital camera includes overlapping imaging data from at least one other digital camera; and

for each of a plurality of synthetic viewpoints, where each synthetic viewpoint is a viewpoint that is within a geometry defined by the viewpoints of the digital cameras in the set, and each synthetic viewpoint is different from each other synthetic viewpoint;

selecting respective imaging data that each has a field of view that overlaps a field of view of the synthetic viewpoint, comprising determining, for each digital camera for which the respective imaging data has been selected, whether the digital camera is obstructed relative to a subject to be depicted in synthetic imaging data that depicts an image captured from a virtual camera positioned at the synthetic viewpoint; and

generating synthetic imaging data that depicts an image captured from a virtual camera positioned at the synthetic viewpoint from only the selected imaging data from digital cameras that are not determined to be obstructed.

14. The system of claim 13, wherein determining whether the digital camera is obstructed relative to a subject to be depicted in the synthetic imaging data comprises determining that the digital camera is obstructed when a similarity measure that measures a similarity of the imaging data of the digital camera to the imaging data of other selected respective imaging data does not meet a similarity threshold.

15. A non-transitory computer storage medium storing instructions executable by a data processing apparatus and that upon such execution cause the data processing apparatus to perform operations comprising:

for a set of two or more digital cameras, receiving from each digital camera in the set respective imaging data from the digital camera, and wherein each digital camera in the set has a viewpoint that is different from the viewpoints of each other digital camera in the set and has a field of view that is overlapping with at least one other digital camera in the set such that the respective imaging data from each digital camera includes overlapping imaging data from at least one other digital camera; and

for each of a plurality of synthetic viewpoints, where each synthetic viewpoint is a viewpoint that is within a geometry defined by the viewpoints of the digital cameras in the set, and each synthetic viewpoint is different from each other synthetic viewpoint;

selecting respective imaging data that each has a field of view that overlaps a field of view of the synthetic viewpoint, comprising;

determining a minimal subset of digital cameras required to generate synthetic imaging data that depicts an image captured from a virtual camera positioned at the synthetic viewpoint, wherein the minimal subset of digital cameras is a proper subset of the set of digital cameras, and

selecting respective imaging data from only digital cameras in the minimal subset of digital cameras; and

generating, from the selected respective imaging data, synthetic imaging data that depicts an image captured from a virtual camera positioned at the synthetic viewpoint.

16. A computer implemented method, comprising:

for a set of two or more digital cameras, receiving from each digital camera in the set respective imaging data from the digital camera, and wherein each digital camera in the set has a viewpoint that is different from the viewpoints of each other digital camera in the set and has a field of view that is overlapping with at least one other digital camera in the set such that the respective imaging data from each digital camera includes overlapping imaging data from at least one other digital camera; and

for each of a plurality of synthetic viewpoints, where each synthetic viewpoint is a viewpoint that is within a geometry defined by the viewpoints of the digital cameras in the set, and each synthetic viewpoint is different from each other synthetic viewpoint;

selecting respective imaging data that each has a field of view that overlaps a field of view of the synthetic viewpoint, comprising;

determining a minimal subset of digital cameras required to generate synthetic imaging data that depicts an image captured from a virtual camera positioned at the synthetic viewpoint, wherein the minimal subset of digital cameras is a proper subset of the set of digital cameras, and

selecting respective imaging data from only digital cameras in the minimal subset of digital cameras; and

generating, from the selected respective imaging data, synthetic imaging data that depicts an image captured from a virtual camera positioned at the synthetic viewpoint.