System and method for utilization of multiple-camera network to capture static and/or motion scenes

• US 10,416,667 B2
• Filed: 02/03/2016
• Issued: 09/17/2019
• Est. Priority Date: 02/03/2016
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Berestov, Alexander, Liu, Ming-Chang, Lee, Chuen-Chien
• Primary Examiner(s): Luo, Kate H
• Application Number:
  US15/014,743
• Publication Number:
  US 20170220037A1
• Time in Patent Office: 1,322 Days
• Field of Search: None
• CPC Class Codes: B64U 2101/30 for imaging, photography or videography
• CPC Class Codes: B64U 2201/102 adapted for flying in formations
• CPC Class Codes: B64U 2201/20 Remote controls
• CPC Class Codes: G05D 1/0027 involving a plurality of vehicles, e.g. fleet or convoy travelling fleet control of land vehicles from a control room G05D1/0297; traffic control systems for road vehicles G08G1/00; for marine craft G08G3/00; for aircraft G08G5/00
• CPC Class Codes: G05D 1/0094 involving pointing a payload, e.g. camera, weapon, sensor, towards a fixed or moving target
• CPC Class Codes: G05D 1/104 involving a plurality of aircrafts, e.g. formation flying traffic control systems for aircraft G08G5/00
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Various aspects of a system and method for utilization of multiple-camera network to capture static and/or motion scenes are disclosed herein. The system comprises a plurality of unmanned aerial vehicles (UAVs). Each of the plurality of UAVs is associated with an imaging device configured to capture a plurality of images. A first UAV of the plurality of UAVs comprises a first imaging device configured to capture a first set of images of one or more static and/or moving objects. The first UAV is configured to receive focal lens information, current location and current orientation from one or more imaging devices. A target location and a target orientation of each of the one or more imaging devices are determined. Control information is communicated to the one or more other UAVs to modify the current location and current orientation to the determined target location and orientation of each of the one or more imaging devices.

Claims

1. A system, comprising:

a plurality of imaging devices; and

a network comprising a plurality of unmanned aerial vehicles (UAVs), wherein

each of said plurality of UAVs is associated with an imaging device of said plurality of imaging devices,

said imaging device is configured to capture a plurality of images, and

a first UAV of said plurality of UAVs comprises;

a first imaging device of said plurality of imaging devices, wherein said first imaging device is configured to capture a first set of images of at least one of at least one static object or at least one moving object; and

at least one circuit configured to;

receive, from at least one imaging device of said plurality of imaging devices, focal lens information, a current location and a current orientation of said at least one imaging device,

wherein said at least one imaging device is associated with at least one second UAV of said plurality of UAVs, and said at least one second UAV is different from said first UAV;

determine a target location and a target orientation of said at least one imaging device based on said received focal lens information, said received current location, said received current orientation, and a current location of said at least one of said at least one static object or said at least one moving object;

determine calibration information based on said received focal lens information and at least one image of a specific calibration object;

communicate control information and said calibration information to said at least one second UAV;

modify said current location and said current orientation of said at least one imaging device to said determined target location and said determined target orientation based on said communicated control information; and

enable calibration of said at least one imaging device of said at least one second UAV based on said communicated calibration information.

2. The system according to claim 1, wherein

said communicated control information further comprises capture information,

said at least one imaging device is configured to capture, based on said capture information, a second set of images of said at least one of said at least one static object or said at least one moving object, and

said second set of images is captured at said determined target location and said determined target orientation.

3. The system according to claim 2, wherein said at least one circuit is further configured to reconstruct a three dimensional (3D) structure from said captured first set of images and said captured second set of images.

4. The system according to claim 3, wherein said at least one circuit is further configured to dynamically synchronize said captured first set of images and said captured second set of images to reconstruct said 3D structure.

5. The system according to claim 1, wherein

said at least one imaging device is configured to capture said at least one image of said specific calibration object.

6. The system according to claim 1, wherein said at least one circuit is further configured to adjust at least one of intrinsic parameters or extrinsic parameters of said at least one imaging device based on said calibration.

7. A server, comprising:

at least one circuit configured to;

receive, from a plurality of imaging devices, focal lens information, a current location and a current orientation of said plurality of imaging devices, wherein

said plurality of imaging devices are associated with a plurality of unmanned aerial vehicles (UAVs),

said plurality of imaging devices capture a plurality of images, and

said plurality of UAVs are communicatively coupled with said server;

determine a target location and a target orientation of said plurality of imaging devices based on said received focal lens information, said received current location, said received current orientation and a current location of at least one of at least one static object or at least one moving object;

determine calibration information based on said received focal lens information and at least one image of a specific calibration object;

communicate control information and said calibration information to said plurality of UAVs;

modify said current location and said current orientation of said plurality of imaging devices to said determined target location and said determined target orientation of said plurality of imaging devices based on said communicated control information; and

enable calibration of said plurality of imaging devices based on said communicated calibration information.

8. The server according to claim 7, wherein

said communicated control information further comprises capture information,

and a set of images of said at least one of said at least one static object or said at least one moving object is captured by said plurality of imaging devices based on said capture information, and

said set of images is captured at said determined target location and said determined target orientation.

9. The server according to claim 8, wherein said at least one circuit is further configured to reconstruct a three dimensional (3D) structure from said captured set of images.

10. The server according to claim 9, wherein said at least one circuit is further configured to dynamically synchronize said captured set of images to reconstruct said 3D structure.

11. The server according to claim 7,

wherein said at least one image of said specific calibration object is captured by said plurality of imaging devices.

12. A system, comprising:

a plurality of imaging devices; and

a first network comprising a first set of unmanned aerial vehicles (UAVs) and a second network comprising a second set of UAVs, wherein

each of said first set of UAVs and said second set of UAVs is associated with an imaging device of said plurality of imaging devices,

said imaging device is configured to capture a plurality of images, and

a first UAV of said first set of UAVs comprises;

a first imaging device of said plurality of imaging devices, wherein said first imaging device is configured to capture a first set of images of at least one of at least one static object or at least one moving object; and

at least one first circuit configured to;

receive, from a first set of imaging devices of said plurality of imaging devices, focal lens information, a current location and a current orientation of said first set of imaging devices,

said first set of imaging devices are associated with said first set of UAVs;

determine a target location and a target orientation of said first set of imaging devices based on said received focal lens information, said received current location, said received current orientation and a current location of said at least one of said at least one static object or said at least one moving object;

determine calibration information based on said received focal lens information and at least one image of a specific calibration object;

communicate control information and said calibration information to said first set of UAVs;

modify said current location and said current orientation of said first set of imaging devices to said determined target location and said determined target orientation of said first set of imaging devices based on said communicated control information; and

enable calibration of said at least one imaging device of said first set of UAVs based on said communicated calibration information.

13. The system according to claim 12, wherein a second UAV of said second set of UAVs comprises:

a second imaging device of said plurality of imaging devices, said second imaging device is configured to capture a second set of images of said at least one of said at least one static object or said at least one moving object; and

at least one second circuit configured to;

receive, from a second set of imaging devices, focal lens information, a current location and a current orientation of said second set of imaging devices,

wherein said second set of imaging devices are associated with said second set of UAVs, and said second set of UAVs is different from said first set of UAVs;

determine a target location and a target orientation of said second set of imaging devices based on said received focal lens information, said received current location, and said received current orientation of said second set of imaging devices and said current location of said at least one of said at least one static object or said at least one moving object;

communicate control information to said second set of UAVs; and

modify said current location and said current orientation of said second set of imaging devices to said determined target location and said determined target orientation of said second set of imaging devices based on said communicated control information.

14. The system according to claim 13, wherein said control information communicated by said first UAV further comprises capture information,

and said first set of imaging devices is configured to capture said first set of images of said at least one of said at least one static object or said at least one moving object based on said capture information, and

said first set of images is captured at said determined target location and said determined target orientation.

15. The system according to claim 14, wherein said control information communicated by said second UAV further comprises capture information, and

said second set of imaging devices is configured to capture a second set of images of said at least one of said at least one static object or said at least one moving object based on said capture information, and

said second set of images is captured at said determined target location and said determined target orientation.

16. The system according to claim 15, wherein said at least one first circuit in said first UAV is further configured to dynamically synchronize said captured first set of images and said captured second set of images to reconstruct a three dimensional (3D) structure.

17. A method, comprising:

in a network comprising a plurality of unmanned aerial vehicles (UAVs),

wherein each of said plurality of UAVs is associated with an imaging device of a plurality of imaging devices,

said imaging device is configured to capture a plurality of images;

capturing a first set of images of at least one of at least one static object or at least one moving object, wherein said first set of images is captured by a first imaging device of said plurality of imaging devices, said first imaging device is associated with a first UAV of said plurality of UAVs;

receiving, by said first UAV, focal lens information, a current location and a current orientation of at least one imaging device, of said plurality of imaging devices, from said at least one imaging device,

wherein said at least one imaging device is associated with at least one second UAV of said plurality of UAVs, and said at least one second UAV is different from said first UAV;

determining, by said first UAV, a target location and a target orientation of said at least one imaging device based on said received focal lens information, said received current location, said received current orientation and a current location of said at least one of said at least one static object or said at least one moving object;

determining, by said first UAV, calibration information based on said received focal lens information and at least one image of a specific calibration object;

communicating, by said first UAV, control information and said calibration information to said at least one second UAV;

modifying said current location and said current orientation of said at least one imaging device to said determined target location and said determined target orientation of said at least one imaging device based on said communicated control information; and

enabling calibration of said at least one imaging device based on said calibration information.

18. The method according to claim 17, wherein said communicated control information further comprises capture information,

said method further comprising capturing, by said at least one imaging device, a second set of images of said at least one of said at least one static object or said at least one moving object based on said capture information, and

said second set of images is captured at said determined target location and said determined target orientation.

19. The method according to claim 18, further comprising reconstructing a three dimensional (3D) structure from said captured first set of images and said captured second set of images.

20. The method according to claim 19, further comprising dynamically synchronizing said captured first set of images and said captured second set of images to reconstruct said 3D structure.

21. The method according to claim 17, further comprising:

capturing, by said at least one imaging device, said at least one image of said specific calibration object.

22. The method according to claim 17, further comprising adjusting at least one of intrinsic parameters or extrinsic parameters of said at least one imaging device based on said calibration.

23. A method, comprising:

in a server communicatively coupled with a plurality of unmanned aerial vehicles (UAVs), wherein said plurality of UAVs is associated with a plurality of imaging devices that are configured to capture a plurality of images;

receiving, by said server, focal lens information, a current location and a current orientation of said plurality of imaging devices from said plurality of imaging devices,

wherein said plurality of imaging devices are associated with said plurality of UAVs;

determining, by said server, a target location and a target orientation of said plurality of imaging devices based on said received focal lens information, said received current location, said received current orientation and a current location of at least one of at least one static object of at least one moving object;

determining, by said server, calibration information based on said received focal lens information and at least one image of a specific calibration object;

communicating control information and said calibration information, by said server, to said plurality of UAVs;

modifying said current location and said current orientation of said plurality of imaging devices to said determined target location and said determined target orientation of said plurality of imaging devices based on said communicated control information; and

enabling calibration of said plurality of imaging devices based on said communicated calibration information.

24. The method according to claim 23, wherein said communicated control information further comprises capture information,

a set of images of said at least one of said at least one static object or said at least one moving object is captured by said plurality of imaging devices based on said capture information, and

said set of images is captured at said determined target location and said determined target orientation.

25. The method according to claim 24, further comprising reconstructing a three dimensional (3D) structure from said captured set of images.

26. The method according to claim 25, further comprising dynamically synchronizing said captured set of images to reconstruct said 3D structure.

27. The method according to claim 23,

wherein said at least one image of said specific calibration object is captured by said at least one imaging device.

28. A method, comprising:

in a first network comprising a first set of unmanned aerial vehicles (UAVs) and a second network comprising a second set of UAVs,

wherein each of said first set of UAVs and said second set of UAVs is associated with an imaging device of a plurality of imaging devices,

said imaging device is configured to capture a plurality of images;

receiving, by a first UAV of said first set of UAVs, focal lens information, a current location and a current orientation of a first set of imaging device, of said plurality of imaging devices, from a first set of imaging devices,

wherein said first set of imaging devices are associated with said first set of UAVs;

determining, by said first UAV, a target location and a target orientation of said first set of imaging devices based on said received focal lens information, said received current location, said received current orientation and a current location of at least one of at least one static object or at least one moving object;

determining, by said first UAV, calibration information based on said received focal lens information and at least an image of a specific calibration object;

communicating, by said first UAV, control information and said calibration information to said first set of UAVs;

modifying said current location and said current orientation of said first set of imaging devices to said determined target location and said determined target orientation of said first set of imaging devices based on said communicated control information; and

enable calibration of said first set of imaging devices based on said communicated calibration information.

29. The method according to claim 28, further comprising:

receiving, by a second UAV of said second set of UAVs, focal lens information, a current location and a current orientation of said second set of imaging devices,

wherein each of said focal lens information, said current location, and said second current orientation is received from a second set of imaging devices, and

said second set of imaging devices are associated with said second set of UAVs, and said second set of UAVs is different from said first set of UAVs;

determining, by said second UAV, a target location and a target orientation of said second set of imaging devices based on said received focal lens information, said received current location, said received current orientation and said current location of said at least one of said at least one static object or said at least one moving object;

communicating control information, by said second UAV, to said second set of UAVs; and

modifying said current location and said current orientation of said second set of imaging devices to said determined target location and said determined target orientation of said second set of imaging devices based on said communicated control information.

30. The method according to claim 29, wherein said control information communicated by said first UAV further comprises capture information,

and said method further comprising capturing, by said first set of imaging devices, a first set of images of said at least one of said at least one static object of said at least one moving object based on said capture information, and

wherein said first set of images is captured at said determined target location and said determined target orientation.

31. The method according to claim 30, wherein said control information communicated by said second UAV further comprises capture information,

and said method further comprising capturing, by said second set of imaging devices, a second set of images of said at least one of said at least one static object or said at least one moving object based on said capture information, and

said second set of images is captured at said determined target location and said determined target orientation.

32. The method according to claim 31, further comprising dynamically synchronizing, by said first UAV, said captured first set of images and said captured second set of images to reconstruct a three dimensional (3D) structure.