Three-dimensional reconstruction of scenes and objects

• US 8,228,367 B2
• Filed: 04/09/2009
• Issued: 07/24/2012
• Est. Priority Date: 04/09/2009
• Status: Active Grant

• Current Assignee: Telefonaktiebolaget LM Ericsson
• Original Assignee: Telefonaktiebolaget LM Ericsson
• Inventors: Johansson, Björn, Grafulla-Gonzalez, Beatriz, Xu, Sanbao, Persson, Per
• Primary Examiner(s): Dinh, Khanh
• Application Number:
  US12/420,943
• Publication Number:
  US 20100259594A1
• Time in Patent Office: 1,202 Days
• Field of Search: 348/E5.022, 348/E13.001, 348/43, 345/473
• US Class Current: 348/43
• CPC Class Codes: G01S 13/08 Systems for measuring distance only indirect measurement G01S13/46
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders

Abstract

A radio transceiver included in or attached to an imaging camera is used to facilitate the automatic determination of a scale factor in a three-dimensional (3D) reconstruction. The radio transceiver transmits a radio signal from each of one or more camera positions and measures a reflection profile for the transmitted signal. The measured reflection profile is compared to a simulated reflection profile, generated from a 3D model of the subject/scene of interest, to determine an estimate of an unknown scale for the 3D model. In various embodiments, the radio transceiver may be adapted from a radio device that already exists in or on the imaging camera (e.g., a cellular radio, Bluetooth® radio, or the like) or may be added to an imaging camera.

Claims

1. A method of processing image data, the method comprising:

capturing image data of a subject of interest from each of a plurality of camera positions and constructing a three-dimensional model of the subject of interest based on the captured image data, the three-dimensional model having an unknown scale;

transmitting a radio signal from a first one of the camera positions and measuring a reflection profile for the transmitted radio signal;

generating a simulated reflection profile corresponding to the first one of the camera positions; and

calculating a first estimate of the unknown scale for the three-dimensional model by comparing the measured reflection profile to the simulated reflection profile.

2. The method of claim 1, further comprising transmitting the radio signal from the first one of the camera positions simultaneously with capturing image data from the first one of the camera positions.

3. The method of claim 1, wherein transmitting the radio signal comprises activating a communications transceiver.

4. The method of claim 1, wherein measuring a reflection profile for the transmitted radio signal comprises generating a graph of correlations between the transmitted radio signal and a received reflected radio signal at a plurality of time delays.

5. The method of claim 4, wherein generating a simulated reflection profile comprises calculating round-trip delays corresponding to a plurality of expected reflection paths, based on the three-dimensional model of the subject of interest and a reference scaling metric, and generating a simulated graph of correlations based on the calculated round-trip delays.

6. The method of claim 5, wherein generating a simulated reflection profile further comprises estimating reflection magnitudes corresponding to one or more of the expected reflection paths, based on the three-dimensional model of the subject of interest and the reference scaling metric, and wherein the simulated graph of correlations is based further on the estimated reflection magnitudes.

7. The method of claim 1, wherein calculating a first estimate of the unknown scale for the three-dimensional model comprises determining which of a plurality of trial scaling factors, when applied to the simulated reflection profile to obtain a scaled simulated reflection profile, yields the highest correlation between the scaled simulated reflection profile and the measured reflection profile.

8. The method of claim 1, further comprising:

measuring one or more additional measured reflection profiles corresponding to one or more additional camera positions;

generating one or more additional simulated reflection profiles corresponding to the one or more additional camera positions;

calculating an additional scale estimate for each of the one or more additional camera positions by comparing the additional measured reflection profiles to the corresponding additional simulated reflection profiles; and

computing an average scale estimate from the first estimate of the unknown scale and the additional scale estimates.

9. An image processing apparatus, comprising a digital camera device, a radio transceiver, and processing circuitry configured to:

construct a three-dimensional model of a subject of interest based on image data of the subject of interest captured from each of a plurality of camera positions, the three-dimensional model having an unknown scale;

measure a reflection profile for a radio signal transmitted by the radio transceiver from a first one of the camera positions;

generate a simulated reflection profile corresponding to the first one of the camera positions; and

calculate a first estimate of the unknown scale for the three-dimensional model by comparing the measured reflection profile to the simulated reflection profile.

10. The image processing apparatus of claim 9, wherein the processing circuit is configured to control the radio transceiver to transmit the radio signal simultaneously with the capturing of the image data for the first one of the camera positions.

11. The image processing apparatus of claim 9, wherein the radio transceiver comprises at least one of a cellular communications transceiver, a wireless local-area network transceiver, and a short-range wireless transceiver.

12. The image processing apparatus of claim 9, wherein the processing circuitry is configured to measure the reflection profile for the transmitted radio signal by generating a graph of correlations between the transmitted radio signal and a received reflected radio signal at a plurality of time delays.

13. The image processing apparatus of claim 12, wherein the processing circuitry is configured to generate the simulated reflection profile by calculating round-trip delays corresponding to a plurality of expected reflection paths, based on the three-dimensional model of the subject of interest and a reference scaling metric, and generating a simulated graph of correlations based on the calculated round-trip delays.

14. The image processing apparatus of claim 13, wherein the processing circuitry is configured further to generate the simulated reflection profile by estimating reflection magnitudes corresponding to one or more of the expected reflection paths, based on the three-dimensional model of the subject of interest and the reference scaling metric, and generating the simulated graph of correlations based further on the estimated reflection magnitudes.

15. The image processing apparatus of claim 9, wherein the processing circuitry is configured to calculate the first estimate of the unknown scale for the three-dimensional model by determining which of a plurality of trial scaling factors, when applied to the simulated reflection profile to obtain a scaled simulated reflection profile, yields the highest correlation between the scaled simulated reflection profile and the measured reflection profile.

16. The image processing apparatus of claim 9, wherein the processing circuitry is further configured to:

measure one or more additional measured reflection profiles corresponding to one or more additional camera positions;

generate one or more additional simulated reflection profiles corresponding to the one or more additional camera positions;

calculate an additional scale estimate for each of the one or more additional camera positions by comparing the additional measured reflection profiles to the corresponding additional simulated reflection profiles; and

compute an average scale estimate from the first estimate of the unknown scale and the additional scale estimates.

17. A method of processing image data, the method comprising:

receiving image data of a subject of interest captured from each of a plurality of camera positions and constructing a three-dimensional model of the subject of interest based on the captured image data, the three-dimensional model having an unknown scale;

receiving reflection profile data corresponding to a reflection profile measured by transmitting a radio signal from a first one of the camera positions and measuring a reflected return signal;

generating a simulated reflection profile corresponding to the first one of the camera positions; and

calculating a first estimate of the unknown scale for the three-dimensional model by comparing the reflection profile data to the simulated reflection profile.

18. The method of claim 17, wherein the reflection profile data comprises a graph of correlations between the transmitted radio signal and the reflection return radio signal at a plurality of time delays.

19. The method of claim 18, wherein generating a simulated reflection profile comprises calculating round-trip delays corresponding to a plurality of expected reflection paths, based on the three-dimensional model of the subject of interest and a reference scaling metric, and generating a simulated graph of correlations based on the calculated round-trip delays.

20. The method of claim 19, wherein generating a simulated reflection profile further comprises estimating reflection magnitudes corresponding to one or more of the expected reflection paths, based on the three-dimensional model of the subject of interest and the reference scaling metric, and wherein the simulated graph of correlations is based further on the estimated reflection magnitudes.

21. The method of claim 17, wherein calculating a first estimate of the unknown scale for the three-dimensional model comprises determining which of a plurality of trial scaling factors, when applied to the simulated reflection profile to obtain a scaled simulated reflection profile, yields the highest correlation between the scaled simulated reflection profile and the reflection profile data.

22. The method of claim 17, further comprising:

receiving additional reflection profile data for one or more additional measured reflection profiles corresponding to one or more additional camera positions;

generating one or more additional simulated reflection profiles corresponding to the one or more additional camera positions;

calculating an additional scale estimate for each of the one or more additional camera positions by comparing the additional measured reflection profiles to the corresponding additional simulated reflection profiles; and

computing an average scale estimate from the first estimate of the unknown scale and the additional scale estimates.

23. An image processing apparatus, comprising a communications interface configured for communicating with an image capture apparatus and a processing subsystem configured to:

receive, via the communications interface, image data of a subject of interest captured by the image capture apparatus from each of a plurality of camera positions;

construct a three-dimensional model of the subject of interest based on the captured image data, the three-dimensional model having an unknown scale;

receive, via the communications interface, reflection profile data corresponding to a reflection profile measured by the image capture apparatus by transmitting a radio signal from a first one of the camera positions and measuring a reflected return signal;

generate a simulated reflection profile corresponding to the first one of the camera positions; and

calculate a first estimate of the unknown scale for the three-dimensional model by comparing the reflection profile data to the simulated reflection profile.

24. The image processing apparatus of claim 23, wherein the processing subsystem is configured to generate the simulated reflection profile by calculating round-trip delays corresponding to a plurality of expected reflection paths, based on the three-dimensional model of the subject of interest and a reference scaling metric, and generating a simulated graph of correlations based on the calculated round-trip delays.

25. The image processing apparatus of claim 23, wherein the processing subsystem is further configured to generate the simulated reflection profile by estimating reflection magnitudes corresponding to one or more of the expected reflection paths, based on the three-dimensional model of the subject of interest and the reference scaling metric, and generating the simulated graph of correlations based further on the estimated reflection magnitudes.

26. The image processing apparatus of claim 23, wherein the processing subsystem is configured to calculate the first estimate of the unknown scale for the three-dimensional model by determining which of a plurality of trial scaling factors, when applied to the simulated reflection profile to obtain a scaled simulated reflection profile, yields the highest correlation between the scaled simulated reflection profile and the reflection profile data.

27. The image processing apparatus of claim 23, wherein the processing subsystem is further configured to:

receive, via the communications interface, additional reflection profile data for one or more additional measured reflection profiles corresponding to one or more additional camera positions;

generate one or more additional simulated reflection profiles corresponding to the one or more additional camera positions;

calculate an additional scale estimate for each of the one or more additional camera positions by comparing the additional measured reflection profiles to the corresponding additional simulated reflection profiles; and

compute an average scale estimate from the first estimate of the unknown scale and the additional scale estimates.