System and methods for scanning with integrated radar detection and image capture

• US 10,254,395 B2
• Filed: 12/04/2013
• Issued: 04/09/2019
• Est. Priority Date: 12/04/2013
• Status: Active Grant

• Current Assignee: Trimble Inc.
• Original Assignee: Trimble Inc.
• Inventors: Nichols, Mark Edward, Wallace, Gregory Craig
• Primary Examiner(s): Barker, Matthew M
• Application Number:
  US14/096,330
• Publication Number:
  US 20160223647A1
• Time in Patent Office: 1,952 Days
• CPC Class Codes: G01S 13/426 Scanning radar, e.g. 3D radar G01S13/66 takes precedence
• CPC Class Codes: G01S 13/726 Multiple target tracking
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/89 for mapping or imaging
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 2013/932 using own vehicle data, e.g. ground speed, steering wheel direction
• CPC Class Codes: G01S 7/046 using an intermediate storage device, e.g. a recording/reproducing device
• CPC Class Codes: G01S 7/10 Providing two-dimensional and co-ordinated display of distance and direction
• CPC Class Codes: G01S 7/2955 Means for determining the position of the radar coordinate system for evaluating the position data of the target in another coordinate system G01S7/24 takes precedence; sighting devices adapted for indirect laying of fire F41G3/16; inertial navigation G01C21/16

Abstract

A device and methods are provided for determining data points with an integrated radar sensor. In one embodiment, a method includes determining position of a device, scanning one or more objects, wherein scanning includes detecting data points by an integrated radar sensor of the device and capturing image data of the one or more objects, and determining data points for one or more objects based on the scanning. The method may also include correlating data points to one or more portions of the image data, assigning correlated data points to one or more portions of the image data, and storing, by the device, image data with data points. The device and methods may advantageously be employed for one or more of mapping, modeling, navigation and object tracking.

Claims

1. A method for mapping radar sensor data points to positions in image data, the method comprising:

determining a position of a device using a positioning module of the device;

capturing radar data points of one or more objects using an integrated radar sensor of the device, wherein the radar data points are associated with a polar coordinate system, and wherein the integrated radar sensor has an operational frequency within the range of 76-82 GHz;

capturing image data of the one or more objects using an integrated image sensor of the device;

determining a subset of the radar data points that correspond with the one or more objects;

determining radar geospatial data for each radar data point of the subset of radar data points, wherein determining the radar geospatial data for each radar data point of the subset of radar data points includes converting the radar data point from the polar coordinate system to a Cartesian grid and generating a spatial model of the one or more objects based on the radar data points, wherein the spatial model corresponds to a three-dimensional point cloud generated by rotating or repositioning the integrated radar sensor to scan a horizontal axis and scan a vertical axis using servo controls of the device, wherein the spatial model is generated relative to the position of the device, wherein a grid spacing of the Cartesian grid varies as a function of distance from the integrated radar sensor, and wherein a measurement resolution of the integrated radar sensor is not limited to the grid spacing for the Cartesian grid;

matching radar data points of the subset of radar data points to positions in the image data, wherein matching includes overlaying the image data and data from the spatial model to assign the radar geospatial data determined for each radar data point of the subset of radar data points to positions in the image data;

storing, by the device, the image data in association with matched radar data points; and

displaying the image data on a display of the device with the matched radar data points.

2. The method of claim 1, wherein the position of the device is determined based on at least one of global positioning data received by the positioning module and an orientation determined for the device.

3. The method of claim 1, wherein the integrated radar sensor includes at least one of an adaptive radar sensor, a solid-state radar sensor, or a radar on a chip sensor.

4. The method of claim 1, wherein the integrated radar sensor has a range resolution of 2 cm and an azimuth resolution of 0.2 degrees.

5. The method of claim 1, wherein the integrated radar sensor has a resolution of 1.7 cm at 5 m and 10 cm at 30 m.

6. The method of claim 1, wherein the integrated radar sensor has a bandwidth of 1 GHz or within the range of 2-3 GHz.

7. The method of claim 1, wherein at least one radar data point is associated with an interior of a structure or an exterior of a structure.

8. The method of claim 1, wherein the radar data points correspond to one or more surfaces of the one or more objects.

9. The method of claim 1, wherein each particular radar data point of the subset of radar data points is matched to a position in the image data based on elevation, azimuth, and reference to the particular radar data point relative to the position of the device.

10. The method of claim 9, further comprising:

detecting input corresponding to selection of multiple sets of radar data points; and

updating the display for presentation of the multiple sets of radar data points.

11. The method of claim 1, wherein the device is a handheld mobile device, and the radar data points are captured for one or more reference positions of the handheld mobile device.

12. The method of claim 1, further comprising tracking the one or more objects by identifying moving object in a field of view of the integrated radar sensor.

13. A device for determining data points, the device comprising:

a positioning module configured to determine positions;

an integrated radar sensor configured to capture radar data points, wherein the integrated radar sensor has an operational frequency within the range of 76-82 GHz;

an integrated image sensor configured to capture image data;

a display device; and

a processor in data communication with the positioning module, the integrated radar sensor, the integrated image sensor, and the display device, the processor programmed with instructions for performing operations including;

determining a position of the device using the positioning module;

capturing radar data points of one or more objects using the integrated radar sensor, wherein the radar data points are associated with a polar coordinate system;

capturing image data of the one or more objects using the integrated image sensor;

determining a subset of the radar data points that correspond with the one or more objects,

determining radar geospatial data for each radar data point of the subset of radar data points, wherein determining the radar geospatial data for each radar data point of the subset of radar data points includes converting the radar data point from the polar coordinate system to a Cartesian grid and generating a spatial model of the one or more objects based on the radar data points, wherein the spatial model corresponds to a three-dimensional point cloud generated by rotating or repositioning the integrated radar sensor to scan a horizontal axis and scan a vertical axis using servo controls of the device, wherein the spatial model is generated relative to the position of the device, wherein a grid spacing of the Cartesian grid varies as a function of distance from the integrated radar sensor, and wherein a measurement resolution of the integrated radar sensor is not limited to the grid spacing for the Cartesian grid;

matching radar data points of the subset of radar data points to positions in the image data, wherein matching includes overlaying the image data and data from the spatial model to assign the radar geospatial data determined for each data point of the subset or radar data points to positions in the image data;

storing the image data in association with matched radar data points; and

displaying the image data on the display with the matched radar data points.

14. The device of claim 13, wherein the position of the device is determined based on at least one of global positioning data received by the positioning module and an orientation determined for the device.

15. The device of claim 13, wherein the integrated radar sensor includes at least one of an adaptive radar sensor, a solid-state radar sensor, or a radar on a chip sensor.

16. The device of claim 13, wherein the integrated radar sensor has a range resolution of 2 cm and an azimuth resolution of 0.2 degrees.

17. The device of claim 13, wherein the integrated radar sensor has a resolution of 1.7 cm at 5 m and 10 cm at 30 m.

18. The device of claim 13, wherein the integrated radar sensor has a bandwidth of 1 GHz or within the range of 2-3 GHz.

19. The device of claim 13, wherein each radar data point is associated with an interior of a structure or an exterior of a structure.

20. The device of claim 13, wherein the radar data points correspond to one or more surfaces of the one or more objects.

21. The device of claim 13, wherein each particular radar data point of the subset of radar data points is matched to a position in the image data based on elevation, azimuth, and reference to the particular radar data point relative to the position of the device.

22. The device of claim 21, wherein the operations further comprise:

detecting input corresponding to selection of multiple sets of radar data points; and

updating the display for presentation of the multiple sets of radar data points.

23. The device of claim 13, wherein the device is a handheld mobile device, and the radar data points are captured for one or more reference positions of the handheld mobile device.

24. The device of claim 13, wherein the operations further comprise tracking the one or more objects by identifying moving objects in a field of view of the integrated radar sensor.

25. A non-transitory computer readable medium including computer executable instructions for mapping radar sensor data points to positions in image data, wherein the instructions, when executed by one or more processors cause the one or more processors to perform operations including:

determining a position of a device using a position module of the device;

capturing radar data points of one or more objects using an integrated radar sensor of the device;

capturing image data of the one or more objects using an integrated image sensor of the device, wherein the radar data points are associated with a polar coordinate system, and wherein the integrated radar sensor has an operational frequency within the range of 76-82 GHz;

determining a subset of the radar data points that correspond with the one or more objects;

determine radar geospatial data for each radar data point of the subset of radar data points, wherein determining the radar geospatial data for each radar data point of the subset of radar data points includes converting the radar data point from the polar coordinate system to a Cartesian grid and generating a spatial model of the one or more objects based on the captured radar data points, wherein the spatial model corresponds to a three-dimensional point cloud generated by rotating or repositioning the integrated radar sensor to scan a horizontal axis and scan a vertical axis using servo controls of the device, wherein the spatial model is generated relative to the position of the device, wherein a grid spacing of the Cartesian grid varies as a function of distance from the integrated radar sensor, and wherein a measurement resolution of the integrated radar sensor is not limited to the grid spacing for the Cartesian grid;

matching radar data points of the subset of radar data points to positions in the image data, wherein matching includes overlaying the image data and data from the spatial model to assign the radar geospatial data determined for each radar data point of the subset or radar data points to positions in the image data;

storing the image data in association with matched radar data points; and

displaying the image data on a display of the device with the matched radar data points.

26. The non-transitory computer readable medium of claim 25, wherein each particular radar data point of the subset of radar data points is matched to a position in the image data based on elevation, azimuth, and reference to the particular radar data point relative to the position of the device.

27. The non-transitory computer readable medium of claim 25, wherein the operations further comprise:

detecting input corresponding to selection of multiple sets of radar data points; and

updating the display for presentation of the multiple sets of radar data points.

28. The non-transitory computer readable medium of claim 25, wherein the device is a handheld mobile device, and the radar data points are captured for one or more reference positions of the handheld mobile device.

29. The non-transitory computer readable medium of claim 25, wherein the operations further comprise tracking the one or more objects by identifying moving objects in a field of view of the radar sensor.