Multi-dimensional data capture of an environment using plural devices

• US 9,161,019 B2
• Filed: 09/10/2013
• Issued: 10/13/2015
• Est. Priority Date: 09/10/2012
• Status: Active Grant

• Current Assignee: Aemass, Inc.
• Original Assignee: Aemass, Inc.
• Inventors: Millett, Marshall Reed
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Le, Peter D
• Application Number:
  US14/022,624
• Publication Number:
  US 20140071234A1
• Time in Patent Office: 763 Days
• US Class Current: 1/1
• CPC Class Codes: G01S 17/06 Systems determining position data of a target
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/282 for generating image signals corresponding to three or more geometrical viewpoints, e.g. multi-view systems

Abstract

Embodiments of the invention describe apparatuses, systems, and methods related to data capture of objects and/or an environment. In one embodiment, a user can capture time-indexed three-dimensional (3D) depth data using one or more portable data capture devices that can capture time indexed color images of a scene with depth information and location and orientation data. In addition, the data capture devices may be configured to captured a spherical view of the environment around the data capture device.

Claims

1. A non-transitory machine-readable storage medium carrying one or more sequences of instructions which, when executed by one or more processors, cause the one or more processors to carry out a method of capturing a scene in motion, the method comprising:

receiving first time-indexed depth data from a first data capture device comprising a plurality of 3D data capturing modules, the first time-indexed depth data comprising distance measurements from the first data capture device to a plurality of different points on first surfaces of the scene in motion for each of a plurality of times over a period of time;

receiving second time-indexed depth data from a second data capture device comprising a plurality of 3D data capturing modules, the second time-indexed depth data comprising distance measurements from the second data capture device to a plurality of different points on second surfaces of the scene in motion for each of the plurality of times over the same period of time;

receiving first location data indicating the first data capture device'"'"'s location during the period of time;

receiving second location data indicating the second data capture device'"'"'s location during the period of time;

receiving relative location data indicating how the first data capture device and the second data capture device are located relative to each other during the period of time;

generating time-indexed three dimensional (3D) frames, wherein generating the time-indexed 3D frames comprises;

determining, for each of the plurality of times, a 3D coordinate in a coordinate system for each of the pluralities of different points on the first and second surfaces of the scene in motion based on the first time-indexed depth data, the second time-indexed depth data, the first location data, the second location data, and the relative location data; and

storing, to an electronic memory, the time-indexed 3D frames capturing the scene in motion.

2. The non-transitory machine-readable storage medium of claim 1, wherein the one or more sequences of instructions cause the one or more processors to carry out the method further comprising:

receiving first color data from the first data capture device for the plurality of different points on the first surfaces of the scene in motion during the period of time; and

receiving second color data from the second data capture device for the plurality of different points on the second surfaces of the scene in motion during the period of time;

wherein generating the time-indexed 3D frames further comprises associating, for each of the plurality of times, the first and second color data with the 3D coordinates for each of the pluralities of different points.

3. The non-transitory machine-readable storage medium of claim 1, wherein the first and second time-indexed depth data each comprise the distance measurements within 4π

steradian.

4. The non-transitory machine-readable storage medium of claim 1, wherein the first and second location data each comprise averaged motion data from a plurality of accelerometers.

5. The non-transitory machine-readable storage medium of claim 1, wherein the relative location data comprises first geospatial data from a first GPS indicating a first geospatial location of the first data capture device, and second geospatial data from a second GPS indicating a second geospatial location of the second data capture device.

6. The non-transitory machine-readable storage medium of claim 1, wherein the relative location data is based on recognition of a shape of the second data capture device by the first data capture device.

7. The non-transitory machine-readable storage medium of claim 1, wherein the one or more sequences of instructions cause the one or more processors to carry out the method further comprising:

receiving first orientation data indicating the first data capture device'"'"'s orientation during the period of time, wherein the first orientation data is based on detected motion of a first defined locus and orientation of the first data capture device; and

receiving second orientation data indicating the second data capture device'"'"'s orientation during the period of time, wherein the second orientation data is based on detected motion of a second defined locus and orientation of the second data capture device;

wherein determining, for each of the plurality of times, the 3D coordinate in the coordinate system for each of the pluralities of different points is further based on the first and second orientation data.

8. The non-transitory machine-readable storage medium of claim 1, wherein the one or more sequences of instructions cause the one or more processors to carry out the method further comprising:

recognizing a spatial overlap in a portion of the first and second time-indexed depth data; and

discarding the overlapping portion from one of the first and second time-indexed depth data;

wherein determining, for each of the plurality of times, the 3D coordinate in the coordinate system for each of the pluralities of different points is based on the overlapping portion from the other of the first and second time-indexed depth data.

25. The non-transitory machine-readable storage medium of claim 1, wherein:

the first and second time-indexed depth data is captured at least 24 times or more per second; and

wherein the time indexed 3D frames have a frame rate of 24 or more frames per second.

26. The non-transitory machine-readable storage medium of claim 1, wherein the time indexed 3D frames capture the scene in motion from a first perspective of the first data capture device and a second perspective of the second data capture device, as the scene changes over the period of time.

27. The non-transitory machine-readable storage medium of claim 7, wherein:

the distance measurements from the first data capture device were captured while the first data capture device was moving; and

wherein determining, for each of the plurality of times, the 3D coordinate in the coordinate system for each of the pluralities of different points is further based on changes in the first location data and changes in the first orientation data as the first data capture device moved.

9. An apparatus comprising:

a plurality of 3D data capturing modules, wherein each of the plurality of 3D data capturing modules captures time-indexed depth data comprising distance measurements to a plurality of different points on surfaces of a scene in motion for each of a plurality of times over a period of time;

one or more digital image sensors that detect image data of the surfaces of the scene in motion;

one or more sensors that detect changes in the apparatus'"'"'s location over the period of time;

a self-localization module that generates geospatial data indicating a location of the apparatus at each of the plurality of times over the period of time based on data from the one or more sensors; and

a communication interface that sends the time-indexed depth data, the image data, and the geospatial data to a computing device, wherein the computing device receives second time-indexed depth data, second image data, and second geospatial data from a second apparatus captured over the same period of time, and generates time indexed three dimensional (3D) frames capturing the scene in motion, wherein a given one of the time-indexed 3D frames comprises 3D coordinates in a coordinate system for each of the pluralities of different points on the surfaces of the scene in motion based on the first and second time-indexed depth data, the first and second color data, and the first and second geospatial data.

10. The apparatus of claim 9, wherein

the plurality of 3D data capturing modules are to capture 4 steradian depth data per frame of the time-indexed depth data.

11. The apparatus of claim 9, wherein the computing device is a device remote from the apparatus.

12. The apparatus of claim 9, wherein the apparatus comprises the computing device that receives the first and second time-indexed depth data, the first and second color data, and the first and second geospatial data and generates the time indexed 3D frames capturing the scene in motion.

13. The apparatus of claim 9, wherein the one or more sensors that detect changes in the apparatus'"'"'s location comprise a plurality of accelerometers to detect motion, and wherein the self-localization module receives motion data from the plurality of accelerometers, averages the motion data from the plurality of accelerometers, and generates the geospatial data based on averaged motion data.

14. The apparatus of claim 13, wherein the one or more sensors further comprise:

a vibration sensor, and wherein the plurality of accelerometers comprise at least three accelerometers.

15. A system for capturing a scene in motion, the system comprising:

a first data capture device that includes a plurality of 3D data capturing modules that record first distance data from the first data capture device to a plurality of different points on first surfaces of one or more objects of the scene in motion and a first image camera that records first image data of the scene in motion and the objects therein for each of a plurality of times over a period of time, and a first sensor that senses first location data;

a second data capture device that includes a plurality of 3D data capturing modules that record second distance data from the second data capture device to a plurality of different points on second surfaces of the one or more objects of the scene in motion and a second image camera that records second image data of the scene in motion and the objects therein for each of the plurality of times over the period of time, and a second sensor that senses second location data; and

a data processor that receives the first and second distance data, and the first and second image data, and the first and second location data, and processes the data to create a unified data cloud of the scene in motion and the objects therein, wherein the unified data cloud comprises 3D coordinates in a coordinate system for each of the pluralities of different points on the first and second surfaces the one or more objects of the scene in motion.

16. The system of claim 15, wherein the first and second image data is grayscale image data.

17. The system of claim 15, wherein the first and second image data is color image data.

18. The system of claim 15, wherein the first distance data and the first image data are recorded from a first point-of-view (POV) and the second distance data and the second image data are recorded from a second POV and the data processor creates the unified data cloud such that the scene in motion and the objects therein may be viewed from multiple POVs when the data is formatted and viewed.

19. The system of claim 15, wherein the data processor determines if first and second image data record image data of the same objects and performs data compression where image data overlaps.

20. The system of claim 15, wherein the first and second sensors comprise motion sensors that track relative motion.

21. The system of claim 20, wherein the first and second sensors each include plural motion sensors, and the location data from the plural motion sensors in each data capture device are averaged to provide greater accuracy of location data.

22. The system of claim 15, wherein the first and second sensors further sense orientation of the respective data capture device.

23. The system of claim 15, wherein the first data capture device and the second data capture device record sequential first and second depth data respectively at different times within the same period of time so that the system can record data at a data capture rate greater than a maximum data capture rate of the first or second data capture device alone.

24. The system of claim 15, wherein the plurality of 3D data capturing modules of the first data capture device record sequential first and third depth data respectively at different times within the same period of time so that the first data capture device can record data at a data capture rate greater than a maximum data capture rate of one of the plurality of 3D data capturing modules alone.