Temporal processes for aggregating multi dimensional data from discrete and distributed collectors to provide enhanced space-time perspective

• US 10,455,134 B2
• Filed: 10/26/2015
• Issued: 10/22/2019
• Est. Priority Date: 10/26/2014
• Status: Active Grant

• Current Assignee: Galileo Group, Inc.
• Original Assignee: Galileo Group, Inc.
• Inventors: Barnes, Donald Michael, Grichnik, James Michael
• Primary Examiner(s): Johns, Andrew W
• Application Number:
  US15/521,871
• Publication Number:
  US 20180253862A1
• Time in Patent Office: 1,457 Days
• CPC Class Codes: G01C 19/5776 Signal processing not specific to any of the devices covered by groups G01C19/5607 - G01C19/5719
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 18/2137 based on criteria of topology preservation, e.g. multidimensional scaling or self-organising maps
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 7/40 Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04W 4/70 Services for machine-to-machine communication [M2M] or machine type communication [MTC]

Abstract

At a central controlling system, a composite map is constructed based on a previous dataset that includes (A) images of a target obtained by imaging devices at a first time, and (B) respective meta data associated with the images. Imaging devices collect image data during a second time after the first time. In accordance with the composite map, the imaging devices obtain images of the target and associate meta data with the images. The images and respective meta data are communicated to the central controlling system. The positions and orientations of the imaging devices when the respective images were obtained, and when the images were obtained, are used to index the images against the target, thereby aggregating multi-dimensional data for the target. Temporal information about a characteristic of the target over time is then extracted from the aggregated multi-dimensional data.

Claims

1. A computer-implemented method of aggregating multi-dimensional data for a target at a central controlling system, the method comprising:

constructing a composite map based on a previous dataset, wherein the previous dataset includes;

a plurality of two-dimensional pixilated images of the target obtained by one or more computer-enabled imaging devices during a first time, and

respective meta data associated with each of the plurality of two-dimensional pixilated images, wherein the respective meta data indicates (i) positional and orientation data that includes a position and orientation of a respective computer-enabled imaging device when the respective image is obtained, (ii) an indication of when the respective image is obtained, and an (iii) identity of the respective computer-enabled imaging device;

using each of the one or more computer-enabled imaging devices to collect respective image data of the target during a second time subsequent to the first time, by causing each respective computer-enabled imaging device of the one or more computer-enabled imaging devices to execute a method of data capture, including;

in accordance with the composite map based on the previous dataset, obtaining a respective two-dimensional pixilated image of the target;

associating meta data with the respective two-dimensional pixilated image, wherein the respective meta data indicates respective (i) positional and orientation data that includes a position and orientation of the respective computer-enabled imaging device when the respective image is obtained, (ii) an indication of when the respective image is obtained, and an (iii) identity of the respective computer-enabled imaging device; and

communicating to the central controlling system the respective two-dimensional pixilated image of the target and the respective meta data;

at the central controlling system, the central controlling system having one or more processors and memory for storing one or more programs for execution by the one or more processors, executing the method of;

receiving one or more respective two-dimensional pixilated images and associated meta data collected from the second time from each of the one or more computer-enabled imaging devices;

using the position and orientation of each of the respective computer-enabled imaging devices when the respective images were obtained, and the indication of when the images were obtained, to index the images against the target, thereby aggregating multi-dimensional data for the target;

extracting, from the aggregated multi-dimensional data for the target, temporal information about a characteristic of the target over time; and

identifying a deficiency in the composite map, wherein the obtaining, for each respective computer-enabled imaging device, of the respective two-dimensional pixilated image is responsive to and in accordance with the identified deficiency.

2. The computer-implemented method of claim 1, further comprising identifying a two-dimensional feature of the target using the composite map, wherein the obtaining, for each respective computer-enabled imaging device, of the respective two-dimensional pixilated image is in accordance with the identified two-dimensional feature.

3. The computer-implemented method of claim 1, further comprising identifying a three-dimensional feature of the target using the composite map, wherein the obtaining, for each respective computer-enabled imaging device, of the respective two-dimensional pixilated image is in accordance with the identified three-dimensional feature.

4. The computer-implemented method of claim 1, wherein obtaining, for each respective computer-enabled imaging device, the respective two-dimensional pixilated image in accordance with the composite map comprises:

obtaining the respective two-dimensional pixilated image in accordance with the positional and orientation data associated with the plurality of two-dimensional pixilated images of the previous dataset.

5. The computer-implemented method of claim 1, wherein the position and orientation of the respective computer-enabled imaging device, when the respective two-dimensional pixilated image is obtained, are substantially aligned with the positional and orientation data associated with at least one of the plurality of two-dimensional pixilated images from the previous dataset.

6. The computer-implemented method of claim 1, wherein:

the image data collected by each of the one or more computer-enabled imaging devices comprises a plurality of two-dimensional pixilated images obtained during the second time; and

a position and orientation of a respective one of the one or more computer-enabled imaging devices, for each of the plurality of two-dimensional pixilated images obtained during the second time, are substantially aligned with the positional and orientation data associated with a respective one of the plurality of two-dimensional pixilated images from the previous dataset.

7. The computer-implemented method of claim 1, wherein using each respective computer-enabled imaging device to collect respective image data further comprises:

providing information indicating a proximity of a current position and orientation of a respective computer-enabled imaging device, to a respective position and orientation associated with at least one of the plurality of two-dimensional pixilated images from the previous dataset.

8. The computer-implemented method of claim 7, wherein providing information indicating the proximity comprises displaying, on the respective computer-enabled imaging device, visual information indicating the proximity so as to enable alignment of the respective position and orientation of the respective computer-enabled imaging device to match the position and orientation associated with the at least one of the plurality of two-dimensional pixilated images from the previous dataset.

9. The computer-implemented method of claim 1, further comprising:

using the composite map, spatially registering the one or more respective two-dimensional pixilated images obtained during the second time.

10. The computer-implemented method of claim 9, wherein constructing the composite map comprises:

extracting two-dimensional features from each image of the previous dataset; and

using the two-dimensional features extracted from the previous dataset, creating a three-dimensional composite map comprising a plurality of three-dimensional points, wherein each respective three-dimensional point of the plurality of three-dimensional points is associated with one of a plurality of respective sets of the two-dimensional features extracted from the previous dataset,

each of the respective sets representing an appearance of an associated respective three-dimensional point across the previous dataset.

11. The computer-implemented method of claim 10, wherein the spatially registering comprises:

extracting two-dimensional features from each image obtained during the second time; and

matching a set of the two-dimensional features extracted from the images obtained during the second time, with one of the plurality of respective sets of the two-dimensional features extracted from the previous dataset,

thereby associating the set of the two-dimensional features extracted from the images obtained during the second time with a respective three-dimensional point, of the three-dimensional composite map, that is associated with the matched set of the two-dimensional features extracted from the previous dataset.

12. The computer-implemented method of claim 1, wherein constructing the composite map comprises:

matching a two-dimensional feature in a first image and in a second image of the plurality of two-dimensional pixilated images;

estimating a parallax between the first image and the second image using respective meta data associated with the first image and the second image; and

adding, when the parallax between the first image and the second image satisfies a parallax threshold, a two or three-dimensional point to the composite map at a distance obtained by triangulating the first image and the second image using respective meta data associated with the first image and the second image.

13. The computer-implemented method of claim 1, wherein:

the composite map based on the previous dataset is a first composite map, and

the image data collected during the second time by using each of the one or more computer-enabled imaging devices comprises a plurality of two-dimensional pixilated images of the target and the associated meta data,

the method further comprising;

constructing a second composite map based on the image data collected during the second time; and

comparing the second composite map with the first composite map to identify respective differences in one or more three-dimensional features from the first time to the second time.

14. The computer-implemented method of claim 13, wherein the respective identified differences include a respective shape of the one or more three-dimensional features.

15. The computer-implemented method of claim 13, wherein the respective identified differences include a respective volume of the one or more three-dimensional features.

16. The computer-implemented method of claim 13, wherein the respective identified differences include a respective surface contour of the one or more three-dimensional features.

17. The computer-implemented method of claim 13, wherein the respective identified differences include a respective smoothness or roughness of the one or more three-dimensional features.

18. The computer-implemented method of claim 13, wherein the respective identified differences include a respective opacity of the one or more three-dimensional features.

19. The computer-implemented method of claim 13, wherein the respective identified differences include a respective color gradient of the one or more three-dimensional features.