Method and apparatus for virtual 3D model generation and navigation using opportunistically captured images

• US 9,998,684 B2
• Filed: 08/15/2014
• Issued: 06/12/2018
• Est. Priority Date: 08/16/2013
• Status: Expired due to Fees

• Current Assignee: Indiana University Research & Technology Corporation (Indiana University), the united states of america as represented by the secretary of the navy
• Original Assignee: Indiana University Research & Technology Corporation (Indiana University), the united states of america as represented by the secretary of the navy
• Inventors: Kapadia, Apu, Rahman, Md Zahidur, Crandall, David, Templeman, Robert E.
• Primary Examiner(s): VOLENTINE, REBECCA A
• Application Number:
  US14/460,914
• Publication Number:
  US 20150049170A1
• Time in Patent Office: 1,397 Days
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

A method and device for opportunistically collecting images of a location via a mobile device, the device including algorithms that allow the selective winnowing of collected images to facilitate the creation of a 3D model from the images while minimizing the storage needed to hold the pictures.

Claims

1. A method of assembling a data set for the creation of a 3D model of an environment including:

obtaining a plurality of images that have been passively and opportunistically captured by a mobile device;

storing the plurality of images on the mobile device in mobile device storage;

automatically electronically analyzing the stored images by the mobile device to assess at least one of quality, content, and redundancy;

automatically deleting one or more images from the mobile device storage responsive to the automatic electronic analysis to create a retained subset of the obtained plurality of images; and

constructing the 3D model of the environment in which the mobile device was located from the output retained subset of images, wherein the 3D model allows a user to select a portion of the model such that the selection causes presentation of a first set of images of the plurality of images obtained by the mobile device, the first set of images being selected from a set of images that were used to create the selected portion of the 3D model.

2. The method of claim 1, further including passively capturing the plurality of images at regular time intervals.

3. The method of claim 2, wherein the time interval is modified based on inputs from a motion detector within the mobile device.

4. The method of claim 1, further including capturing the plurality of images in a manner that is undetectable by a user of the mobile device.

5. The method of claim 1, further including outputting the retained subset from the mobile device.

6. The method of claim 1, wherein the automatic analysis of stored images includes a measurement of anisotropy of each image.

7. The method of claim 6, wherein images with anisotropy below a defined threshold are automatically deleted from the mobile device.

8. The method of claim 1, wherein images that have a similarity to other images stored on the mobile device above a threshold similarity value are automatically deleted from the mobile device.

9. The method of claim 1, further including obtaining orientation information for the mobile device and associating such orientation information with each of the obtained images.

19. The method of claim 1, further comprising estimating a lag between a hardware capture timestamp and an image system creation timestamp for each image of the plurality of opportunistically captured images.

10. A mobile device including:

a camera,

a memory; and

an image analyzer, the image analyzer operable to execute code that causes the image analyzer to;

opportunistically obtain a plurality of images captured by the camera and stored in the memory;

automatically electronically analyze the stored images to assess at least one of quality, content, and redundancy;

automatically delete one or more images from the mobile device storage responsive to the automatic electronic analysis to create a retained subset of the obtained plurality of images; and

constructing a 3D model of an environment in which the mobile device was located from the output retained subset of opportunistically obtained images, wherein the 3D model allows a user to select a portion of the model such that the selection causes presentation of a first set of images of the plurality of images obtained by the mobile device, the first set of images being selected from a set of images that were used to create the selected portion of the 3D model.

11. The mobile device of claim 10, further including at least one of a motion detector and an orientation detector.

12. The mobile device of claim 10, further including a transmitter/receiver operable to transmit images of the retained subset.

13. The mobile device of claim 10, further including an image collector, the collector operable to control the camera to obtain images, where such operation of the camera is done without a mobile device user being contemporaneously conscious that the camera is obtaining images.

14. The mobile device of claim 10, wherein the automatic analysis includes a measurement of anisotropy of each image and images with anisotropy below a defined threshold are automatically deleted from the mobile device.

15. A non-transitory computer readable medium having instructions thereon that when interpreted by a processor cause the processor to:

opportunistically obtain a plurality of images captured by a camera of a mobile device and stored in a memory of the mobile device;

automatically electronically analyze the stored images to assess at least one of quality, content, and redundancy;

automatically delete one or more images from the mobile device storage responsive to the automatic electronic analysis to create a retained subset of the obtained plurality of images; and

constructing a 3D model of an environment in which the mobile device was located from the output retained subset of opportunistically obtained images, wherein the 3D model allows a user to select a portion of the model such that the selection causes presentation of a first set of images of the plurality of images obtained by the mobile device, the first set of images being selected from a set of images that were used to create the selected portion of the 3D model.

16. The computer readable medium of claim 15, wherein the instructions further cause the processor to:

passively capture the plurality of images at regular time intervals.

17. The computer readable medium of claim 15, wherein the instructions further cause the processor to capture the plurality of images in a manner that is undetectable by a user of the mobile device.

18. The computer readable medium of claim 15, wherein the instructions further cause the processor to determine a measurement of anisotropy of each image such that images with anisotropy below a defined threshold are automatically deleted from the mobile device.