System and method for producing multi-angle views of an object-of-interest from images in an image dataset

• US 9,858,717 B2
• Filed: 12/17/2015
• Issued: 01/02/2018
• Est. Priority Date: 12/18/2007
• Status: Active Grant

• Current Assignee: HERE Global B.V. (Nokia Corporation)
• Original Assignee: HERE Global B.V. (Nokia Corporation)
• Inventors: Lynch, James, Gale, William
• Primary Examiner(s): BROOME, SAID A
• Application Number:
  US14/972,894
• Publication Number:
  US 20160110918A1
• Time in Patent Office: 747 Days
• Field of Search: None
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/5838 using colour
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06V 20/582 of traffic signs
• CPC Class Codes: G06V 20/584 of vehicle lights or traffic lights
• CPC Class Codes: G06V 20/588 Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
• CPC Class Codes: H04N 23/63 by using electronic viewfinders

Abstract

Disclosed are a system and method for creating multi-angle views of an object-of-interest from images stored in a dataset. A user specifies the location of an object-of-interest. As the user virtually navigates through the locality represented by the image dataset, his current virtual position is determined. Using the user'"'"'s virtual position and the location of the object-of-interest, images in the image dataset are selected and interpolated or stitched together, if necessary, to present to the user a view from his current virtual position looking toward the object-of-interest. The object-of-interest remains in the view no matter where the user virtually travels. From the same image dataset, another user can select a different object-of-interest and virtually navigate in a similar manner, with his own object-of-interest always in view. The object-of-interest also can be “virtual,” added by computer-animation techniques to the image dataset. For some image datasets, the user can virtually navigate through time as well as through space.

Claims

1. A method comprising:

receiving an image collected by a camera, wherein the image includes a plurality of pixels;

receiving distance data from a range-finding device;

registering the plurality of pixels of the image data with the distance data;

identifying, using a processor, a first location for a viewpoint;

adding a virtual object-of-interest to the image, wherein the virtual object-of-interest is added as a part of a landscape represented by the image;

identifying, using the processor, a second location for the virtual object-of-interest in the image; and

determining, using the processor, a view of the virtual object-of-interest from the first location toward the second location to test for the utility of the virtual object-of-interest to determine if it is visible from a distance and at an angle of view associated with the first location with respect to the second location.

2. The method of claim 1, wherein the virtual object-of-interest is a geographic indicator.

3. The method of claim 2, wherein the geographic indicator is a virtual street sign that includes a road name, a number of lanes, a location of a traffic signal or sign, a speed limit, or a turn restriction.

4. The method of claim 1, further comprising:

receiving a third location for a viewpoint; and

modifying the view of the virtual object-of-interest to be from the third location toward the second location.

5. The method of claim 1, wherein the first location is a current location of a mobile device.

6. The method of claim 1, further comprising:

analyzing the image for an object of concern; and

modifying the object of concern.

7. The method of claim 6, wherein at least a portion of the object of concern is replaced by the virtual object-of-interest.

8. The method of claim 1, further comprising:

sending the view to a mobile device.

9. The method of claim 1, wherein registering the plurality of pixels of the image data with the distance data comprises:

receiving a geographic location where the image was collected; and

assigning three-dimensional locations to the plurality of pixels using the geographic location where the image was collected and the distance data.

10. An apparatus comprising:

a memory configured to store an image collected by a camera, wherein the image includes a plurality of pixels, and to store distance data collected by a range-finding device; and

a processor configured to register the plurality of pixels of the image data with the distance data, to add a virtual object to the image at a user specified location, and to generate a view from a first location toward a second location associated with the user specified location of the virtual object,

wherein the virtual object is added to as part of the scene captured in the image to test the utility of the object-of-interest by determining if the virtual object-of-interest is visible from different distances and different angles of view.

11. The apparatus of claim 10, wherein the virtual object is a geographic indicator.

12. The apparatus of claim 11, wherein the geographic indicator is a virtual street sign that includes a road name, a number of lanes, a location of a traffic signal or sign, a speed limit, or a turn restriction.

13. The apparatus of claim 10, wherein the processor is configured to modify the view to be from a third location toward the second location, wherein the third location is received from a mobile device.

14. The apparatus of claim 11, wherein the third location is a current location of the mobile device.

15. A non-transitory computer-readable medium containing computer-executable instructions for performing a method comprising:

accessing an image collected by a camera, wherein the image includes a plurality of pixels and a virtual object added by a user within the image, the virtual object added to a street scene represented by the image;

accessing distance data collected by a range-finding device;

assigning three-dimensional locations to the plurality of pixels based on the distance data;

identifying, using a processor, a first location for a viewpoint;

identifying, using the processor, a second location for the virtual object in the image; and

generating, using the processor, a modified image with the viewpoint from the first location and facing toward the second location, the modified image generated to assess the utility of the of the virtual object added in the street scene from the distance and angle of view from the first location to the second location.

16. The non-transitory computer-readable medium of claim 15, wherein the virtual object is a geographic indicator.

17. The non-transitory computer-readable medium of claim 16, wherein the geographic indicator includes a road characteristic.

18. The non-transitory computer-readable medium of claim 16, wherein the geographic indicator includes a location of a traffic signal or sign.

19. The non-transitory computer-readable medium of claim 15, wherein the virtual object is a landscape object.

20. The non-transitory computer-readable medium of claim 19, wherein the landscape object comprises flowers, trees, planters or a combination thereof.