System and method for revealing occluded objects in an image dataset

• US 8,086,071 B2
• Filed: 10/30/2007
• Issued: 12/27/2011
• Est. Priority Date: 10/30/2007
• Status: Active Grant

• Current Assignee: HERE Global B.V. (HERE Holding Corporation)
• Original Assignee: Navteq North America LLC (HERE Holding Corporation)
• Inventors: Chen, Xin, Gale, William
• Primary Examiner(s): CHEN, WENPENG
• Application Number:
  US11/980,033
• Publication Number:
  US 20090110239A1
• Time in Patent Office: 1,519 Days
• Field of Search: 382/284, 382/154, 382/103, 345/629, 345/421
• US Class Current: 382/284
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 15/205 Image-based rendering

Abstract

Disclosed are a system and method for identifying objects in an image dataset that occlude other objects and for transforming the image dataset to reveal the occluded objects. In some cases, occluding objects are identified by processing the image dataset to determine the relative positions of visual objects. Occluded objects are then revealed by removing the occluding objects from the image dataset or by otherwise de-emphasizing the occluding objects so that the occluded objects are seen behind it. A visual object may be removed simply because it occludes another object, because of privacy concerns, or because it is transient. When an object is removed or de-emphasized, the objects that were behind it may need to be “cleaned up” so that they show up well. To do this, information from multiple images can be processed using interpolation techniques. The image dataset can be further transformed by adding objects to the images.

Claims

1. A method for transforming an image dataset, the image dataset captured by a camera on a vehicle driving along a road, and the image dataset comprising data representing a first object and a second object, the first object at least partially occluding the second object from at least one point of view, the method comprising:

reading the image dataset from a first computer-readable medium;

determining relative positions of the first and second objects;

based, at least in part, on the relative positions of the first and second objects, transforming the image dataset to reveal a portion of the second object occluded by a portion of the first object from the at least one point of view, wherein the portion of the first object occluding the portion of the second object is deemphasized by being semi-transparent to reveal the portion of the second object previously occluded by the portion of the first object from the at least one point of view; and

storing the transformed image dataset on a second computer-readable medium.

2. The method of claim 1 wherein the first and second computer-readable media are the same medium.

3. The method of claim 1 wherein the image dataset comprises multiple images taken from multiple points of view, at least some of which images overlap.

4. The method of claim 1 wherein the image dataset represents a locality selected from the group consisting of:

a real-world locality and a virtual-world locality.

5. The method of claim 1 wherein the image dataset comprises 3D pixels.

6. The method of claim 5 wherein determining relative positions of the first and second objects comprises comparing position information of sets of 3D pixels that represent the first and second objects.

7. The method of claim 1 wherein the first object represents an item selected from the group consisting of:

a person, a vehicle, a structure, a sign, and a plant.

8. The method of claim 1 wherein determining relative positions of the first and second objects comprises:

comparing a plurality of images in the image dataset; and

based, at least in part, on the comparing, finding a parallax movement of the first object with respect to the second object.

9. The method of claim 1 wherein transforming the image dataset is further based, at least in part, on detecting a face in the first object.

10. The method of claim 1 wherein transforming the image dataset is further based, at least in part, on:

comparing a plurality of images in the image dataset; and

based, at least in part, on the comparing, finding a translational movement of the first object with respect to the second object.

11. The method of claim 1 wherein transforming the image dataset comprises interpolating 3D pixels in the second object.

12. The method of claim 11 wherein interpolating 3D pixels in the second object comprises:

comparing 3D pixels from a plurality of images in the image dataset; and

replacing 3D pixels occluded by the first object in a first image with 3D pixels corresponding to a same location in a second image and not occluded by the first object.

13. The method of claim 12 further comprising applying an affine transformation to at least some of the 3D pixels in the second image not occluded by the first object.

14. The method of claim 1 wherein transforming the image dataset comprises interpolating a texture of an object adjacent to the second object.

15. The method of claim 1 further comprising transforming the image dataset by adding data representing a third object.

16. A non-transitory computer-readable medium containing computer-executable instructions for a method for transforming an image dataset, the image dataset comprising data representing a first object and a second object, the first object at least partially occluding the second object from at least one point of view, the method comprising:

reading the image dataset from a first computer-readable medium;

determining relative positions of the first and second objects;

based, at least in part, on the relative positions of the first and second objects, transforming the image dataset to reveal a portion of the second object occluded by a portion of the first object from the at least one point of view, wherein the portion of the first object occluding the portion of the second object is deemphasized by being semi-transparent to reveal the portion of the second object previously occluded by the portion of the first object from the at least one point of view; and

storing the transformed image dataset on a second computer-readable medium.

17. A system for transforming an image dataset, the system comprising:

a camera system for gathering multiple images representing a real-world locality, the images taken from multiple points of view, at least some of which images overlap, wherein the camera system includes a LIDAR device to gather position information;

a first computer-readable medium for containing an image dataset, the image dataset comprising images gathered by the camera system;

a program for determining 3D positions of pixels in the image dataset using data gathered via the LIDAR device;

a program for determining relative positions of objects represented by data in the image dataset based on the determined 3D positions of the pixels;

a program for determining whether a first object in the image dataset at least partially occludes a second object from at least one point of view;

a program for transforming the image dataset to reveal an occluded portion of the second object from the at least one point of view, wherein a portion of the first object occluding the portion of the second object is deemphasized by being semi-transparent to reveal the portion of the second object previously occluded by the portion of the first object from the at least one point of view; and

a second computer-readable medium for containing the transformed image dataset.

18. The system of claim 17 wherein the first and second computer-readable media are the same medium.

19. The system of claim 17 further comprising a program for transforming the image dataset by adding data representing an object.

20. A method for accessing a transformed image dataset, the transformed image dataset based on image data captured by a camera on a vehicle driving along a road, and the transformed image dataset derived, at least in part, from an initial image dataset, wherein deriving comprises flagging a portion of an object of the image dataset that at least partially occludes another object, the method comprising:

reading from a computer-readable medium data representing images in the transformed image dataset;

receiving navigation information;

based, at least in part, on the received navigation information, determining a point of view; and

presenting images from the transformed image dataset based on the point of view, wherein a portion of the other object occluded by the flagged object portion is revealed in a first display application from the point of view, wherein the portion of the other object occluded by the flagged object portion is not revealed in a second display application from the point of view, and wherein the decision to reveal the portion of the occluded object in the first display application is based on the flagging.

21. The method of claim 20 wherein the method comprises an application selected from the group consisting of:

an image-based navigation application, a route calculator, a route guidance application, a destination selector, electronic yellow pages, a vehicle-positioning application, and a map display application.

22. The method of claim 20 wherein the image dataset represents a locality selected from the group consisting of:

a real-world locality and a virtual-world locality.

23. The method of claim 20 wherein reading from a computer-readable medium comprises reading over a network.

24. The method of claim 20 further comprising receiving updates to the transformed image dataset.

25. A non-transitory computer-readable medium containing computer-executable instructions for performing a method for accessing a transformed image dataset, the transformed image dataset derived, at least in part, from an initial image dataset, wherein deriving comprises flagging a portion of an object of the image dataset that at least partially occludes another object, the method comprising:

reading from a computer-readable medium data representing images in the transformed image dataset;

receiving navigation information;

based, at least in part, on the received navigation information, determining a point of view; and

presenting images from the transformed image dataset based on the point of view, wherein a portion of the other object occluded by the flagged object portion is revealed in a first display application from the point of view, wherein the portion of the other object occluded by the flagged object portion is not revealed in a second display application from the point of view, and wherein the decision to reveal the portion of the occluded object in the first display application is based on the flagging.