System and method for revealing occluded objects in an image dataset

• US 20090110239A1
• Filed: 10/30/2007
• Published: 04/30/2009
• Est. Priority Date: 10/30/2007
• Status: Active Grant

• Current Assignee: HERE Global B.V. (Nokia Corporation)
• Original Assignee: Navteq North America LLC (Nokia Corporation)
• Inventors: Chen, Xin, Gale, William
• Granted Patent: US 8,086,071 B2
• Time in Patent Office: Days
• US Class Current: 382/103
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 15/205 Image-based rendering

Abstract

Disclosed are a system and method for identifying objects in an image dataset that occlude other objects and for transforming the image dataset to reveal the occluded objects. In some cases, occluding objects are identified by processing the image dataset to determine the relative positions of visual objects. Occluded objects are then revealed by removing the occluding objects from the image dataset or by otherwise de-emphasizing the occluding objects so that the occluded objects are seen behind it. A visual object may be removed simply because it occludes another object, because of privacy concerns, or because it is transient. When an object is removed or de-emphasized, the objects that were behind it may need to be “cleaned up” so that they show up well. To do this, information from multiple images can be processed using interpolation techniques. The image dataset can be further transformed by adding objects to the images.

Claims

1. A method for transforming an image dataset, the image dataset comprising data representing a first object and a second object, the first object at least partially occluding the second object from at least one point of view, the method comprising:

reading the image dataset from a first computer-readable medium;

determining relative positions of the first and second objects;

based, at least in part, on the relative positions of the first and second objects, transforming the image dataset to reveal the second object; and

storing the transformed image dataset on a second computer-readable medium.

2. The method of claim 1 wherein the first and second computer-readable media are the same medium.

3. The method of claim 1 wherein the image dataset comprises multiple images taken from multiple points of view, at least some of which images overlap.

4. The method of claim 1 wherein the image dataset represents a locality selected from the group consisting of:

a real-world locality and a virtual-world locality.

5. The method of claim 1 wherein the image dataset comprises 3D pixels.

6. The method of claim 5 wherein determining relative positions of the first and second objects comprises comparing position information of sets of 3D pixels that represent the first and second objects.

7. The method of claim 1 wherein the first object represents an item selected from the group consisting of:

a person, a vehicle, a structure, a sign, and a plant.

8. The method of claim 1 wherein determining relative positions of the first and second objects comprises:

comparing a plurality of images in the image dataset; and

based, at least in part, on the comparing, finding a parallax movement of the first object with respect to the second object.

9. The method of claim 1 wherein transforming the image dataset is further based, at least in part, on detecting a face in the first object.

10. The method of claim 1 wherein transforming the image dataset is further based, at least in part, on:

comparing a plurality of images in the image dataset; and

based, at least in part, on the comparing, finding a translational movement of the first object with respect to the second object.

11. The method of claim 1 wherein transforming the image dataset comprises applying a technique selected from the group consisting of:

deleting at least some of the data representing the first object, flagging at least some of the data representing the first object, and rendering at least some of the first object semi-transparent.

12. The method of claim 1 wherein transforming the image dataset comprises interpolating 3D pixels in the second object.

13. The method of claim 12 wherein interpolating 3D pixels in the second object comprises:

comparing 3D pixels from a plurality of images in the image dataset; and

replacing 3D pixels occluded by the first object in a first image with 3D pixels corresponding to a same location in a second image and not occluded by the first object.

14. The method of claim 13 further comprising applying an affine transformation to at least some of the 3D pixels in the second image not occluded by the first object.

15. The method of claim 1 wherein transforming the image dataset comprises interpolating a texture of an object adjacent to the second object.

16. The method of claim 1 further comprising transforming the image dataset by adding data representing a third object.

17. A computer-readable medium containing computer-executable instructions for a method for transforming an image dataset, the image dataset comprising data representing a first object and a second object, the first object at least partially occluding the second object from at least one point of view, the method comprising:

reading the image dataset from a first computer-readable medium;

determining relative positions of the first and second objects;

based, at least in part, on the relative positions of the first and second objects, transforming the image dataset to reveal the second object; and

storing the transformed image dataset on a second computer-readable medium.

18. A system for transforming an image dataset, the system comprising:

a camera system for gathering multiple images representing a real-world locality, the images taken from multiple points of view, at least some of which images overlap;

a first computer-readable medium for containing an image dataset, the image dataset comprising images gathered by the camera system;

a program for determining relative positions of objects represented by data in the image dataset;

a program for determining whether a first object in the image dataset at least partially occludes a second object from at least one point of view;

a program for transforming the image dataset to reveal the at least partially occluded object; and

a second computer-readable medium for containing the transformed image dataset.

19. The system of claim 18 wherein the first and second computer-readable media are the same medium.

20. The system of claim 18 wherein the camera system comprises a LIDAR system to gather position information.

21. The system of claim 18 wherein the image dataset comprises 3D pixels.

22. The system of claim 18 further comprising a program for transforming the image dataset by adding data representing an object.

23. A computer-readable medium containing a transformed image dataset, the transformed image dataset derived, at least in part, from an initial image dataset, wherein deriving comprises transforming the image dataset to reveal an object that is at least partially occluded by an occluding object.

24. The computer-readable medium of claim 23 wherein the transformed image dataset represents a locality selected from the group consisting of:

a real-world locality and a virtual-world locality.

25. The computer-readable medium of claim 23 wherein the transformed image dataset comprises 3D pixels.

26. The computer-readable medium of claim 23 wherein transforming the image dataset comprises applying a technique selected from the group consisting of:

deleting at least some of the data representing the occluding object, flagging at least some of the data representing the occluding object, and rendering at least some of the occluding object semi-transparent.

27. A method for accessing a transformed image dataset, the transformed image dataset derived, at least in part, from an initial image dataset, wherein deriving comprises transforming the image dataset to reveal an object that is at least partially occluded by another object, the method comprising:

reading from a computer-readable medium data representing images in the transformed image dataset;

receiving navigation information;

based, at least in part, on the received navigation information, determining a point of view; and

presenting images from the transformed image dataset based on the point of view.

28. The method of claim 27 wherein the method comprises an application selected from the group consisting of:

an image-based navigation application, a route calculator, a route guidance application, a destination selector, electronic yellow pages, a vehicle-positioning application, and a map display application.

29. The method of claim 27 wherein the image dataset represents a locality selected from the group consisting of:

a real-world locality and a virtual-world locality.

30. The method of claim 27 wherein reading from a computer-readable medium comprises reading over a network.

31. The method of claim 27 further comprising receiving updates to the transformed image dataset.

32. A computer-readable medium containing computer-executable instructions for performing a method for accessing a transformed image dataset, the transformed image dataset derived, at least in part, from an initial image dataset, wherein deriving comprises transforming the image dataset to reveal an object that is at least partially occluded by another object, the method comprising:

reading from a computer-readable medium data representing images in the transformed image dataset;

receiving navigation information;

based, at least in part, on the received navigation information, determining a point of view; and

presenting images from the transformed image dataset based on the point of view.