Method, medium, and system with 3 dimensional object modeling using multiple view points

• US 8,681,150 B2
• Filed: 01/24/2008
• Issued: 03/25/2014
• Est. Priority Date: 02/16/2007
• Status: Expired due to Fees

• Current Assignee: Samsung Electronics Co. Ltd. (Samsung Group)
• Original Assignee: Samsung Electronics Co. Ltd. (Samsung Group)
• Inventors: Kim, Keun-ho, Zhang, Hui, Kim, Do-kyoon, Jung, Seok-yoon, Konouchine, Anton, Vezhnevets, Vladimir
• Primary Examiner(s): McDowell, Jr., Maurice L
• Application Number:
  US12/010,440
• Publication Number:
  US 20080211809A1
• Time in Patent Office: 2,252 Days
• Field of Search: 345/420, 345/630
• US Class Current: 345/420
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06V 20/38 Outdoor scenes
• CPC Class Codes: G06V 20/39 Urban scenes

Abstract

A modeling method, medium, and system. The modeling method may include specifying an object in a plurality of 2 dimensional (2D) images expressed from different view points, considering an edge and contents of each component of each 2D image, generating a 3 dimensional (3D) model of each specified object, and matching the generated 3D models considering relative locations of each generated 3D model. Accordingly, a realistic 3D model of an object expressed in an image can be accurately generated.

Claims

1. A modeling method comprising:

identifying a detected object in an input image according to a corresponding view point of the input image by comparing characteristic information of the detected object with characteristic information of each of plural pre-defined objects, where the comparing performs a comparing of generated color pixel value based image information of a group of pixels corresponding to a portion of the detected object with respective color pixel value based image information of a group of pixels of respective portions of each of the plural pre-defined objects in combination with a comparing of generated non-color pixel value based information of the group of pixels corresponding to the portion of the detected object with respective non-color pixel value based information of the group of pixels of the respective portions of each of the plural pre-defined objects for the identifying of the detected object;

combining, as a combined 3D model, at least two respective 3D models, including a 3D model of the identified object from the view point of the input image, respectively from at least two different view points representing separate and/or overlapping portions of the identified object;

using at least one processing device placing the input image including the identified object over the combined 3D model; and

outputting the combined 3D model with the over-placed image.

2. The modeling method of claim 1, wherein each pre-defined object is represented by at least one pre-defined 3D model, and the method further comprises combining, as the combined 3D model, at least two respective 3D models based upon respective relationships with the identified object through different view points.

3. At least one non-transitory medium comprising computer readable code to control at least one processing element to implement the method of claim 1.

4. A modeling method generating a 3D model for an identified 2D object, comprising:

determining whether the object in a first image of a first view point overlaps with another identified object in the first image;

comparing, using at least one processing device, a portion of the first image corresponding to the object and a portion of a second image of a second view point including the object;

modifying the first image to include a portion of the second image corresponding to the object, through modification of the portion of the second image for insertion into the first image to replace the corresponding portion of the first image including the object, based upon the determining indicating that the object in the first image overlaps with the other object in the first image, to modify the first image so the object in the first image does not overlap the other object in the first image; and

placing the modified first image, using at least one processing device, over a select 3D model of the object corresponding to one or more view points.

5. The modeling method of claim 4, wherein the select 3D model is selected from a plurality of pre-defined 3D models based upon identified characteristics of the object.

6. The modeling method of claim 4, wherein the first view point is different from the second view point, and the modification of the portion of the second image includes view point modification of the portion of the second image to the first view point.

7. The modeling method of claim 4, further comprising combining, as a combined 3D model, at least two respective 3D models from at least two different view points representing separate and/or overlapping portions of the object and placing the modified first image over the combined 3D model.

8. At least one non-transitory medium comprising computer readable code to control at least one processing element to implement the method of claim 4.

9. A modeling method, comprising:

identifying a detected object present in at least a first image of a first view point by comparing analyzed edges and/or contents of components of the first image with pre-defined edge and/or content information of plural pre-defined objects corresponding to one or more different view points, where the comparing performs a comparing of generated color pixel value based image information of a group of pixels corresponding to a portion of the detected object with respective color pixel value based image information of a group of pixels of respective portions of each of the plural pre-defined objects in combination with a comparing of generated non-color pixel value based information of the group of pixels corresponding to the portion of the detected object with respective non-color pixel value based information of the group of pixels of the respective portions of each of the plural pre-defined objects for identifying the detected object;

generating a 3D model based upon results of the comparing of the analyzed edges and/or contents of components and based upon 3D positional information of the identified object in the first image and pre-defined 3D positional information of one or more of the plural pre-defined objects corresponding to the identified object; and

placing, using at least one processing device, the first image over the generated 3D model.

10. The modeling method of claim 9, further comprising combining, as a combined 3D model, at least two respective 3D models from at least two different view points representing separate and/or overlapping portions of the identified object and placing the first image over the combined 3D model.

11. The modeling method of claim 9, further comprising arranging a plurality of generated 3D models of respective identified objects, including the generated 3D model, in a 3D space according to respective 3D positional information of each respective identified object so that relative locations between different identified objects in the first image, including the identified object, are relatively reflected in the 3D space.

12. The modeling method of claim 11, wherein the arranging of the plurality of generated 3D models includes merging generated 3D models that are determined to have same 3D positional information.

13. The modeling method of claim 9, further comprising:

comparing the first image and a second image of a second view point;

determining whether the identified object in the first image overlaps with another object in the first image;

modifying the first image to include a portion of the second image with reference to the identified object, as included in the second image, through modification of the portion of the second image and insertion into the first image to replace a corresponding portion of the first image, based upon the determining indicating that the identified object in the first image overlaps with the other object in the first image, to modify the first image so the object in the first image does not overlap the other object in the first image; and

placing the modified first image over the generated 3D model.

14. The modeling method of claim 13, wherein the first view point is different from the second view point, and the modification of the portion of the second image includes at least view point modification of the portion of the second image to the first view point.

15. The modeling method of claim 9, further comprising:

comparing the first image and a second image of a second view point;

determining whether the identified object in the first image overlaps with another object in the first image;

modifying the first image to include a portion of the second image with reference to the identified object, as included in the second image, through modification of the portion of the second image and insertion into the first image to replace a corresponding portion of the first image, based upon the determining indicating that the identified object in the first image overlaps with the other object in the first image, to modify the first image so the object in the first image does not overlap the other object in the first image;

combining, as a combined 3D model, at least two respective 3D models from at least two different view points representing separate and/or overlapping portions of the identified object; and

placing the modified image over the combined 3D model.

16. At least one non-transitory medium comprising computer readable code to control at least one processing element to implement the method of claim 15.

17. The modeling method of claim 9, wherein the generating of the 3D model includes calculating depth values of pixels corresponding to the identified object.

18. The modeling method of claim 9, wherein the comparing of the analyzed edges and/or contents of components comprises comparing detected texture information in the first image respectively with pre-defined texture information of the plural pre-defined objects for identifying the detected object in the first image.

19. The modeling method of claim 9, wherein the comparing of the analyzed edges and/or contents of components comprises comparing detected edge information in the first image respectively with pre-defined edge information of the plural pre-defined objects for identifying the detected object in the first image.

20. The modeling method of claim 9, wherein the comparing of the analyzed edges and/or contents of components comprises comparing detected color component information in the first image respectively with pre-defined color component information of the plural pre-defined objects for identifying the detected object in the first image.

21. The modeling method of claim 9, wherein the comparing of the analyzed edges and/or contents of components comprises comparing detected location information in the first image respectively with pre-defined location information of the plural pre-defined objects for identifying the detected object in the first image.

22. The modeling method of claim 9, wherein the comparing of the analyzed edges and/or contents of components comprises converting the first image into multiple images, each representing differing information of the first image, including at least two of edge information, texture information, color information, location information, and/or 3D geometry information for the comparing with the edge and/or content information of the plural pre-defined objects.

23. The modeling method of claim 22, further comprising dividing the first image according to super pixels representing preset similarities between pixels for the converting of the first image into the multiple images.

24. At least one non-transitory medium comprising computer readable code to control at least one processing element to implement the method of claim 9.

25. A modeling system including at least one processing device, the system comprising:

an identification unit to identify a detected object in an input image according to a corresponding view point of the input image by comparing characteristic information of the detected object with characteristic information of each of plural pre-defined objects, where the comparing performs a comparing of generated color pixel value based image information of a group of pixels corresponding to a portion of the detected object with respective color pixel value based image information of a group of pixels of respective portions of each of the plural pre-defined objects in combination with a comparing of generated non-color pixel value based information of the group of pixels corresponding to the portion of the detected object with respective non-color pixel value based information of the group of pixels of the respective portions of each of the plural pre-defined objects for the identifying of the detected object;

a model matching unit to combine, as a combined 3D model, at least two respective 3D models, including a 3D model of the identified object from the view point of the input image, respectively from at least two different view points representing separate and/or overlapping portions of the identified object; and

an arranging unit, using the at least one processing device, to place the input image including the identified object over the combined 3D model.

26. The modeling system of claim 25, wherein each pre-defined object is represented by at least one pre-defined 3D model, such that the model matching unit combines, as the combined 3D model, at least two respective 3D models based upon respective relationships with the identified object through different view points.

27. A modeling system including at least one processing device, generating a 3D model for a 2D object, the system comprising:

a model generator to compare a first image of a first view point including the object and a second image of a second view point including the object, determining whether the object in the first image overlaps with another object in the first image, and modifying the first image to include a portion of the second image corresponding to the object through modification of the portion of the second image for insertion into the first image to replace a corresponding portion of the first image, based upon the determining indicating that the object in the first image overlaps with the other object in the first image, to modify the first image so the object in the first image does not overlap the other object in the first image; and

an arranging unit, using the at least one processing device, to place the modified first image over a select 3D model of the object corresponding to one or more view points.

28. The modeling system of claim 27, wherein the select 3D model is selected by an identification unit from a plurality of pre-defined 3D models based upon identified characteristics of the object.

29. The modeling system of claim 27, wherein the first view point is different from the second view point, and the modification of the portion of the second image includes at least view point modification of the portion of the second image to the first view point.

30. The modeling system of claim 27, further comprising a model matching unit to combine, as a combined 3D model, at least two respective 3D models from at least two different view points representing separate and/or overlapping portions of the object, such that the arranging unit places the modified first image over the combined 3D model.

31. A modeling system including at least one processing device, the system comprising:

an object identification unit to identify a detected object present in at least a first image of a first view point by comparing analyzed edges and/or contents of components of the first image with respective pre-defined edge and/or content information for each of plural pre-defined objects, obtained from an object storage arrangement configured for enabling an identifying of which of the plural pre-defined objects is specified for respective plural detected objects of input images of one or more view points;

a model generator to generate a 3D model based upon results of the comparing of the analyzed edges and/or contents of components and based upon 3D positional information of the identified object in the first image and pre-defined 3D positional information of one or more of the plural pre-defined objects corresponding to the identified object; and

an arranging unit, using the at least one processing device, to place the first image over the generated 3D model.

32. The modeling system of claim 31, further comprising a model matching unit to combine, as a combined 3D model, at least two respective 3D models from at least two different view points representing separate and/or overlapping portions of the identified object,

wherein the arranging unit places the first image over a corresponding combined 3D model.

33. The modeling system of claim 31, wherein the arranging unit arranges a plurality of generated 3D models of respective identified objects, including the generated 3D model, in a 3D space according to respective 3D positional information of each respective identified object so that relative locations between different identified objects in the first image, including the identified object, are relatively reflected in the 3D space.

34. The modeling system of claim 33, wherein the arranging of the plurality of generated 3D models includes merging generated 3D models that are determined to have same 3D positional information.

35. The modeling system of claim 31, wherein the model generator further compares the first image and a second image of a second view point, determines whether the identified object in the first image overlaps with another object in the first image, and modifies the first image to include a portion of the second image with reference to the identified object, as included in the second image, through modification of the portion of the second image for insertion into the first image to replace a corresponding portion of the first image, based upon the determining indicating that the identified object in the first image overlaps with the other object in the first image, to modify the first image so the object in the first image does not overlap the other object in the first image,

wherein the arranging unit places the modified first image over the generated 3D model.

36. The modeling system of claim 35, wherein the first view point is different from the second view point, and the modification of the portion of the second image includes at least view point modification of the portion of the second image to the first view point.

37. The modeling system of claim 31, wherein the model generator compares the first image and a second image of a second view point, determines whether the identified object in the first image overlaps with another object in the first image, and modifies the first image to include a portion of the second image with reference to the identified object, as included in the second image, through modification of the portion of the second image for insertion into the first image to replace a corresponding portion of the first image, based upon the determining indicating that the identified object in the first image overlaps with the other object in the first image, to modify the first image so the object in the first image does not overlap the other object in the first image, and the system further comprises:

a model matching unit to combine, as a combined 3D model, at least two respective 3D models from at least two different view points representing separate and/or overlapping portions of the identified object,

wherein the arranging unit places the modified image over the combined 3D model.

38. The modeling system of claim 31, wherein the arranging unit generates the 3D model by at least calculating depth values of pixels corresponding to the identified object.

39. The modeling system of claim 31, wherein the comparing of the analyzed edges and/or contents of components comprises comparing detected texture information in the first image respectively with pre-defined texture information of the plural pre-defined objects for identifying the detected object in the first image.

40. The modeling system of claim 31, wherein the comparing of the analyzed edges and/or contents of components comprises comparing detected edge information in the first image respectively with pre-defined edge information of the plural pre-defined objects for identifying the detected object in the first image.

41. The modeling system of claim 31, wherein the comparing of the analyzed edges and/or contents of components comprises comparing detected color component information in the first image respectively with pre-defined color component information of the plural pre-defined objects for identifying the detected object in the first image.

42. The modeling system of claim 31, wherein the comparing of the analyzed edges and/or contents of components comprises comparing detected location information in the first image respectively with pre-defined location information of the plural pre-defined objects for identifying the detected object in the first image.

43. The modeling system of claim 31, wherein the comparing of the analyzed edges and/or contents of components comprises converting the first image into multiple images, each representing differing information of the first image, including at least two of edge information, texture information, color information, location information, and/or 3D geometry information for the comparing with the edge and/or content information of the plural pre-defined objects.

44. The modeling system of claim 43, wherein the comparing of the analyzed edges and/or contents of components further comprises dividing the first image according to super pixels representing preset similarities between pixels for the converting of the first image into the multiple images.