Multiframe image processing

• US 20040222987A1
• Filed: 05/08/2003
• Published: 11/11/2004
• Est. Priority Date: 05/08/2003
• Status: Abandoned Application

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Lin, I-Jong, Chang, Nelson Liang An
• Application Number:
  US10/434,351
• Publication Number:
  US 20040222987A1
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G01B 11/2509 Color coding
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G01B 11/2545 with one projection direction and several detection directions, e.g. stereo
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings

Abstract

Systems and methods of multiframe image processing are described. In one aspect, correspondence mappings from one or more anchor views of a scene to a common reference anchor view are computed, and anchor views are interpolated based on the computed correspondence mappings to generate a synthetic view of the scene.

Claims

1. A method of multiframe image processing, comprising:

computing correspondence mappings from one or more anchor views of a scene to a common reference anchor view; and

interpolating between anchor views based on the computed correspondence mappings to generate a synthetic view of the scene.

2. The method of claim 1, wherein computing correspondence mappings comprises:

projecting onto the scene a sequence of patterns of light symbols that temporally encode two-dimensional position information in the reference anchor view with unique light code symbols;

capturing light patterns reflected from the scene at one or more anchor views; and

computing a correspondence mapping between the reference anchor view and the one or more other anchor views based at least in part on correspondence between light symbol sequence codes captured at the one or more anchor views and light symbol sequence codes projected from the reference anchor view.

3. The method of claim 1, further comprising storing the computed correspondence mappings in a data structure including an array of points defined in a reference anchor view space and linked to respective lists of corresponding points in the one or more other anchor views.

4. The method of claim 1, wherein the synthetic view is generated by interpolating between the reference anchor view and at least one other anchor view.

5. The method of claim 1, wherein the synthetic view is generated by interpolating between anchor views in N dimensions, wherein N is an integer greater than 0.

6. The method of claim 5, wherein interpolating between anchor views comprises parameterizing an N-dimensional space of synthesizable views, and weighting contributions from anchor views being interpolated to the synthetic view based at least in part on relative proximity of the synthetic view to the anchor views being interpolated.

7. The method of claim 6, wherein the synthetic view is generated by interpolating between anchor views with contributions to the synthetic view weighted in accordance with Barycentric coordinates of the synthetic view defined relative to the anchor views being interpolated.

8. The method of claim 6, wherein the N-dimensional space of synthesizable views is parameterized discretely.

9. The method of claim 1, wherein the synthetic view is generated based on points visible in all anchor views being interpolated.

10. The method of claim 1, further comprising identifying in a given anchor view one or more regions occluded from visualizing the scene.

11. The method of claim 10, further comprising computing color information for occluded regions of the given anchor view based on color information in corresponding regions of at least one other anchor view.

12. The method of claim 10, further comprising computing coordinate information for occluded regions of the given anchor view by interpolating between neighboring non-occluded regions of the given anchor view.

13. The method of claim 1, further comprising ordering multiple points of the scene mapping to a common point in the synthetic view.

14. The method of claim 1, further comprising rotating an object in the scene about an axis.

15. A method of multiframe image processing, comprising:

computing correspondence mappings between one or more pairs of anchor views of a scene;

parameterizing a discretized space of synthesizable views referenced to the anchor views of the scene; and

interpolating between anchor views in the parameterized discretized space based on the computed correspondence mappings to generate a synthetic view of the scene.

16. The method of claim 15, wherein the discretized space of synthesizable views is parameterized in N dimensions and the synthetic view is generated by interpolating between anchor views in N dimensions, wherein N is an integer greater than 0.

17. The method of claim 15, wherein contributions from anchor views being interpolated to the synthetic view are weighted based at least in part on relative proximity of the synthetic view to the anchor views being interpolated.

18. The method of claim 15, wherein the synthetic view is generated by interpolating between anchor views with contributions to the synthetic view weighted in accordance with Barycentric coordinates of the synthetic view defined relative to the anchor views being interpolated.

19. A method of multiframe image processing, comprising:

computing correspondence mappings between one or more pairs of anchor views of a scene;

identifying in a given anchor view one or more regions occluded from visualizing the scene; and

computing color information for occluded regions of the given anchor view based on color information in corresponding regions of at least one other anchor view.

20. The method of claim 19, further comprising computing coordinate information for occluded regions of the given anchor view by interpolating between neighboring non-occluded regions of the given anchor view.

21. A method of multiframe image processing, comprising:

computing correspondence mappings between two or more pairs of anchor views of a scene;

presenting to a user a graphical user interface comprising an N-dimensional space of synthesizable views parameterized based on the computed correspondence mappings and comprising an interface shape representing relative locations of the anchor views, wherein N is an integer greater than 0; and

generating a synthetic view of the scene by interpolating between anchor views based on the computed correspondence mappings with anchor view contributions to the synthetic view weighted based on a location in the graphical user interface selected by the user.

22. The method of claim 21, wherein vertices of the interface shape are computed based on correspondence differences computed for successive anchor view pairs in an ordered sequence of anchor views.

23. The method of claim 22, wherein vertices of the interface shape correspond to medians of correspondence coordinate differences.

24. The method of claim 21, wherein further comprising identifying a set of three vertices of the interface shape defining an interface triangle closest to the user selected location in the graphical user interface.

25. The method of claim 24, wherein the user selected location is circumscribed by the interface triangle.

26. The method of claim 24, wherein the user selected location is outside the interface triangle.

27. The method of claim 24, wherein the user selected location is along a boundary of the interface triangle.

28. The method of claim 24, wherein contributions of anchor views to the synthetic view are weighted in accordance with Barycentric coordinates of the user selected location defined relative to the vertices of the interface triangle.

29. The method of claim 24, wherein vertices of the interface triangle correspond to respective anchor views of the scene.

30. A method of multiframe image processing, comprising:

projecting onto a scene a sequence of patterns of light symbols that temporally encode two-dimensional position information in a projection plane with unique light symbol sequence codes;

capturing light patterns reflected from the scene at a capture plane of an image sensor;

computing a correspondence mapping between the capture plane and the projection plane based at least in part on correspondence between light symbol sequence codes captured at the capture plane and light symbol sequence codes projected from the projection plane; and

computing calibration parameters for the image sensors based at least in part on the computed correspondence mapping.

31. The method of claim 30, wherein computing calibration parameters comprises computing intrinsic image sensor parameters.

32. The method of claim 31, wherein computing intrinsic image sensor parameters comprises computing focal length, aspect ratio, skew, and radial lens distortion parameters for the image sensor.

33. The method of claim 30, wherein computing calibration parameters comprises computing extrinsic image sensor parameters.

34. The method of claim 33, wherein computing extrinsic image sensor parameters comprises computing relative position and orientation parameters for the image sensors with respect to some common reference coordinate system.

35. The method of claim 30, further comprising computing calibration parameters for a light source projecting the light symbol patterns.

36. The method of claim 35, wherein computing light source calibration parameters comprises computing focal length, aspect ratio, skew, and radial lens distortion parameters for the projector.

37. The method of claim 30, wherein calibration parameters are computed based at least in part on a correspondence mapping computed for a known scene.

38. The method of claim 37, wherein the known scene has a blank planar surface oriented to receive the projected sequence of light patterns.

39. The method of claim 38, wherein the blank planar surface is colored with a uniform non-dark color.

40. The method of claim 37, wherein the known scene includes an object of interest positioned between the projection plane and the planar surface.

41. The method of claim 40, wherein computing calibration parameters comprises identifying regions in the capture plane corresponding to regions of the planar surface.

42. The method of claim 30, wherein among the projected light symbol patterns are light patterns respectively comprising different spatial variations of light in the projection plane.

43. The method of claim 42, wherein each light symbol pattern comprises a binary pattern of light and dark rectangular stripe symbols.

44. The method of claim 43, wherein each light symbol pattern includes light and dark stripe symbols of substantially equal size in the projection plane, and light and dark stripe symbols of different light symbol patterns are of substantially different size in the projection plane.

45. The method of claim 43, wherein a first subset of light symbol patterns encodes rows of a reference grid in the projection plane, and a second subset of light symbol patterns encodes columns of the reference grid in the projection plane.

46. The method of claim 42, wherein each light pattern comprises a multicolor pattern of light.

47. The method of claim 30, further comprising storing the computed correspondence mapping in a data structure including an array of points defined in the projection plane and linked to respective corresponding points in the capture plane.

48. The method of claim 30, further comprising capturing patterns reflected from the scene at one or more additional capture planes of one or more respective image sensors, and computing a correspondence mapping between each capture plane and the projection plane based at least in part on correspondence between light symbol sequence codes captured at each capture plane and light symbol sequence codes projected from the projection plane.

49. The method of claim 30, further comprising computing a three-dimensional coordinate system based on the computed calibration parameters and the computed correspondence mapping.

50. The method of claim 49, wherein computing calibration parameters comprises assigning real world coordinates for points in the scene.

51. The method of claim 50, further comprising back-projecting points in the scene into the computed three-dimensional coordinate system, and modifying the assigned world coordinates based at least in part on a comparison between the back-projected points and the assigned world coordinates.

52. A system for multiframe image processing, comprising:

a light source operable to project onto a scene a sequence of patterns of light symbols that temporally encode two-dimensional position information in a projection plane with unique light symbol sequence codes;

at least one imaging device operable to capture light patterns reflected from the scene at a respective capture plane;

processing system operable to compute a correspondence mapping between the capture plane and the projection plane based at least in part on correspondence between light symbol sequence codes captured at the capture plane and light symbol sequence codes projected from the projection plane, and to compute calibration parameters for the image sensor based at least in part on the computed correspondence mapping.

53. The system of claim 52, further comprising a turntable operable to rotate an object in the scene about an axis.

54. A method of multiframe image processing, comprising:

(a) projecting onto an object a sequence of patterns of light symbols that temporally encode two-dimensional position information in a projection plane with unique light symbol sequence codes;

(b) capturing light patterns reflected from the object at a pair of capture planes with optical axes separated by an angle θ

;

(c) computing a correspondence mapping between the pair of capture planes based at least in part on correspondence between light symbol sequence codes captured at the capture planes and light symbol sequence codes projected from the projection plane;

(d) rotating the object through an angle θ

; and

(e) repeating steps (a)-(d) until the object has been rotated through a prescribed angle.

55. The method of claim 54, further comprising interpolating between pairs of anchor views based on the corresponding computed correspondence mappings to generate one or more synthetic views of the object.