Method and apparatus for visualization and manipulation of real 3-D objects in networked environments

• US 6,983,064 B2
• Filed: 06/01/2001
• Issued: 01/03/2006
• Est. Priority Date: 04/19/2001
• Status: Expired due to Fees

• Current Assignee: Openvisual, Inc.
• Original Assignee: Openvisual, Inc.
• Inventors: Song, Samuel Moon-Ho
• Primary Examiner(s): Patel, Kanjibhai
• Assistant Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US09/873,056
• Publication Number:
  US 20030031380A1
• Time in Patent Office: 1,677 Days
• Field of Search: 382/128, 382/129, 382/130, 382/131, 382/132, 382/133, 382/232, 382/240, 382/154, 382/114, 345/9, 345/418, 345/419, 348/42, 348/44, 600/437, 600/558
• US Class Current: 382/131
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: G06T 2200/16 involving adaptation to the client's capabilities
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/234381 by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
• CPC Class Codes: H04N 21/4143 embedded in a Personal Computer [PC]
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 7/17336 Handling of requests in head-ends

Abstract

A new method and apparatus for visualization and manipulation of real 3-D objects are disclosed. This new approach utilizes compressed digital video taking advantage of modern video compression technology. The entire system consists of the image acquisition device, the viewing/manipulation program and the 3-D image database. The system first captures images of the object from all possible view angles as a sequence of images (a video). As neighboring images within the sequence will generally be similar, the sequence will be amenable to standard video compression techniques such as MPEG. The acquired video data (rendition of a real 3-D object) is then stored into a database for later access, for instance, over the Internet.

Through specially developed software, the stored data can be manipulated while viewing the object. The user simply inputs—via a mouse, for instance—the view angle and the software automatically displays the appropriate frame within the video (stored data). The software allows various object manipulations such as rotation in arbitrary direction, as well as zoom, pan, etc. A particular implementation of the viewing program using a Pentium III processor, randomly accessing an MPEG sequence, through pre-decoding of certain anchor frames, is able to decode fast enough for users to perform real-time manipulation of the object.

The captured 3-D image data of an object is stored into a centralized (or distributed) database and users may access the data through the assigned URL for the object over the Internet. The data may or may not be password protected. As such, the overall system is an implementation of a one-stop service for individuals wishing to provide 3-D visualization of real 3-D objects over the Internet.

Claims

1. A method of visualizing a desired view of an object by a user using a computer, comprising the steps of:

capturing the images of the object as a sequence of images according to an image capturing sequence;

encoding the sequence of images as video having video frames, each video frame tagged with viewing angle information of the corresponding image; and

visualizing the object by displaying the frame substantially corresponding to the user'"'"'s desired view angle.

2. The method of claim 1, wherein the step of visualization includes the step of pre-decoding the video for real-time display.

3. The method of visualizing a desired view of an object by a user using a computer, comprising the steps of:

capturing images of the object as a sequence of images according to an image capturing sequence;

encoding the sequence of images as video having video frames, each video frame tagged with viewing angle information of the corresponding images; and

visualizing the object by displaying the frame substantially corresponding to the user'"'"'s desired view angle,

wherein the step of capturing is performed by taking samples of the taking samples of the images of the object from a viewing angle of an azimuth angle θ and

an elevation angle ø

in the spherical coordinates.

4. The method of claim 3, wherein the video frames are tagged with θ

and ø

.

5. The method of claim 3, wherein the step of taking samples uses a slicing sampling technique where a series of samples are taken with θ

updated in constant steps and with ø

fixed before another series of samples are taken with the updated ø

.

6. The method of claim 3, wherein the step of taking samples uses an anisotrophic spiral sampling technique, where a series of samples are taken with θ

and ø

changed in constant steps.

7. The method of claim 3, wherein the step of taking samples uses an isotrophic spiral sampling technique, where a series of samples are with θ

and ø

changed so that the distance between two adjacent samples is substantially equal.

8. The method of claim 3, wherein the video is streamed so that the visualization can start as soon as one or more frames have been received.

9. The method of claim 3, wherein the step of encoding includes the step of compressing the video.

10. The method of claim 9, wherein the step of compressing uses MPEG.

11. The method of claim 9, wherein the step of compressing uses H.261.

12. The method of claim 9, wherein the step of compressing uses H.263.

13. The method of claim 9, wherein the step of compressing uses H.263+.

14. The method of claim 3, wherein the step of visualization includes the step of pre-decoding the video for real-time display.

15. The method of claim 3, wherein the step of visualizing the object includes the step of streaming the most important frames first.

16. The method of claim 3, further including the step of editing the captured images before encoding.

17. The method of claim 3, further comprising the step of manipulating the viewing angle of the object while visualizing the object.

18. The method of claim 17, wherein the step of manipulating includes the step of rotating the object.

19. The method of claim 17, wherein the step of manipulating includes the step of zooming the object.

20. The method of claim 17, wherein the step of manipulating includes the step of manipulating the object in the NEWS (north-east-west-south) mode, where the object is rotated along the east-west direction or along the north-south direction.

21. The method of claim 17, wherein the step of manipulating includes the step of manipulating the object in the AVIATION mode, where the object is rotated along the direction of the two orthogonal great circle of a sphere having the object at the center.

22. The method of claim 3, further comprising the step of storing the image sequence in a database.

23. The method of claim 22, wherein the database is a centralized database.

24. The method of claim 22, wherein the database is a distributed database.

25. The method of claim 22, wherein the database is accessed through a network.

26. The method of claim 25, wherein the network is the Internet.

27. The method of claim 26, further comprising the step of publishing the URL corresponding to the location of the image sequence within the database.

28. The method of claim 27, wherein the URL is password protected.

29. The method of claim 3, wherein the step of visualization including the step of downloading a separate multimedia stream.

30. The method of claim 29, wherein the separate multimedia stream includes zoom data requested by the user.

31. The method of claim 29, wherein the separate multimedia stream includes other related data requested by the user.

32. The method of claim 31, wherein the related data is video.

33. The method of claim 31, wherein the related data is still images.

34. The method of claim 31, wherein the related data is sound.

35. The method of claim 31, wherein the related data is coordinates of points on the object.

36. A system of visualizing a desired view of an object by a user using a computer, comprising the steps of:

an image capture device for capturing images of the object as a sequence of images;

an encoder for encoding the sequence of images as video having video frames,

each video frame tagged with viewing angle information of the corresponding image; and

a visualizer for visualizing the object in 3-D by displaying the frame substantially corresponding to the user'"'"'s desired view angle.

37. The system of claim 36, wherein the image capture device includes a camera.

38. The system of claim 37, further comprising a database for the image sequence in a database.

39. The system of claim 38, wherein the database is a centralized database.

40. The system of claim 38, wherein the database is a distributed database.

41. The system of claim 38, further comprising a network for access the database.

42. The system of claim 41, wherein the network is the Internet.

43. A system of visualizing a desired view of an object by a user using a computer, comprising the steps of:

an image capture device for capturing images of the object as a sequence of images;

an encoder for encoding the sequence of images as video having video frames, each video frame tagged with viewing angle information of the corresponding image; and

a visualizer for visualizing the object in 3-D by displaying the frame substantially corresponding to the user'"'"'s desired view angle,

wherein the image capture device has means for taking samples of the images of the object from a viewing angle of an azimuth angle θ

in the horizontal plane and an elevation angle ø

from the horizontal plane.

44. The system of claim 43, wherein the video frames are tagged with θ

and ø

.

45. The system of claim 43, wherein the video is streamed so that the visualization can start as soon as one or more frames have been received.

46. The system of claim 43, wherein the encoder includes a video compressor for compressing the video.

47. The system of claim 46, wherein the video compressor is an MPEG encoder.

48. The system of claim 46, wherein the video compressor is an H.261 encoder.

49. The system of claim 46, wherein the video compressor is an H.263 encoder.

50. The system of claim 43, wherein the visualizer includes a look-up-table (LUT) for mapping the view angle to a frame number.

51. The system of claim 43, wherein the visualizer includes a pre-decoder for pre-decoding the video for fast retrieval.

52. The system of claim 43, further including an editor for editing the captured images after encoding.

53. The system of claim 43, wherein the visualizer includes means for manipulating the object while visualizing.

54. A method for enabling a user to visualize a desired view of an object, comprising the steps of:

capturing the images of the object as a sequence of images according to a particular image capturing sequence;

encoding the sequence of images as video having video frames, each frame tagged with viewing information of the corresponding image;

storing the video in a database; and

sending the video at the request of the user over a network.

55. The method of claim 54, wherein the video is encoded using MPEG.

56. The method of claim 54, wherein the network is the Internet.

57. The method of claim 54, further comprising the steps of:

receiving the video by a user; and

visualizing the object by displaying the frame substantially corresponding to the user'"'"'s desired view angle.

58. A method of visualizing 3-D volume data of an object, comprising the steps of:

capturing the 3D volume data as a stack of 2D tomographic images of the object;

obtaining a sequence of projected 2D images by projecting the 3D volume data at prescribed azimuth and elevation angles;

encoding the sequence of projected 2D images wherein each images are tagged with azimuth and elevation angles;

storing the sequence of 2D images; and

viewing the object in 3-D by random access of stored 2D image sequence.

59. The method of claim 58 wherein the step of capturing the 3D volume data uses positron emission tomography (PET).

60. The method of claim 58, wherein the step of capturing the 3D volume date uses single positron emission computed tomography (SPECT).

61. The method of claim 58, wherein the step of capturing the 3D volume data uses X-ray computed tomography (CT).

62. The method of claim 58, wherein the step of capturing the 3D volume data uses ultrasound imaging.

63. The method of claim 58, wherein the step of capturing the 3D volume data uses magnetic resonance imaging (MRI).

64. The method of claim 58 wherein the step of encoding includes video encoding using MPEG.

65. The method of claim 58, wherein the object includes human bodies.

66. The method of claim 58, wherein the object includes biological tissues.

67. The method of claim 58, wherein the object includes organic materials.

68. The method of claim 58, wherein the object includes inorganic materials.

69. The method of claim 58, further comprising the step of manipulating the object.

70. The method of claim 69, wherein the step of manipulating includes the step of rotating the object.

71. A method of visualizing 3-D volume data of an object, comprising the steps of:

obtaining a sequence of 2D images at prescribed azimuth and elevation angles rendered by a CAD program;

encoding the sequence of 2D images where each images are tagged with azimuth and elevation angles;

storing the sequence of 2D images; and

viewing the object in 3-D by random access of stored 2D image sequence.

72. The method of claim 71, wherein the step of encoding includes video encoding using MPEG.

73. The method of claim 71, further including the step of manipulating the object.

74. The method of claim 73, wherein the step of manipulating includes the step of rotating the object.

75. A method of providing a customer with a one-stop service for 3-D visualization of a real object over the Internet, comprising the steps of:

receiving the object from the customer;

capturing images of the object as a sequence of images each images tagged with an azimuth angle θ and

an elevation angle ø

in the spherical coordinates; and

storing the image sequence in a database for transmission over the Internet at the request of a user.

76. The method of claim 75, wherein the image sequence is assigned a URL for a user of the Internet to access the image sequence.

77. The method of claim 76, wherein the accessed image sequence is used to view the object in 3-D by the user.

78. The method of claim 76, wherein the accessed image sequence is used to manipulate the object in 3-D by the user.

79. The method of claim 75, wherein the image sequence is encoded as video.

80. The method of claim 79, wherein the video encoding uses MPEG.