Image processing method and apparatus

• US 6,496,598 B1
• Filed: 03/01/2000
• Issued: 12/17/2002
• Est. Priority Date: 09/02/1997
• Status: Expired due to Term

• Current Assignee: dynamic digital depth Research Pty Ltd
• Original Assignee: dynamic digital depth Research Pty Ltd
• Inventors: Harman, Philip Victor
• Primary Examiner(s): Mancuso, Joseph
• Assistant Examiner(s): BALI, VIKKRAM
• Application Number:
  US09/516,326
• Time in Patent Office: 1,021 Days
• Field of Search: 382/100, 382/107, 382/154, 382/232, 382/236, 382/103, 348/42, 348/47, 348/51, 348/413, 345/419, 345/433, 345/473, 359/462, 359/464
• US Class Current: 382/154
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 13/122 Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues H04N13/128 takes precedence
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/144 for flicker reduction
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/261 with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/305 using lenticular lenses, e.g. arrangements of cylindrical lenses
• CPC Class Codes: H04N 13/334 using spectral multiplexing
• CPC Class Codes: H04N 13/337 using polarisation multiplexing
• CPC Class Codes: H04N 13/341 using temporal multiplexing
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/363 using image projection screens volumetric display H04N13/388
• CPC Class Codes: H04N 13/388 Volumetric displays, i.e. systems where the image is built up from picture elements distributed through a volume
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0085 Motion estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals
• CPC Class Codes: H04N 2213/002 : Eyestrain reduction by processing stereoscopic signals or controlling stereoscopic devices

Abstract

An image conversion system for converting monoscopic images for viewing in three dimensions including: an input means adapted to receive the monoscopic images; a preliminary analysis means to determine if there is any continuity between a first image and a second image of the monoscopic image sequence; a secondary analysis means for receiving monoscopic images which have a continuity, and analyzing the images to determine the speed and direction of motion, and the depth, size and position of objects; a first processing means for processing the monoscopic images based on data received from the preliminary analysis means or the secondary analysis means; a second processing means capable of further processing images received from the first processing means; a transmission means capable of transferring the processed images to a stereoscopic display system.

Claims

1. A method for converting monoscopic images for viewing in three dimensions comprising:

receiving monoscopic images;

analyzing said monoscopic images to determine characteristics of the images and to determine if there is any continuity between successive first and second of said monoscopic images;

processing said monoscopic images based on the determined image characteristics and/or the determination if there is any continuity between the first and second monoscopic images; and

outputting the processed images to suitable storage and/or stereoscopic display systems,

wherein said analyzing further includes determining the motion of said monoscopic images by;

dividing each image into a plurality of blocks, wherein corresponding blocks on an adjacent image are offset horizontally and/or vertically; and

comparing each said block with said corresponding blocks to find a minimum mean square error and thereby the motion of the block.

2. The method as claimed in claim 1, wherein said processing includes at least one of the following methods:

motion parallax, forced parallax, parallax zones, image rotation and/or object processing.

3. The method as claimed in claim 1, wherein said monoscopic image is digitized before any said analyzing or said processing is performed.

4. The method as claimed in claim 1, further comprising compressing said monoscopic image prior to said analyzing.

5. The method as claimed in claim 1, further comprising scaling said monoscopic image prior to said analyzing.

6. The method as claimed in claim 5, wherein said scaling includes scaling said monoscopic image by a scaling factor that depends on a digital video resolution of each image.

7. The method as claimed in claim 1, wherein said analyzing includes analyzing successive first and second images for continuity before determining the image characteristics.

8. The method as claimed in claim 7, where said analyzing successive first and second images includes comparing median luminance values between the successive first and second images to determine continuity.

9. The method as claimed in claim 8, wherein no continuity is assumed when a difference in the median luminance values exceeds 30.

10. The method as claimed in claim 7, wherein said analyzing successive first and second images includes comparing the top few lines of the successive images to assist in determining continuity.

11. The method as claimed in claim 7, wherein where no continuity is determined said processing includes introducing a field delay to one eye such that the image that lacks continuity is viewed by one eye of a viewer prior to being viewed by the other eye of the viewer.

12. The method as claimed in claim 1, wherein said analyzing includes identifying objects within the monoscopic images to assist said processing.

13. The method as claimed in claim 12, wherein said identifying includes comparing luminance values, chrominance values, motion vectors and/or picture details of adjacent pixels or groups of pixels.

14. The method as claimed in claim 1, wherein said analyzing includes determining the motion of objects within the monoscopic images to assist said processing.

15. The method as claimed in claim 14, wherein the motion of the monoscopic images and/or said objects is categorized into one of a predetermined range of motion categories.

16. The method as claimed in claim 14, wherein a motion vector is defined for each image based on a comparison of the motion detected with predefined motion categories ranging from no motion to a complete scene change.

17. The method as claimed in claim 1, wherein the motion categories include scene change, simple pan, complex pan, moving object, moving background, and no motion.

18. The method as claimed in claim 1, wherein any said block without details is not compared with said corresponding blocks.

19. The method as claimed in claim 7, wherein no continuity is assumed when said comparing of the majority of blocks with said corresponding blocks results in large error values.

20. The method as claimed in claim 1, wherein said processing of each image includes using motion parallax by introducing a field delay such that one eye of a viewer views the image before the other eye of the viewer.

21. The method as claimed in claim 20, wherein the amount of motion is inversely proportional to the field delay.

22. The method as claimed in claim 20, further including storing each field delay, and averaging the field delay for each new image with previous field delays.

23. The method as claimed in claim 22, further including deleting each stored field delay when a non-continuity is detected.

24. The method as claimed in claim 1, wherein said processing of each image includes using forced parallax by introducing a lateral shift through displacement of left and right eye images.

25. The method as claimed in claim 1, wherein said processing of each image includes using parallax zones by introducing a greater lateral shift to one portion of the image.

26. The method as claimed in claim 25, wherein a top portion of the image is shifted laterally a greater amount than a bottom portion of the image.

27. The method as claimed in claim 25, further including applying a different parallax shift to a left side of the image as opposed to a right side of the image.

28. The method as claimed in claim 1, wherein said processing of each image includes using a combination of forced parallax and motion parallax on various parts of the image.

29. The method as claimed in claim 1, wherein said processing of each image includes rotating left and right eye images about the y axis an equal amount in an opposite direction.

30. The method as claimed in claim 1, wherein said processing of each image includes using at least one of the following object processing techniques:

mesh distortion and morphing;

object barrelling;

object edge enhancement;

object brightness enhancement; and

/or object rotation.

31. The method as claimed in claim 1, further including processing each processed image by applying a forced parallax to the processed image.

32. The method as claimed in claim 31, wherein the degree of forced parallax is determined by the amount of parallax added during said processing of each image, such that the total parallax added during said processing of each processed image and the forced parallax, is substantially equal to the total parallax of adjacent images.

33. The method as claimed in claim 31, wherein the degree of the forced parallax is modulated between predetermined minimum and maximum settings over a predetermined time frame.

34. The method as claimed in claim 1, further including optimizing to further enhance the processed images prior to said outputting of the processed images to the stereoscopic display and/or storage systems.

35. The method as claimed in claim 1, further including adding a reference point to the processed image.

36. The method as claimed in claim 35, wherein the reference point is at least one of:

a border around the perimeter of the processed image;

a plurality of concentric borders;

a partial border;

a logo; and

/or

a picture.

37. The method as claimed in claim 1, wherein said processing includes adding an amount of depth to the monoscopic images, the amount being adjustable in response to a viewer'"'"'s preference.

38. The method as claimed in claim 1, wherein said processing includes randomly moving the background of each image in increments that are not consciously detectable by a viewer.

39. The method as claimed in claim 1, further including testing each image for reverse 3D and manipulating objects individually in each image to compensate for any reverse 3D found during said testing.

40. The method as claimed in claim 1, further including cut and paste techniques to further emphasize a stereoscopic effect.

41. An image conversion system for converting monoscopic images for viewing in three dimensions including:

an input means adapted to receive monoscopic images;

a preliminary analysis means to determine if there is any continuity between a first image and a second image in a sequence of the monoscopic images;

a secondary analysis means for receiving the monoscopic images which have a continuity, and analyzing the monoscopic images to determine at least one of the speed and direction of motion, or the depth, size and position of objects, wherein analyzing of said monoscopic images to determine the motion includes;

dividing each image into a plurality of blocks, wherein corresponding blocks on an adjacent image are offset horizontally and/or vertically, and

comparing each said block with said corresponding blocks to find a minimum mean square error and thereby the motion of the block;

a first processing means for processing the monoscopic images based on data received from the preliminary analysis means and/or the secondary analysis means.

42. The image conversion system as claimed in claim 41 further including a transmission means adapted to transfer the processed images to a stereoscopic display system or a storage system.

43. The image conversion system as claimed in claim 41, wherein said first processing means processes the monoscopic images by using at least one of motion parallax, forced parallax, parallax zones, image rotation or object processing.

44. The image conversion system as claimed in claim 41, further including a second processing means adapted to process the images received from said first processing means.

45. The image conversion system as claimed in claim 44, wherein said second processing means uses forced parallax to process each image.

46. The image conversion system as claimed in claim 41, further including a second processing means adapted to optionally enhance the images prior to transmitting the images to a stereoscopic display device.

47. The image conversion system as claimed in claim 46, wherein said second processing means enhances the images by using at least one of reference points, parallax adjustment, parallax smoothing, parallax modulation, movement synthesis, reverse 3D correction or cut and paste techniques.

48. The system as claimed in claim 41, wherein said input means is further adapted to digitize the monoscopic images.

49. The system as claimed in claim 41, further including a compression means adapted to compress the monoscopic images prior to analysis by said preliminary analysis means.

50. The system as claimed in claim 41, further including a scaling means adapted to scale each monoscopic image prior to analysis by said preliminary analysis means.

51. The system as claimed in claim 50, wherein the scaling factor by which each monoscopic image is scaled depends on a digital video resolution of each monoscopic image.

52. The system as claimed in claim 41, wherein said preliminary analysis means is adapted to determine objects within said monoscopic images.

53. The system as claimed in claim 41, wherein said preliminary analysis means is adapted to determine the motion of the monoscopic images and/or the motion of objects within the monoscopic images.

54. The system as claimed in claim 53, wherein said preliminary analysis means is adapted to categorize the motion into one of a predetermined range of motion categories.

55. The system as claimed in claim 54, wherein the motion categories include at least one of scene change, simple pan, complex pan, moving object, moving background, and no motion.

56. The system as claimed in claim 41, further including means adapted to control the level of depth added to the monoscopic images.

57. The system as claimed in claim 41, further including means adapted to add a reference point to each processed image.

58. The system as claimed in claim 41, further including means for optimizing the processed image to further improve a stereoscopic effect.

59. A system for converting monoscopic images for viewing in three dimensions including:

a first module adapted to receive a monoscopic image;

a second module adapted to receive the monoscopic image and analyze the monoscopic image to create image data, wherein analyzing the monoscopic image includes determining the motion of a plurality of monoscopic images by;

dividing each monoscopic image into a plurality of blocks, wherein corresponding blocks on an adjacent image are offset horizontally and/or vertically, and

comparing each said block with said corresponding blocks to find a minimum mean square error and thereby the motion of the block;

a third module adapted to create stereoscopic image pairs from the monoscopic image using at least one predetermined technique selected as a function of the image data;

a fourth module adapted to transfer the stereoscopic image pairs; and

a fifth module including a stereoscopic display means adapted to receive the stereoscopic pairs transferred by said fourth module.

60. The system as claimed in claim 59, wherein said first module is further adapted to convert an analog monoscopic image into a digital monoscopic image.

61. The system as claimed in claim 59, wherein said second module is adapted to detect objects in a scene and to determine to the speed and direction of motion of the detected objects.

62. The system as claimed in claim 59, wherein the monoscopic image is compressed prior to said second module analyzing the monoscopic image to create image data.

63. The system as claimed in claim 59, wherein the third module further includes an optimization stage to further enhance the stereoscopic image pairs.

64. The system as claimed in claim 59, wherein operation of said system is adapted to be suspended for later processing by any of said first through fifth modules.

65. The method as claimed in claim 59, wherein the fourth module is adapted store the stereoscopic image pairs and transfer the stereoscopic image pairs at a later time.