Method and apparatus for electronically distributing motion panoramic images

• US 6,466,254 B1
• Filed: 06/07/2000
• Issued: 10/15/2002
• Est. Priority Date: 05/08/1997
• Status: Expired due to Term

• Current Assignee: Cedar Lane Technologies Inc.
• Original Assignee: Be Here Corporation
• Inventors: Furlan, John L. Warpakowski, Fluker, Derek
• Primary Examiner(s): RAO, ANAND SHASHIKANT
• Application Number:
  US09/589,645
• Time in Patent Office: 860 Days
• Field of Search: 348/36-60
• US Class Current: 348/36
• CPC Class Codes: H04N 19/103 Selection of coding mode or of prediction mode
• CPC Class Codes: H04N 19/162 User input
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/234345 the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• CPC Class Codes: H04N 21/6587 Control parameters, e.g. trick play commands, viewpoint selection
• CPC Class Codes: H04N 21/816 involving special video data, e.g 3D video
• CPC Class Codes: H04N 23/58 Means for changing the camera field of view without moving the camera body, e.g. nutating or panning of optics or image sensors
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

This patent discloses an electronic image distribution apparatus for distributing motion panoramic images. The main portion of the apparatus is a motion panoramic image server that has a source of motion panoramic images. The source of panoramic image frames may be a motion panoramic image playback device or a connection to a motion panoramic camera system. The motion panoramic image server transforms the panoramic image frames into an intermediate format that is more conducive for electronic transmission. In particular, portions of the panoramic image are encoded at a higher quality as compared to other portions of the panoramic image. A client computer obtains information about the encoding and subscribes to one or more tiles depending on the viewpoint of interest. The client computer receives and presents the data in these tiles. In addition, the patent discloses how to adjust the subscription based on the actual and/or predicted pan operations to obtain a quality presentation.

Claims

1. A computer controlled method for presenting a view into a motion panoramic image, said method including steps of:

obtaining availability information related to said motion panoramic image, said availability information including tier information;

determining a viewpoint into said motion panoramic image, said viewpoint defining said view;

receiving a frame of said motion panoramic image;

simultaneously encoding a plurality of portions of said frame into a plurality of encoded portions in accordance with said availability information;

providing said plurality of encoded portions responsive to a selection of said availability information; and

presenting said view responsive to said data.

2. A computer controlled method for presenting a view into a motion panoramic image, said method including steps of:

obtaining availability information related to said motion panoramic image, said availability information including tier information;

determining an available bandwidth for accessing one or more tiles;

determining a viewpoint into said motion panoramic image, said viewpoint defining said view;

accessing said one or more tiles, responsive to said tier information, said viewpoint and said available bandwidth, said one or more tiles containing data used to present said view; and

presenting said view responsive to said data.

3. The computer controlled method of claim 2 wherein each of said one or more tiles is encoded at a particular quality for that tile.

4. The computer controlled method of claim 2 wherein each of said one or more tiles is encoded using a distributed quality encoding for that tile.

5. The computer controlled method of claim 2 wherein the step of accessing further includes steps of:

subscribing to said one or more tiles responsive to said viewpoint and said availability information to establish a tile subscription; and

receiving said one or more tiles responsive to said tile subscription.

6. The computer controlled method of claim 5 further including steps of:

panning said viewpoint;

detecting that the step of panning was a significant pan of said viewpoint; and

changing said tile subscription.

7. The computer controlled method of claim 5 further including steps of:

detecting that said viewpoint is stable; and

changing said tile subscription responsive to the step of detecting.

8. The computer controlled method of claim 5 further including steps of:

detecting that said viewpoint is substantially constant over a specified period; and

changing said tile subscription to increase a quality of said view as presented.

9. The computer controlled method of claim 8 wherein said quality is selected from the set consisting of presented resolution, field of view, view size, frame rate, compression, and color depth.

10. The computer controlled method of claim 2 wherein said tier information includes tile information.

11. The computer controlled method of claim 2 wherein said tier information includes compression information.

12. The computer controlled method of claim 5 wherein the step of obtaining said availability information further includes accessing a server computer.

13. The computer controlled method of claim 12 wherein said server computer performs steps of:

sending said availability information;

receiving said tile subscription; and

sending said one or more tiles responsive to said tile subscription.

14. The computer controlled method of claim 2 wherein the step of obtaining includes accessing information from a computer readable media.

15. A computer controlled method for encoding a motion panoramic image, said method including steps of:

receiving a frame of said motion panoramic image;

simultaneously encoding a plurality of portions of said frame into a plurality of encoded portions in accordance with availability information; and

providing said plurality of encoded portions responsive to a selection of said availability information.

16. The computer controlled method of claim 15 wherein said availability information includes tier information.

17. The computer controlled method of claim 15 further comprising storing said plurality of encoded portions in a memory in accordance with said availability information.

18. The computer controlled method of claim 15 wherein said availability information includes tile information.

19. The computer controlled method of claim 15 further including a step of repeating the steps of receiving, of simultaneously encoding and of providing.

20. The computer controlled method of claim 15 further including a step of sending one or more of said plurality of encoded portions to a server computer.

21. The computer controlled method of claim 15 wherein said availability information represents a data structure in memory having a plurality of tiers.

22. The computer controlled method of claim 21 wherein each of said plurality of tiers includes one or more tiles, each of said one or more tiles referencing data encoded respective to a quality.

23. The computer controlled method of claim 22 wherein said quality is selected from the set consisting of presented resolution, field of view, view size, frame rate, compression, and color depth.

24. The computer controlled method of claim 22 wherein at least one of said plurality of tiers is respectively associated with a tier quality.

25. The computer controlled method of claim 22 wherein at least one of said one or more tiles is respectively associated with a tile quality.

26. The computer controlled method of claim 15 wherein the step of simultaneously encoding is performed by an encoder array.

27. The computer controlled method of claim 15 wherein said plurality of encoded portions is contained within one or more tiles, each of said one or more tiles including data from a portion of an arc covering at least 270-degrees of said motion panoramic image.

28. The computer controlled method of claim 15 wherein each of said plurality of encoded portions is included with one of said one or more tiles respectively.

29. An apparatus having a central processing unit (CPU) and a memory coupled to said CPU for presenting a view into a motion panoramic image, said apparatus includes:

a first access mechanism configured to obtain availability information related to said motion panoramic image, said availability information including tier information;

a bandwidth determination mechanism configured to determine the available bandwidth for accessing one or more tiles;

a viewpoint determination mechanism configured to determine a viewpoint into said motion panoramic image, said viewpoint defining said view;

a second access mechanism configured to access said one or more tiles, responsive to the viewpoint determination mechanism, the bandwidth determination mechanism and said tier information, said one or more tiles containing data used to present said view; and

a presentation mechanism configured to present said view responsive to said data accessed by the second access mechanism.

30. The apparatus of claim 29 wherein each of said one or more tiles is encoded at a particular quality for that tile.

31. The apparatus of claim 29 wherein each of said one or more tiles is encoded using a distributed quality encoding for that tile.

32. The apparatus of claim 29 wherein the second access mechanism further includes:

a subscription mechanism configured to subscribe to said one or more tiles responsive to said viewpoint and said availability information to establish a tile subscription; and

a receiver mechanism configured to receive said one or more tiles responsive to said tile subscription determined by the subscription mechanism.

33. The apparatus of claim 32 further including:

a pan mechanism configured to pan said viewpoint;

a detection mechanism configured to detect that the pan mechanism performed a significant pan of said viewpoint responsive to the pan mechanism; and

a subscription modification mechanism configured to change said tile subscription responsive to the detection mechanism.

34. The apparatus of claim 32 further including:

a stability determination mechanism configured to detect that said viewpoint is stable; and

a subscription modification mechanism configured to change said tile subscription responsive to the stability determination mechanism.

35. The apparatus of claim 32 further including:

a detection mechanism configured to detect that said viewpoint is substantially constant over a specified period; and

a subscription modification mechanism configured to change said tile subscription to increase a quality of said view as presented, the subscription modification mechanism responsive to the detection mechanism.

36. The apparatus of claim 35 wherein said quality is selected from the set consisting of presented resolution, field of view, view size, frame rate, compression, and color depth.

37. The apparatus of claim 29 wherein said tier information includes tile information.

38. The apparatus of claim 29 wherein said tier information includes compression information.

39. The apparatus of claim 32 wherein the first access mechanism is further configured to access a server computer.

40. he apparatus of claim 39 wherein said server computer includes:

a first sending mechanism configured to send said availability information;

a subscription receiver mechanism configured to receive said tile subscription; and

a second sending mechanism configured to send said one or more tiles responsive to said tile subscription.

41. The apparatus of claim 29 wherein the first access mechanism includes a media reader device configured to access information from a computer readable media.

42. An apparatus having a central processing unit (CPU) and a memory coupled to said CPU for encoding a motion panoramic image, said apparatus includes:

a frame receiver configured to receive a frame of said motion panoramic image;

an encoder array configured to simultaneously encode a plurality of portions of said frame received from the frame receiver into a plurality of encoded portions in accordance with availability information; and

a distribution mechanism configured to provide said plurality of encoded portions responsive to a selection of said availability information.

43. The apparatus of claim 42 wherein said availability information includes tier information.

44. The apparatus of claim 42 further comprising a storage mechanism configured to store said plurality of encoded portions in said memory in accordance with said availability information.

45. The apparatus of claim 42 wherein said availability information includes tile information.

46. The apparatus of claim 42 further including a repetition mechanism configured to repeatedly invoke the frame receiver, the encoder array and the distribution mechanism.

47. The apparatus of claim 42 wherein the encoder array further includes a communication mechanism configured to send one or more of said plurality of encoded portions to a server computer.

48. The apparatus of claim 42 wherein said availability information represents a data structure in memory having a plurality of tiers.

49. The apparatus of claim 48 wherein each of said plurality of tiers includes one or more tiles, each of said one or more tiles referencing data encoded respective to a quality.

50. The apparatus of claim 49 wherein said quality is selected from the set consisting of presented resolution, field of view, view size, frame rate, compression, and color depth.

51. The apparatus of claim 49 wherein at least one of said plurality of tiers is respectively associated with a tier quality.

52. The apparatus of claim 49 wherein at least one of said one or more tiles is respectively associated with a tile quality.

53. The apparatus of claim 42 wherein said plurality of encoded portions is contained within one or more tiles, each of said one or more tiles including data from a portion of an arc covering at least 270-degrees of said motion panoramic image.

54. The apparatus of claim 42 wherein each of said plurality of encoded portions is included with one of said one or more tiles respectively.

55. A computer program product including:

a computer usable data carrier having computer readable code embodied therein for causing a computer to present a view into a motion panoramic image, said computer readable code including;

computer readable program code configured to cause said computer to effect a first access mechanism configured to obtain availability information related to said motion panoramic image, said availability information including tier information;

computer readable program code configured to cause said computer to effect a bandwidth determination mechanism configured to determine the available bandwidth for accessing one or more tiles;

computer readable program code configured to cause said computer to effect a viewpoint determination mechanism configured to determine a viewpoint into said motion panoramic image, said viewpoint defining said view;

computer readable program code configured to cause said computer to effect a second access mechanism configured to access said one or more tiles, responsive to the viewpoint determination mechanism, the bandwidth determination mechanism and said tier information, said one or more tiles containing data used to present said view; and

computer readable program code configured to cause said computer to effect a presentation mechanism configured to present said view responsive to said data accessed by the second access mechanism.

56. The computer program product of claim 55 wherein the computer usable data carrier is a computer readable media.

57. The computer program product of claim 55 wherein the computer usable data carrier is a carrier wave.

58. The computer program product of claim 55 wherein each of said one or more tiles is encoded at a particular quality for that tile.

59. The computer program product of claim 55 wherein each of said one or more tiles is encoded using a distributed quality encoding for that tile.

60. The computer program product of claim 55 wherein the second access mechanism further includes:

computer readable program code configured to cause said computer to effect a subscription mechanism configured to subscribe to said one or more tiles responsive to said viewpoint and said availability information to establish a tile subscription; and

computer readable program code configured to cause said computer to effect a receiver mechanism configured to receive said one or more tiles responsive to said tile subscription determined by the subscription mechanism.

61. The computer program product of claim 60 further including:

computer readable program code configured to cause said computer to effect a pan mechanism configured to pan said viewpoint;

computer readable program code configured to cause said computer to effect a detection mechanism configured to detect that the pan mechanism performed a significant pan of said viewpoint responsive to the pan mechanism; and

computer readable program code configured to cause said computer to effect a subscription modification mechanism configured to change said tile subscription responsive to the detection mechanism.

62. The computer program product of claim 60 further including:

computer readable program code configured to cause said computer to effect a stability determination mechanism configured to detect that said viewpoint is stable; and

computer readable program code configured to cause said computer to effect a subscription modification mechanism configured to change said tile subscription responsive to the stability determination mechanism.

63. The computer program product of claim 60 further including:

computer readable program code configured to cause said computer to effect a detection mechanism configured to detect that said viewpoint is substantially constant over a specified period; and

computer readable program code configured to cause said computer to effect a subscription modification mechanism configured to change said tile subscription to increase a quality of said view as presented, the subscription modification mechanism responsive to the detection mechanism.

64. The computer program product of claim 63 wherein said quality is selected from the set consisting of presented resolution, field of view, view size, frame rate, compression, and color depth.

65. The computer program product of claim 60 wherein the first access mechanism further includes computer readable program code configured to cause said computer to access a server computer.

66. A computer program product including:

a computer usable data carrier having computer readable code embodied therein for causing a computer to encode a motion panoramic image, said computer readable code including;

computer readable program code configured to cause said computer to effect a frame receiver configured to receive a frame of said motion panoramic image;

computer readable program code configured to cause said computer to effect an encoder array configured to simultaneously encode a plurality of portions of said frame received from the frame receiver into a plurality of encoded portions in accordance with availability information; and

computer readable program code configured to cause said computer to effect a distribution mechanism configured to provide said plurality of encoded portions responsive to a selection of said availability information.

67. The computer program product of claim 66 wherein the computer usable data carrier is a computer readable media.

68. The computer program product of claim 66 wherein the computer usable data carrier is a carrier wave.

69. The computer program product of claim 66 wherein said availability information includes tier information.

70. The computer program product of claim 66 further comprising computer readable program code configured to cause said computer to effect a storage mechanism configured to store said plurality of encoded portions in said memory in accordance with said availability information.

71. The computer program product of claim 66 wherein said availability information includes tile information.

72. The computer program product of claim 66 further including computer readable program code configured to cause said computer to effect a repetition mechanism configured to repeatedly invoke the frame receiver, the encoder array and the distribution mechanism.

73. The computer program product of claim 66 wherein the encoder array further includes computer readable program code configured to cause said computer to effect a communication mechanism configured to send one or more of said plurality of encoded portions to a server computer.

74. The computer program product of claim 66 wherein said availability information represents a data structure in memory having a plurality of tiers.

75. The computer program product of claim 74 wherein each of said plurality of tiers includes one or more tiles, each of said one or more tiles referencing data encoded respective to a quality.

76. The computer program product of claim 75 wherein said quality is selected from the set consisting of presented resolution, field of view, view size, frame rate, compression, and color depth.

77. The computer program product of claim 75 wherein at least one of said plurality of tiers is respectively associated with a tier quality.

78. The computer program product of claim 75 wherein at least one of said one or more tiles is respectively associated with a tile quality.

79. The computer program product of claim 66 wherein said plurality of encoded portions is contained within one or more tiles, each of said one or more tiles including data from a portion of an arc covering at least 270-degrees of said motion panoramic image.

80. The computer program product of claim 66 wherein each of said plurality of encoded portions is included with one of said one or more tiles respectively.