Method and system for generation of multiple viewpoints into a scene viewed by motionless cameras and for presentation of a view path

• US 6,741,250 B1
• Filed: 10/17/2001
• Issued: 05/25/2004
• Est. Priority Date: 02/09/2001
• Status: Expired due to Term

• Current Assignee: DigiMedia Tech, LLC (IP Investments Group LLC)
• Original Assignee: Be Here Corporation
• Inventors: Furlan, John L. W., Shiffer, Katerina L., Fluker, Derek, Hoffman, Robert G., Driscoll, Edward C. Jr., Curtis, Daniel B., Garimella, Venugopal, Bechtel, Karen L.
• Primary Examiner(s): Nguyen, Phu K.
• Application Number:
  US09/982,614
• Time in Patent Office: 951 Days
• Field of Search: 345/419, 345/427, 345/428, 345/473, 345/619, 345/581, 345/582
• US Class Current: 345/427
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure

Abstract

Methods and apparatus for defining a view path through at least one wide-angle video stream and for creation of a resultant unwarped video stream responsive to the view path. Aspects of the invention allow an operator to monitor the wide-angle video stream(s) and select which portion of the wide-angle frame to unwarp in real-time. Further, the operator can stop the playback of the wide-angle video stream and dwell within a particular wide-angle frame to create special effects such as camera tilt, pan and zoom operations in a stop action situation.

Claims

1. A method of:

recording a video stream comprising a plurality of frames, wherein said plurality of frames define a plurality of distorted images;

designating a portion of said video stream to be a video segment; and

specifying a view path through said video segment.

2. The method of claim 1 wherein the step of designating said portion further comprises steps of:

designating a first segment frame;

designating a second segment frame; and

including at least one intervening frame between said first segment frame and said second segment frame.

3. The method of claim 1 wherein said video segment is one of a plurality of video segments and said view path also extends through another of said plurality of video segments.

4. The method of claim 1 wherein the step of specifying said view path further comprises associating said view path with some of said plurality of frames.

5. The method of claim 4 wherein the step of associating further comprises steps of:

presenting at least one of said plurality of frames;

selecting a portion of said at least one of said plurality of frames as presented; and

determining view information corresponding to said selected portion of said at least one of said plurality of frames, said view information comprising at least two coordinates selected from the set consisting of a pan coordinate, a tilt coordinate, a zoom coordinate and a time coordinate.

6. The method of claim 5 wherein the step of selecting further comprises a step of displaying a distorted-outline that indicates said portion as selected, whereby a user selecting said portion knows what is contained in said portion.

7. The method of claim 5 further comprising storing said view information within said at least one of said plurality of frames, said stored view information defining a location on said view path, said view path configured to specify how said view information into said plurality of frames changes throughout said video segment.

8. The method of claim 5 further comprising associating said view information with said at least one of said plurality of frames, said associated view information defining a location on said view path, said view path specifying how said view information into said plurality of frames changes throughout said video segment.

9. The method of claim 5 further comprising steps of:

defining said view path as a parameterized path using said view information, said view path specifying how said view information into said plurality of frames changes throughout said video segment; and

storing parameters specifying said parameterized path.

10. The method of claim 9 wherein said parameterized path is defined by a B-spline.

11. The method of claim 1 further comprising presenting a view into said video segment, said view corresponding to said view path.

12. The method of claim 11 further comprising dynamically altering said view path during presentation of said view, said view presentation correspondingly altered.

13. The method of claim 11 further comprising predicting alterations in said view path during presentation of said view.

14. The method of claim 13 wherein the step of predicting uses a predictor.

15. The method of claim 14 wherein said predictor is selected from the group consisting of an infinite impulse response filter, a kalman filter, and a block matching filter.

16. The method of claim 11 wherein the step of presenting further comprises steps of:

generating a viewing vector into one of said plurality of distorted images responsive to said view path; and

transforming a portion of said one of said plurality of distorted images responsive to said viewing vector to create said view.

17. The method of claim 11 wherein the step of presenting further comprises steps of:

generating a viewing vector into one of said plurality of frames responsive to said view path; and

selecting a portion of said one of said plurality of frames responsive to said viewing vector to create said view.

18. A method of:

recording a video stream comprising a plurality of frames that define a plurality of distorted images;

designating a portion of said video stream to be a video segment;

specifying a view path through said video segment;

generating a viewing vector into one of said plurality of distorted images responsive to said view path;

transforming a portion of said one of said plurality of distorted images responsive to said viewing vector to create a view; and

presenting said view.

19. An apparatus comprising:

a recording mechanism configured to record a video stream, said video stream comprising a plurality of frames;

a segment specification mechanism configured to designate a portion of said video stream to be a video segment; and

a view path specification mechanism configured to specify a view path through said video segment designated by the segment specification mechanism.

20. The apparatus of claim 19 wherein the segment specification mechanism further comprises:

a first frame designation mechanism configured to designate a first segment frame; and

a second frame designation mechanism configured to designate a second segment frame with at least one intervening frame between said first segment frame and said second segment frame.

21. The apparatus of claim 19 wherein said video segment is one of a plurality of video segments and said view path also extends through another of said plurality of video segments.

22. The apparatus of claim 19 wherein the view path specification mechanism further comprises an association mechanism configured to associate said view path with some of said plurality of frames.

23. The apparatus of claim 22 wherein the association mechanism further comprises:

a presentation mechanism configured to present at least one of said plurality of frames;

a view selection mechanism configured to select a portion of said at least one of said plurality of frames as presented by the presentation mechanism; and

view information mechanism, responsive to the view selection mechanism, configured to determine view information corresponding to said selected portion of said at least one of said plurality of frames, said view information comprising at least two coordinates selected from the set consisting of a pan coordinate, a tilt coordinate, a zoom coordinate and a time coordinate.

24. The apparatus of claim 23 wherein the view information mechanism further comprises a selection display mechanism configured to display a distorted-outline that indicates said portion as selected, whereby a user selecting said portion knows what is contained in said portion.

25. The apparatus of claim 23 further comprising a storage mechanism configured to store said view information within said at least one of said plurality of frames, said stored view information defining a location on said view path, said view path configured to specify how said view information into said plurality of frames changes throughout said video segment.

26. The apparatus of claim 23 further comprising a storage mechanism configured to associate said view information with said at least one of said plurality of frames, said associated view information defining a location on said view path, said view path specifying how said view information into said plurality of frames changes throughout said video segment.

27. The apparatus of claim 23 further comprising:

a path determination mechanism configured to define said view path as a parameterized path using said view information, said view path specifying how said view information into said plurality of frames changes throughout said video segment; and

a storage mechanism configured to store parameters specifying said parameterized path.

28. The apparatus of claim 27 wherein said parameterized path is defined by a B-spline.

29. The apparatus of claim 19 further comprising a presentation mechanism configured to present a view into said video segment, said view corresponding to said view path.

30. The apparatus of claim 29 further comprising an alter view path mechanism configured to dynamically alter said view path during presentation of said view, said view presentation correspondingly altered.

31. The apparatus of claim 29 further comprising a predictor configured to predict alterations in said view path during presentation of said view.

32. The apparatus of claim 31 wherein the predictor is selected from the group consisting of an infinite impulse response filter, a kalman filter, and a block matching filter.

33. The apparatus of claim 29 wherein the presentation mechanism further comprises:

a vector generator configured to generate a viewing vector into one of said plurality of distorted images responsive to said view path; and

a transformation mechanism, responsive to the vector generator, configured to transform a portion of said one of said plurality of distorted images responsive to said viewing vector to create said view.

34. The apparatus of claim 29 wherein the presentation mechanism further comprises:

a vector generator configured to generate a viewing vector into one of said plurality of frames responsive to said view path; and

a view selection mechanism configured to select a portion of said one of said plurality of frames responsive to said viewing vector to create said view.

35. An apparatus comprising:

a recording mechanism configured to record a video stream comprising a plurality of frames that define a plurality of distorted images;

a segment specification mechanism configured to designate a portion of said video stream to be a video segment;

a view path specification mechanism configured to specify a view path through said video segment specified by the segment specification mechanism;

a vector generator configured to generate a viewing vector into one of said plurality of distorted images responsive to said view path specified by the view path specification mechanism;

a transformation mechanism, responsive to the vector generator, configured to transform a portion of said one of said plurality of distorted images responsive to said viewing vector to create a view; and

a presentation mechanism, responsive to the transformation mechanism, configured to present a view into said video segment, said view corresponding to said view path.

36. A computer program product comprising:

a computer usable data carrier having computer readable code embodied therein for causing a computer to record a video stream, said computer readable code comprising;

computer readable program code configured to cause said computer to effect a recording mechanism configured to record said video stream, said video stream comprising a plurality of frames;

computer readable program code configured to cause said computer to effect a segment specification mechanism configured to designate a portion of said video stream to be a video segment; and

computer readable program code configured to cause said computer to effect a view path specification mechanism configured to specify a view path through said video segment designated by the segment specification mechanism.

37. The computer program product of claim 36 wherein the segment specification mechanism further comprises:

computer readable program code configured to cause said computer to effect a first frame designation mechanism configured to designate a first segment frame; and

computer readable program code configured to cause said computer to effect a second frame designation mechanism configured to designate a second segment frame with at least one intervening frame between said first segment frame and said second segment frame.

38. The computer program product of claim 36 wherein said video segment is one of a plurality of video segments and said view path also extends through another of said plurality of video segments.

39. The computer program product of claim 36 wherein the view path specification mechanism further comprises computer readable program code configured to cause said computer to effect an association mechanism configured to associate said view path with some of said plurality of frames.

40. The computer program product of claim 39 wherein the association mechanism further comprises:

computer readable program code configured to cause said computer to effect a presentation mechanism configured to present at least one of said plurality of frames;

computer readable program code configured to cause said computer to effect a view selection mechanism configured to select a portion of said at least one of said plurality of frames as presented by the presentation mechanism; and

computer readable program code configured to cause said computer to effect view information mechanism, responsive to the view selection mechanism, configured to determine view information corresponding to said selected portion of said at least one of said plurality of frames, said view information comprising at least two coordinates selected from the set consisting of a pan coordinate, a tilt coordinate, a zoom coordinate and a time coordinate.

41. The computer program product of claim 40 wherein the view information mechanism further comprises computer readable program code configured to cause said computer to effect a selection display mechanism configured to display a distorted-outline that indicates said portion as selected, whereby a user selecting said portion knows what is contained in said portion.

42. The computer program product of claim 40 further comprising computer readable program code configured to cause said computer to effect a storage mechanism configured to store said view information within said at least one of said plurality of frames, said stored view information defining a location on said view path, said view path configured to specify how said view information into said plurality of frames changes throughout said video segment.

43. The computer program product of claim 40 further comprising computer readable program code configured to cause said computer to effect a storage mechanism configured to associate said view information with said at least one of said plurality of frames, said associated view information defining a location on said view path, said view path specifying how said view information into said plurality of frames changes throughout said video segment.

44. The computer program product of claim 40 further comprising:

computer readable program code configured to cause said computer to effect a path determination mechanism configured to define said view path as a parameterized path using said view information, said view path specifying how said view information into said plurality of frames changes throughout said video segment; and

computer readable program code configured to cause said computer to effect a storage mechanism configured to store parameters specifying said parameterized path.

45. The computer program product of claim 44 wherein said parameterized path is defined by a B-spline.

46. The computer program product of claim 36 further comprising computer readable program code configured to cause said computer to effect a presentation mechanism configured to present a view into said video segment, said view corresponding to said view path.

47. The computer program product of claim 46 further comprising computer readable program code configured to cause said computer to effect an alter view path mechanism configured to dynamically alter said view path during presentation of said view, said view presentation correspondingly altered.

48. The computer program product of claim 46 further comprising computer readable program code configured to cause said computer to effect a predictor configured to predict alterations in said view path during presentation of said view.

49. The computer program product of claim 48 wherein the predictor is selected from the group consisting of an infinite impulse response filter, a kalman filter, and a block matching filter.

50. The computer program product of claim 46 wherein the presentation mechanism further comprises:

computer readable program code configured to cause said computer to effect a vector generator configured to generate a viewing vector into one of said plurality of distorted images responsive to said view path; and

computer readable program code configured to cause said computer to effect a transformation mechanism, responsive to the vector generator, configured to transform a portion of said one of said plurality of distorted images responsive to said viewing vector to create said view.

51. The computer program product of claim 46 wherein the presentation mechanism further comprises:

computer readable program code configured to cause said computer to effect a vector generator configured to generate a viewing vector into one of said plurality of frames responsive to said view path; and

computer readable program code configured to cause said computer to effect a view selection mechanism configured to select a portion of said one of said plurality of frames responsive to said viewing vector to create said view.

52. The computer program product of claim 36 wherein the computer usable data carrier is a computer readable media.

53. The computer program product of claim 36 wherein the computer usable data carrier is a carrier wave.

54. A computer program product comprising:

a computer usable data carrier having computer readable code embodied therein for causing a computer to present a view from a video stream, said computer readable code comprising;

computer readable program code configured to cause said computer to effect a recording mechanism configured to record said video stream comprising a plurality of frames that define a plurality of distorted images;

computer readable program code configured to cause said computer to effect a segment specification mechanism configured to designate a portion of said video stream to be a video segment;

computer readable program code configured to cause said computer to effect a view path specification mechanism configured to specify a view path through said video segment specified by the segment specification mechanism;

computer readable program code configured to cause said computer to effect a vector generator configured to generate a viewing vector into one of said plurality of distorted images responsive to said view path specified by the view path specification mechanism;

computer readable program code configured to cause said computer to effect a transformation mechanism, responsive to the vector generator, configured to transform a portion of said one of said plurality of distorted images responsive to said viewing vector to create said view; and

computer readable program code configured to cause said computer to effect a presentation mechanism, responsive to the transformation mechanism, configured to present a view into said video segment, said view corresponding to said view path.

55. The computer program product of claim 54 wherein the computer usable data carrier is a computer readable media.

56. The computer program product of claim 54 wherein the computer usable data carrier is a carrier wave.