Method and an apparatus for generating image content

• US 8,427,545 B2
• Filed: 11/20/2007
• Issued: 04/23/2013
• Est. Priority Date: 12/06/2006
• Status: Expired due to Fees

• Current Assignee: Sony Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Europe Limited (Sony Group Corp.)
• Inventors: Porter, Robert Mark Stefan, Gillard, Clive Henry, Stone, Jonathan James, Berry, David, Haynes, Simon Dominic
• Primary Examiner(s): Henn, Timothy J
• Application Number:
  US12/517,982
• Publication Number:
  US 20100318467A1
• Time in Patent Office: 1,981 Days
• Field of Search: 348/48, 348/143, 348/157, 348/218.1, 348/239, 382/293, 382/294
• US Class Current: 348/218.1
• CPC Class Codes: H04N 5/2624 for obtaining an image which is composed of whole input images, e.g. splitscreen

Abstract

A method and a system for generating image content. The method and system allow segments of a panoramic scene, to be generated with reduced distortion. The method and system reduce the amount of distortion by mapping pixel data onto a pseudo camera focal plane which is provided substantially perpendicularly to the focal location of the camera that captured the image. A camera arrangement can implement the method and system.

Claims

1. A method of generating image content, comprising:

receiving image data representing an image generated by a system of camera elements having a predetermined focal location wherein the system of camera elements comprises a plurality of camera elements arranged in a horizontally concave manner, whereby each camera element generates element image data representing the field of view of the camera element, wherein at least part of the field of view of adjacent camera elements is different; and

the image data is generated and representative of a composite image generated from the received element image data;

generating segment image data representing a selected segment of the image by;

(i) determining a substantially flat plane representing the selected segment, the flat plane being positioned such that the flat plane lies substantially perpendicular to the focal location of the camera element at a reference location in the selected segment, and

(ii) mapping pixel data onto the substantially flat plane in accordance with at least one parameter of the camera element.

2. A method according to claim 1, wherein the pixel data is derived from the camera element.

3. A method according to claim 1, wherein the pixel data is derived from the image data, wherein the camera parameter is provided in association with the image data.

4. A method according to claim 1, further comprising:

outlining the segment image data on the image by determining the outline of the flat plane and mapping the outline of the flat plane onto the image in accordance with the angular position of the flat plane relative to the image at the reference location.

5. A method according to claim 1, wherein the system of camera elements comprises a plurality of camera elements each camera element generating element image data representing the field of view of the camera element, wherein at least part of the field of view of adjacent camera elements is different;

and the image data is generated and representative of a composite image generated from the received element image data.

6. A method according to claim 5, wherein at least one camera parameter of each camera element is determined in accordance with an error in an overlap between fields of view of adjacent camera elements.

7. A method according to claim 5, wherein the generating the composite image comprises:

mapping the received element image data onto a pixel on a plane representing the composite image in accordance with at least one of the parameters of a camera element.

8. A method according to claim 7, comprising:

overlapping at least part of the field of view of adjacent camera elements, and

wherein the mapping of the received element image data or pixel data comprises;

interpolating the received element image data or pixel data, respectively, which represents a plurality of pixels in the area of overlap if an error measure between the pixels is below a threshold.

9. A method according to claim 7, further comprising:

overlapping at least part of the field of view of adjacent camera elements, and

wherein the mapping of the received image or pixel data comprises;

selecting the received element image data or pixel data, respectively, which represents a pixel in the area of overlap from one of the camera elements rather than a pixel from an adjacent camera element if an error measure between said pixels is above a threshold.

10. A method according to claim 8, wherein the error measure is a parallax error measure.

11. A method according to claim 5, wherein the focal point of a pair of adjacent camera elements is substantially the same.

12. A method according to claim 1, further comprising:

reducing the chromatic aberration generated by the system of camera elements in accordance with the comparison of the red and green components and the blue and green components of the image data generated by the system of camera elements.

13. A method according to claim 5, further comprising:

reducing the chromatic aberration generated by one of the plurality of the camera elements in accordance with the comparison of the red and green components and the blue and green components of the element image data generated by the camera element.

14. A method according to claim 1, further comprising:

detecting in the image data an object based on an object detection algorithm; and

generating a position signal identifying the location of the detected object in the image data.

15. A method according to claim 14, further comprising tracking the detected object between successive frames of the image based on an object tracking algorithm.

16. A method according to claim 14, further comprising identifying the segment of the image, in accordance with the position signal.

17. A method according to claim 15, wherein if between successive frames of the image the tracking of the detected object between successive frames stalls, the method further comprises:

identifying an area of the image in accordance with the position signal; and

detecting, within the identified area and in accordance with the detection algorithm, the object.

18. A method according to claim 1, further comprising displaying, on a display, the image.

19. A method according to claim 18, wherein the displaying comprises displaying simultaneously the image and the segment image data.

20. A method according to claim 18, wherein the display is a curved screen.

21. A method according to claim 1, further comprising distributing a representation of the or part of the image and/or the segment image data.

22. A method according to claim 21, further comprising distributing ancillary data separately to the representation of the or part of the image and/or segment image data.

23. A method according to claim 22, wherein the ancillary data is metadata.

24. A method according to claim 23, wherein the ancillary data or metadata is distributed in a data carousel.

25. A method according to claim 21, wherein the ancillary data, the representation of the or part of the image and/or the segment image data is distributed by broadcasting.

26. A method according to claim 21, wherein the ancillary data, the representation of the or part of the image and/or the segment image data is distributed over a network.

27. A method according to claim 21, further comprising encrypting said the or at least part of the representation or the ancillary data, using an encryption key, before being distributed.

62. A non-transitory computer readable storage medium storing thereon computer readable instructions which, when loaded onto a computer make the computer perform a method according to claim 1.

28. An image content generating apparatus, comprising:

an image receiving device operable to receive image data representing an image generated by a system of camera elements having a predetermined focal location wherein the system of camera elements comprises a plurality of camera elements arranged in a horizontally concave manner, whereby each camera element generates element image data representing the field of view of the camera element, wherein at least part of the field of view of adjacent camera elements is different; and

the image data is generated and representative of a composite image generated from the received element image data; and

an image processing device operable to generate segment image data representing a selected segment of the image by;

(i) determining a substantially flat plane representing the selected segment, the flat plane being positioned such that the flat plane lies substantially perpendicular to the focal location of the camera element at a reference location in the selected segment, and

(ii) mapping pixel data onto the substantially flat plane in accordance with at least one parameter of the camera element.

29. An apparatus according to claim 28, wherein the pixel data is derived from the camera element.

30. An apparatus according to claim 28, wherein the pixel data is derived from the image data, wherein the camera parameter is provided in association with the image data.

31. An apparatus according to claim 28, wherein the image processing device is operable to outline the segment image data on the image by determining the outline of the flat plane and mapping the outline of the flat plane onto the image in accordance with the angular position of the flat plane relative to the image at the reference location.

32. An apparatus according to claim 28, wherein the system of camera elements comprises a plurality of camera elements each camera element generating element image data representing the field of view of the camera element, wherein at least part of the field of view of adjacent camera elements is different;

and the image data is generated and representative of a composite image generated from the received element image data.

33. An apparatus according to claim 32, wherein the image processing device is operable such that at least one camera parameter of each camera element is determined in accordance with an error in an overlap between fields of view of adjacent camera elements.

34. An apparatus according to claim 32, wherein the image processing device is operable such that the generating the composite image comprises:

mapping the received element image data onto a pixel on a plane representing the composite image in accordance with at least one parameter of a camera element.

35. An apparatus according to claim 34, wherein the image processing device is operable to:

overlap at least part of the field of view of adjacent camera elements, and

wherein the mapping of the received element image data or pixel data, the image processing device is operable to;

interpolate the received element image data or pixel data, respectively, which represents a plurality of pixels in the area of overlap if an error measure between the pixels is below a threshold.

36. An apparatus according to claim 34, wherein the image processing device is operable to:

overlap at least part of the field of view of adjacent camera elements, and

wherein the mapping of the received image or pixel data, the image processing device is operable to;

select the received element image data or pixel data, respectively, which represents a pixel in the area of overlap from one of the camera elements rather than a pixel from an adjacent camera element if an error measure between said pixels is above a threshold.

37. An apparatus according to claim 35, wherein the error measure is a parallax error measure.

38. An apparatus according to claim 33, wherein the focal point of a pair of adjacent camera elements is substantially the same.

39. An apparatus according to claim 28, further comprising:

a chromatic aberration reducing device operable to reduce the chromatic aberration generated by the system of camera elements in accordance with the comparison of the red and green components and the blue and green components of the image data generated by the system of camera elements.

40. An apparatus according to claim 33, further comprising:

a chromatic aberration reducing device operable to reduce the chromatic aberration generated by one of the plurality of the camera elements in accordance with the comparison of the red and green components and the blue and green components of the element image data generated by the camera element.

41. An apparatus according to claim 28, further comprising:

an object detecting device operable to;

detect in the image data an object based on an object detection algorithm; and

generate a position signal identifying the location of the detected object in the image data.

42. An apparatus according to claim 41, wherein the object detecting device is operable to track the detected object between successive frames of the image based on an object tracking algorithm.

43. An apparatus according to claim 41, wherein the object detecting device is operable to identify the segment of the image, in accordance with the position signal.

44. An apparatus according to claim 42, wherein if between successive frames of the image the tracking of the detected object between successive frames stalls, the object detecting device is further operable to:

identify an area of the image in accordance with the position signal; and

detect, within the identified area and in accordance with the detection algorithm, the object.

45. An apparatus according to claim 28, further comprising a display operable to display the image.

46. An apparatus according to claim 45, wherein the display is operable to display simultaneously the image and the segment image data.

47. An apparatus according to claim 45, wherein the display is a curved screen.

48. An apparatus according to claim 28, further comprising a distributor operable to distribute a representation of the or part of the image and/or the segment image data.

49. An apparatus according to claim 48, wherein the distributor is operable to distribute ancillary data separately to the representation of the or part of the image and/or segment image data.

50. An apparatus according to claim 48, wherein the ancillary data is metadata.

51. An apparatus according to claim 49, wherein the ancillary data or metadata is distributed in a data carousel.

52. An apparatus according to claim 48, wherein the distributor is operable to distribute the ancillary data, the representation of the or part of the image and/or the segment image data by broadcasting.

53. An apparatus according to claim 48, wherein the distributor is operable to distribute the ancillary data, the representation of the or part of the image and/or the segment image data is distributed over a network.

54. An apparatus according to claim 48, wherein the distributor is operable to encrypt the or at least part of the representation or the ancillary data, using an encryption key, before being distributed.

55. An apparatus according to claim 32, wherein each camera element in the camera element cluster is arranged to be orientated in portrait mode.

56. A display device arranged to operate with an apparatus according to claim 48 and operable to display the or at least part of the representation of the image and/or the segment image data, the device comprising:

a display for displaying the representation of the or part of the image and/or the segment image data.

57. A display device according to claim 56, further comprising decrypting means for decrypting an encrypted representation of the or part of the part of the image and/or segment image data.

58. A display device according to claim 57, further comprising memory means containing the decrypting key readable by the decrypting means, the memory means being operable to update the decryption key.

59. A display device according to claim 56, further comprising playback means operable to store, in accordance with a predetermined criterion, the or part of the image and/or segment image data for playback.

60. A display device, comprising data receiving means operable to receive a data channel containing ancillary data transmitted from the apparatus according to claim 49.

61. A display device according to claim 56 configured to be a personal display device.

63. A method of generating revenue comprising:

receiving image data representing an image generated by a system of camera elements having a predetermined focal location wherein the system of camera elements comprises a plurality of camera elements arranged in a horizontally concave manner, whereby each camera element generates element image data representing the field of view of the camera element, wherein at least part of the field of view of adjacent camera elements is different; and

the image data is generated and representative of a composite image generated from the received element image data;

generating segment image data representing a selected segment of the image by;

(i) determining a substantially flat plane representing the selected segment, the flat plane being positioned such that the flat plane lies substantially perpendicular to the focal location of the camera element at a reference location in the selected segment, and

(ii) mapping pixel data onto the substantially flat plane in accordance with at least one parameter of the camera element;

and

allowing, in exchange for money, a user to view the segment image data.