Storage and Transmission of Pictures Including Multiple Frames

• US 20110234841A1
• Filed: 06/08/2011
• Published: 09/29/2011
• Est. Priority Date: 04/18/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Ng, Yi-Ren, Knight, Timothy James, Fatahalian, Kayvon, Akeley, Kurt Barton, Waters, Kenneth Wayne, Romanenko, Yuriy Aleksandrovich
• Granted Patent: US 8,908,058 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

According to various embodiments, multiple frames, each having image data and metadata, can be aggregated into pictures. The frames may come from different image capture devices, enabling aggregation of image data from multiple sources. Aggregation can be automatic, or it can be performed in response to user input specifying particular combinations of frames to be aggregated. In various embodiments, pictures are mutable, whereas immutability of the constituent frames is enforced. In various embodiments, certain metadata elements that are not essential to rendering can be selectively removed from frames, so as to address privacy concerns. In various embodiments, frames can be authenticated by the use of digests generated by a hash function.

Claims

1. A system for generating an image from component elements, comprising:

a storage device, configured to store a container file associated with a plurality of component elements, each component element comprising;

image data representing a captured image; and

metadata for the image data;

wherein the container file is further associated with instructions for combining the component elements with one another;

an image renderer, communicatively coupled to the storage device, configured to render a viewable image by retrieving the stored container file and performing at least one operation on at least a subset of the component elements according to the instructions; and

an output device, communicatively coupled to the image renderer, for outputting the viewable image.

2. The system of claim 1, wherein the container file comprises a picture file, and wherein each component element comprises a frame.

3. The system of claim 1, wherein the image renderer performs the at least one operation by combining at least two of the component elements according to the instructions.

4. The system of claim 1, wherein the image data represents an image captured at a camera.

5. The system of claim 1, wherein the image data represents at least one light field image.

6. The system of claim 1, further comprising:

at least one camera, configured to capture image data;

wherein at least one of the stored component elements comprises image data captured by the camera.

7. The system of claim 1, wherein the instructions further specify operations to be performed on at least one of the component elements to render the viewable image.

8. The system of claim 1, wherein:

the component elements associated with the container file are immutable; and

the instructions associated with the container file are mutable.

9. The system of claim 1, further comprising:

an input device, communicatively coupled to the storage device, for receiving user input for modifying the viewable image;

and wherein, responsive to the user input;

the storage device is configured to store new instructions specifying operations to be performed on at least one of the component elements to render a modified viewable image;

the image renderer is configured to render the modified viewable image by combining the component elements according to the new instructions; and

the output device is configured to output the modified viewable image.

10. The system of claim 1, wherein the image data comprises light field data.

11. The system of claim 1, wherein the metadata comprises a plurality of name-value pairs.

12. The system of claim 1, wherein the metadata is represented using JavaScript Object Notation.

13. The system of claim 1, wherein the metadata comprises at least one selected from the group consisting of:

a description of equipment used to capture the captured image;

a description of settings used when capturing the captured image;

a location associated with the captured image;

at least one parameter associated with the image data; and

light field data associated with the image data.

14. The system of claim 1, wherein the metadata comprises at least one removable metadata section.

15. The system of claim 1, further comprising:

a hash module, configured to apply a hash function to generate at least one digest from at least one component element;

wherein;

the storage device is configured to store the at least one digest.

16. The system of claim 15, further comprising:

an authenticator, configured to authenticate a component element by comparing a digest generated by the hash module from the component element with at least one previously stored digest associated with the component element.

17. The system of claim 1, further comprising:

a transmission device, communicatively coupled to the image renderer, configured to transmit the viewable image.

18. The system of claim 1, wherein:

the image renderer is further configured to generate a derived component element from at least one component element;

the storage device is further configured to store the derived component element; and

the image renderer is further configured to render the viewable image by performing at least one operation on at least a subset of the derived component elements according to the instructions.

19. The system of claim 18, wherein the image renderer performs the at least one operation by combining at least one derived component element with at least one component element according to the instructions.

20. The system of claim 1, wherein the storage device, the image renderer, and the output device are components of a camera, and wherein the camera further comprises an image capture mechanism configured to capture the image data.

21. The system of claim 20, wherein the camera further comprises a communications interface configured to receive at least one component element from a second camera.

22. The system of claim 1, wherein the storage device, the image renderer, and the output device are components of a computing device.

23. The system of claim 22, wherein the computing device further comprises a communications interface configured to receive at least one component element from a camera.

24. The system of claim 22, wherein the computing device further comprises a communications interface configured to receive at least one component element via an electronic communications network.

25. The system of claim 1, wherein the storage device is implemented at a server, and wherein the image renderer, and the output device are components of a client computing device communicatively coupled to receive data from the server.

26. The system of claim 1, wherein the plurality of component elements comprises component elements captured from a plurality of image capture devices.

27. The system of claim 1, wherein the plurality of component elements comprises component elements shared between a plurality of devices communicatively coupled with one another via an electronic communications network.

28. The system of claim 1, further comprising:

an input device, communicatively coupled to the storage device, configured to receive user input for modifying parameters for rendering the viewable image; and

wherein the image renderer is configured to render the viewable image according to the received user input.

29. A system for generating an image from a plurality of component elements, comprising:

at least one image capture device, each image capture device comprising;

an image capture mechanism, configured to capture at least one image;

a hash module, configured to apply a hash function to generate at least one digest from at least one captured image;

a storage device, communicatively coupled to the image capture mechanism and to the hash module, configured to store the at least one captured image and the at least one digest; and

a communications interface, configured to transmit the at least one digest to at least one other device;

an image renderer, configured to render a viewable image from a plurality of component elements by;

receiving a plurality of captured images;

authenticating at least one of the received images by applying a hash module to the received image and comparing the result with a previously stored digest; and

combining the plurality of captured images to generate a viewable image; and

an output device, communicatively coupled to the image renderer, for outputting the viewable image.

30. The system of claim 29, wherein the image renderer is associated with a server, and wherein receiving the plurality of captured images comprises receiving the captured images at the server.

31. The system of claim 29, wherein the image renderer is associated with an image capture device, and wherein receiving the plurality of captured images comprises receiving the captured images at the image capture device associated with the image renderer.

32. The system of claim 29, wherein the image capture device is configured to transmit the captured images to the server responsive to a trigger event.

33. The system of claim 32, wherein the trigger event comprises availability of sufficient bandwidth to transmit the captured images.

34. The system of claim 29, wherein each image capture device comprises a camera.

35. The system of claim 29, wherein:

the storage device of each image capture device is further configured to store metadata describing the at least one captured image;

the hash module generates the at least one digest from the captured image and the metadata; and

the image renderer combines the plurality of captured images using at least a subset of the metadata.

36. The system of claim 29, wherein the communications interface is configured to transmit the at least one digest to at least one other device via an electronic network.

37. The system of claim 29, wherein the image renderer is configured to receive the plurality of captured images via an electronic network.

38. The system of claim 29, wherein the storage device is configured to store a container file associated with a plurality of component elements, each component element comprising:

image data representing a captured image; and

metadata for the image data;

wherein the container file is further associated with instructions for combining the component elements with one another;

and wherein the image renderer is configured to retrieve the stored container file and perform at least one operation on at least a subset of the component elements according to the instructions.

39. A method for generating an image from component elements, comprising:

storing a container file associated with a plurality of component elements, each component element comprising;

image data representing a captured image; and

metadata for the image data;

wherein the container file is further associated with instructions for combining the component elements with one another;

retrieving the stored container file;

performing at least one operation on at least a subset of the component elements according to the instructions, to generate a viewable image; and

outputting the viewable image an output device.

40. The method of claim 39, wherein the container file comprises a picture file, and wherein each component element comprises a frame.

41. The method of claim 39, wherein performing at least one operation on at least a subset of the component elements comprises combining at least two of the component elements according to the instructions.

42. The method of claim 39, wherein the instructions further specify operations to be performed on at least one of the component elements to render the viewable image.

43. The method of claim 39, wherein:

the component elements associated with the container file are immutable; and

the instructions associated with the container file are mutable.

44. The method of claim 39, further comprising:

receiving user input for modifying the viewable image;

and, responsive to the user input;

storing new instructions specifying operations to be performed on at least one of the component elements to render a modified viewable image;

rendering the modified viewable image by combining the component elements according to the new instructions; and

outputting the modified viewable image.

45. The method of claim 39, further comprising:

applying a hash function to generate at least one digest from at least one component element; and

storing the at least one digest.

46. The method of claim 45, further comprising authenticating a component element by comparing a digest generated by the hash module from the component element with at least one previously stored digest associated with the component element.

47. The method of claim 39, further comprising transmitting the viewable image.

48. The method of claim 39, further comprising:

generating a derived component element from at least one component element;

storing the derived component element; and

rendering the viewable image by performing at least one operation on at least a subset of the derived component elements according to the instructions.

49. The method of claim 39, further comprising:

receiving user input for modifying parameters for rendering the viewable image; and

rendering the viewable image according to the received user input.

50. A method for generating an image from a plurality of component elements, comprising:

at a first device, capturing at least one image;

storing the at least one captured image;

receiving at least one digest from at least one second device, each digest having been generated by application of a hash function to a remotely-captured image;

subsequently obtaining the at least one remotely-captured image;

authenticating the obtained remotely-captured image by applying a hash module to the obtained remotely-captured image and comparing the result with a previously received digest;

combining the obtained remotely-captured image with at least one image captured at the first device, to generate a viewable image; and

outputting the viewable image.

51. The method of claim 50, wherein receiving at least one digest from at least one second device comprises receiving at least one digest from a server.

52. The method of claim 50, further comprising:

applying a hash function to generate at least one digest from at least one captured image; and

transmitting the generated digest to at least a second device.

53. The method of claim 52, further comprising transmitting at least one of the captured images to a server.

54. The method of claim 53, wherein transmitting at least one of the captured images to a server comprises transmitting at least one of the captured image to the server responsive to a trigger event.

55. The method of claim 54, wherein the trigger event comprises availability of sufficient bandwidth to transmit the at least one captured images.

56. The method of claim 50, further comprising:

storing metadata describing the at least one captured image; and

combining the obtained remotely-captured image with at least one image captured at the first device comprises combining the obtained remotely-captured image with at least one image captured at the first device using at least a subset of the metadata.

57. A computer program product for generating an image from component elements, comprising:

a nontransitory computer-readable storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of;

storing a container file associated with a plurality of component elements, each component element comprising;

image data representing a captured image; and

metadata for the image data;

wherein the container file is further associated with instructions for combining the component elements with one another;

retrieving the stored container file;

performing at least one operation on at least a subset of the component elements according to the instructions, to generate a viewable image; and

causing an output device to output the viewable image.

58. The computer program product of claim 57, wherein the container file comprises a picture file, and wherein each component element comprises a frame.

59. The computer program product of claim 57, wherein the computer program code configured to cause at least one processor to perform at least one operation on at least a subset of the component elements comprises computer program code configured to cause at least one processor to combine at least two of the component elements according to the instructions.

60. The computer program product of claim 57, wherein the instructions further specify operations to be performed on at least one of the component elements to render the viewable image.

61. The computer program product of claim 57, wherein:

the component elements associated with the container file are immutable; and

the instructions associated with the container file are mutable.

62. The computer program product of claim 57, further comprising:

computer program code configured to cause at least one input device to receive user input for modifying the viewable image;

computer program code configured to cause at least one processor to, responsive to the user input, perform the steps of;

causing a storage device to store new instructions specifying operations to be performed on at least one of the component elements to render a modified viewable image;

rendering the modified viewable image by combining the component elements according to the new instructions; and

causing an output device to output the modified viewable image.

63. The computer program product of claim 57, further comprising computer program code configured to cause at least one processor to perform the steps of:

applying a hash function to generate at least one digest from at least one component element; and

storing the at least one digest.

64. The computer program product of claim 63, further comprising computer program code configured to cause at least one processor to perform the step of authenticating a component element by comparing a digest generated by the hash module from the component element with at least one previously stored digest associated with the component element.

65. The computer program product of claim 57, further comprising computer program code configured to cause at least one processor to perform the steps of:

generating a derived component element from at least one component element;

causing a storage device to store the derived component element; and

rendering the viewable image by performing at least one operation on at least a subset of the derived component elements according to the instructions.

66. A computer program product for generating an image from a plurality of component elements, comprising:

a nontransitory computer-readable storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor at a first device to perform the steps of;

capturing at least one image;

causing a storage device to store the at least one captured image;

receiving at least one digest from at least one second device, each digest having been generated by application of a hash function to a remotely-captured image;

subsequently obtaining the at least one remotely-captured image;

authenticating the obtained remotely-captured image by applying a hash module to the obtained remotely-captured image and comparing the result with a previously received digest;

combining the obtained remotely-captured image with at least one image captured at the first device, to generate a viewable image; and

causing an output device to output the viewable image.

67. The computer program product of claim 66, wherein the computer program code configured to cause at least one processor to receive at least one digest from at least one second device comprises computer program code configured to cause at least one processor to receive at least one digest from a server.

68. The computer program product of claim 66, further comprising computer program code configured to cause at least one processor to perform the steps of:

applying a hash function to generate at least one digest from at least one captured image; and

transmitting the generated digest to at least a second device.

69. The computer program product of claim 68, further comprising computer program code configured to cause at least one processor to transmit at least one of the captured images to a server.

70. The computer program product of claim 66, further comprising computer program code configured to cause at least one processor to perform the steps of:

causing a storage device to store metadata describing the at least one captured image; and

combining the obtained remotely-captured image with at least one image captured at the first device comprises combining the obtained remotely-captured image with at least one image captured at the first device using at least a subset of the metadata.