Selective Transmission of Image Data Based on Device Attributes

• US 20120249550A1
• Filed: 06/14/2012
• Published: 10/04/2012
• Est. Priority Date: 04/18/2009
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Lytro Incorporated
• Inventors: Akeley, Kurt Barton, Ng, Yi-Ren, Waters, Kenneth Wayne, Fatahalian, Kayvon, Knight, Timothy James, Romanenko, Yuriy Aleksandrovich, Pitts, Colvin, Liang, Chia-Kai, Hanley, Thomas, Marculescu, Mugur
• Application Number:
  US13/523,776
• Publication Number:
  US 20120249550A1
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: H04N 21/234363 by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/25825 involving client display capabilities, e.g. screen resolution of a mobile phone optimising the visualisation of content during browsing in the Internet G06F16/9577; processing of terminal status or physical abilities in wireless networks H04W8/22; authentication in wireless network security H04W12/06
• CPC Class Codes: H04N 21/25833 involving client hardware characteristics, e.g. manufacturer, processing or storage capabilities allocation of resources considering hardware capabilities in multiprogramming arrangements G06F9/5044; allocation of resources considering software capabilities in multiprogramming arrangements G06F9/5055
• CPC Class Codes: H04N 21/25891 being end-user preferences retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

A system and method are provided for storing, manipulating, and/or transmitting image data, such as light field photographs and the like, in a manner that efficiently delivers different capabilities and features based on device attributes, user requirements and preferences, context, and/or other factors. Acceleration structures are provided, which enable selective use of certain types of data (also referred to as “assets”) based on device attributes such as image size, desired functionality, user preference, and/or the like. In this manner, the system and method of the present invention takes into account specific attributes and parameters in determining which data should be included, so as to optimize transmission, storage, and/or rendering of image data, including light field data, to improve efficiency and avoid waste of resources.

Claims

1. A method for transmitting image-related assets to a device, comprising:

at a processor, receiving a request for image-related assets from the device, the request comprising an indication of at least one attribute;

at the processor, based on the attribute, selecting at least one available asset from a plurality of available assets; and

transmitting the selected at least one available asset to the device.

2. The method of claim 1, wherein selecting at least one available asset comprises selecting at least one available asset based on suitability of each asset with respect to the indicated attribute.

3. The method of claim 1, wherein at least one indicated attribute specifies at least one hardware characteristic of the device.

4. The method of claim 3, wherein at least one hardware characteristic of the device comprises at least one selected from the group consisting of:

a characteristic of an output device associated with the device;

an indication of available memory;

an indication of available storage;

an indication of processing power associated with the device;

size of a screen associated with the device;

an indication as to whether a graphics processing unit is available for rendering images;

an indication as to a type of input device available to the device; and

an indication as to whether an accelerometer is available.

5. The method of claim 1, wherein at least one indicated attribute specifies at least one characteristic of software running at the device.

6. The method of claim 1, wherein at least one indicated attribute specifies at least one desired feature for displaying at least one image on the device.

7. The method of claim 6, wherein at least one desired feature comprises at least one selected from the group consisting of:

an ability to interact with an image;

an ability to refocus an image at any of a number of different focus depths;

an ability to perform depth-based processing on an image;

an ability to present an image in a three-dimensional format;

an ability to provide stereoscopic viewing of an image;

an ability to present a parallax shift for an image;

an ability to present an image having extended depth-of-field;

an ability to process different parts of an image differently depending on depicted distance;

an ability to display a sequence of images over time;

an ability to allow a user to perform at least one of adding, modifying and removing information associated with an image; and

an ability to allow a user to edit an image.

8. The method of claim 1, wherein at least one indicated attribute comprises an indication of image size.

9. The method of claim 1, wherein the steps of receiving the request and selecting at least one available asset are performed at a server, and wherein transmitting the selected at least one available asset comprises transmitting the selected at least one available asset from the server to the device.

10. The method of claim 1, wherein transmitting the selected at least one available asset to the device comprises:

providing, to the device, at least one link to at least one available asset.

11. The method of claim 1, wherein transmitting the selected at least one available asset to the device comprises:

retrieving the at least one asset from storage; and

transmitting the retrieved at least one asset to the device.

12. The method of claim 1, wherein transmitting the selected at least one available asset to the device comprises:

generating the at least one asset from stored image data; and

transmitting the generated at least one asset to the device.

13. The method of claim 12, wherein generating the at least one asset from stored image data comprises generating the at least one asset from stored light field data.

14. The method of claim 1, further comprising, subsequently to transmitting the selected at least one available asset, rendering and outputting the image at the device.

15. The method of claim 1, wherein at least one available asset comprises at least one selected from the group consisting of:

light field data;

metadata;

at least one extended depth-of-field image;

at least one sub-aperture image; and

a depth map.

16. The method of claim 1, wherein at least one available asset comprises a focus stack comprising a plurality of images associated with different focus depths.

17. The method of claim 1, wherein at least one available asset comprises a tiled focus stack, comprising a plurality of tiles representing portions of an image, wherein at least two of the tiles are associated with different focus depths.

18. The method of claim 17, wherein the tiled focus stack is generated based on determined focal depths for objects within an image.

19. The method of claim 17, further comprising:

at the processor, generating an image by blending at least two tiles in the focus stack with one another.

20. The method of claim 1, wherein at least one available asset is generated from at least one light field image.

21. A method for requesting image-related assets at a device, comprising:

at a processor, determining at least one attribute for display of an image at a device;

at the processor, determining a set of available image-related assets for display of the image;

at the processor, based on the determined attribute and the determined available assets, selecting at least one of the available assets;

at the processor, requesting the selected at least one asset from a server;

at the device, receiving the selected at least one asset from the server;

at the processor, rendering the image using the received at least one asset; and

displaying the rendered image at an output device.

22. The method of claim 21, wherein determining a set of available image-related assets comprises:

querying a server; and

receiving a response from the server, the response specifying the set of available image-related assets.

23. The method of claim 21, wherein displaying the rendered image at an output device comprises displaying an interactive image.

24. A computer program product for transmitting image-related assets to a device, comprising:

a non-transitory computer-readable storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of;

receiving a request for image-related assets from the device, the request comprising an indication of at least one attribute;

based on the attribute, selecting at least one available asset from a plurality of available assets; and

transmitting the selected at least one available asset to the device.

25. The computer program product of claim 24, wherein the computer program code configured to cause at least one processor to select at least one available asset comprises computer program code configured to cause at least one processor to select at least one available asset based on suitability of each asset with respect to the indicated attribute.

26. The computer program product of claim 24, wherein at least one indicated attribute specifies at least one hardware characteristic of the device.

27. The computer program product of claim 26, wherein at least one hardware characteristic of the device comprises at least one selected from the group consisting of:

a characteristic of an output device associated with the device;

an indication of available memory;

an indication of available storage;

an indication of processing power associated with the device;

size of a screen associated with the device;

an indication as to whether a graphics processing unit is available for rendering images;

an indication as to a type of input device available to the device; and

an indication as to whether an accelerometer is available.

28. The computer program product of claim 24, wherein at least one indicated attribute specifies at least one characteristic of software running at the device.

29. The computer program product of claim 24, wherein at least one indicated attribute specifies at least one desired feature for displaying at least one image on the device.

30. The computer program product of claim 29, wherein at least one desired feature comprises at least one selected from the group consisting of:

an ability to interact with an image;

an ability to refocus an image at any of a number of different focus depths;

an ability to perform depth-based processing on an image;

an ability to present an image in a three-dimensional format;

an ability to provide stereoscopic viewing of an image;

an ability to present a parallax shift for an image;

an ability to present an image having extended depth-of-field;

an ability to process different parts of an image differently depending on depicted distance;

an ability to display a sequence of images over time;

an ability to allow a user to perform at least one of adding, modifying and removing information associated with an image; and

an ability to allow a user to edit an image.

31. The computer program product of claim 24, wherein at least one indicated attribute comprises an indication of image size.

32. The computer program product of claim 24, wherein the computer program code configured to cause at least one processor to transmit the selected at least one available asset to the device comprises:

computer program code configured to cause at least one processor to provide, to the device, at least one link to at least one available asset.

33. The computer program product of claim 24, further comprising computer program code configured to cause at least one processor to, subsequently to transmitting the selected at least one available asset, render and output the image at the device.

34. The computer program product of claim 24, wherein at least one available asset comprises at least one selected from the group consisting of:

light field data;

metadata;

at least one extended depth-of-field image;

at least one sub-aperture image; and

a depth map.

35. The computer program product of claim 24, wherein at least one available asset comprises a focus stack comprising a plurality of images associated with different focus depths.

36. The computer program product of claim 24, wherein at least one available asset comprises a tiled focus stack, comprising a plurality of tiles representing portions of an image, wherein at least two of the tiles are associated with different focus depths.

37. The computer program product of claim 36, wherein the tiled focus stack is generated based on determined focal depths for objects within an image.

38. The computer program product of claim 36, further comprising computer program code configured to cause at least one processor to generate an image by blending at least two tiles in the focus stack with one another.

39. The computer program product of claim 24, wherein at least one available asset is generated from at least one light field image.

40. A computer program product for requesting image-related assets at a device, comprising:

a non-transitory computer-readable storage medium; and

computer program code, encoded on the medium, configured to cause at least one processor to perform the steps of;

determining at least one attribute for display of an image at a device;

determining a set of available image-related assets for display of the image;

based on the determined attribute and the determined available assets, selecting at least one of the available assets;

requesting the selected at least one asset from a server;

receiving the selected at least one asset from the server;

rendering the image using the received at least one asset; and

displaying the rendered image at an output device.

41. The computer program product of claim 40, wherein the computer program code configured to cause at least one processor to determine a set of available image-related assets comprises computer program code configured to cause at least one processor to perform the steps of:

querying a server; and

receiving a response from the server, the response specifying the set of available image-related assets.

42. A system for transmitting image-related assets to a device, comprising:

a processor, configured to receive a request for image-related assets from the device, the request comprising an indication of at least one attribute, and to, based on the attribute, selecting at least one available asset from a plurality of available assets; and

a transmitter, communicatively coupled to the processor, configured to transmit the selected at least one available asset to the device.

43. The system of claim 42, wherein the processor is configured to select at least one available asset by selecting at least one available asset based on suitability of each asset with respect to the indicated attribute.

44. The system of claim 42, wherein at least one indicated attribute specifies at least one hardware characteristic of the device.

45. The system of claim 44, wherein at least one hardware characteristic of the device comprises at least one selected from the group consisting of:

a characteristic of an output device associated with the device;

an indication of available memory;

an indication of available storage;

an indication of processing power associated with the device;

size of a screen associated with the device;

an indication as to whether a graphics processing unit is available for rendering images;

an indication as to a type of input device available to the device; and

an indication as to whether an accelerometer is available.

46. The system of claim 42, wherein at least one indicated attribute specifies at least one characteristic of software running at the device.

47. The system of claim 42, wherein at least one indicated attribute specifies at least one desired feature for displaying at least one image on the device.

48. The system of claim 47, wherein at least one desired feature comprises at least one selected from the group consisting of:

an ability to interact with an image;

an ability to refocus an image at any of a number of different focus depths;

an ability to perform depth-based processing on an image;

an ability to present an image in a three-dimensional format;

an ability to provide stereoscopic viewing of an image;

an ability to present a parallax shift for an image;

an ability to present an image having extended depth-of-field;

an ability to process different parts of an image differently depending on depicted distance;

an ability to display a sequence of images over time;

an ability to allow a user to perform at least one of adding, modifying and removing information associated with an image; and

an ability to allow a user to edit an image.

49. The system of claim 42, wherein at least one indicated attribute comprises an indication of image size.

50. The system of claim 42, wherein the transmitter is configured to transmit the selected at least one available asset to the device by providing, to the device, at least one link to at least one available asset.

51. The system of claim 42, further comprising:

a renderer, communicatively coupled to the transmitter, configured to render the image; and

an output device, communicatively coupled to the renderer, configured to display the image.

52. The system of claim 42, wherein at least one available asset comprises at least one selected from the group consisting of:

light field data;

metadata;

at least one extended depth-of-field image;

at least one sub-aperture image; and

a depth map.

53. The system of claim 42, wherein at least one available asset comprises a focus stack comprising a plurality of images associated with different focus depths.

54. The system of claim 42, wherein at least one available asset comprises a tiled focus stack, comprising a plurality of tiles representing portions of an image, wherein at least two of the tiles are associated with different focus depths.

55. The system of claim 54, wherein the tiled focus stack is generated based on determined focal depths for objects within an image.

56. The system of claim 54, further comprising a renderer, communicatively coupled to the transmitter, configured to generate an image by blending at least two tiles in the focus stack with one another.

57. The system of claim 42, wherein at least one available asset is generated from at least one light field image.

58. A system for requesting image-related assets at a device, comprising:

a processor, configured to perform the steps of;

determining at least one attribute for display of an image at a device;

determining a set of available image-related assets for display of the image; and

based on the determined attribute and the determined available assets, selecting at least one of the available assets;

a communication module, communicatively coupled to the processor, configured to perform the steps of;

requesting the selected at least one asset from a server;

receiving the selected at least one asset from the server;

a renderer, communicatively coupled to the communication module, configured to render the image using the received at least one asset; and

an output device, communicatively coupled to the renderer, configured to display the image.

59. The system of claim 58, wherein the processor is configured to determine a set of available image-related assets by performing the steps of:

querying a server; and

receiving a response from the server, the response specifying the set of available image-related assets.