Methods and apparatus for communicating and/or using frames including a captured image and/or including additional image content

• US 10,567,733 B2
• Filed: 03/09/2018
• Issued: 02/18/2020
• Est. Priority Date: 03/06/2017
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, Moss, Alan McKay, Medina, Hector M, Sheridan, Ryan Michael
• Primary Examiner(s): Alcon, Fernando
• Application Number:
  US15/917,569
• Publication Number:
  US 20180262745A1
• Time in Patent Office: 711 Days
• Field of Search: None
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/183 On-screen display [OSD] information, e.g. subtitles or menus
• CPC Class Codes: H04N 13/189 Recording image signals; Reproducing recorded image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor
• CPC Class Codes: H04N 13/293 Generating mixed stereoscopic images; Generating mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Methods and apparatus for packing images into a frame and/or including additional content and/or graphics are described. A composite image is generated including at least one image in addition to another image and/or additional image content. A playback device received an encoded frame including a captured image of a portion of and environment and the additional image content. The additional image content is combined with or used to replace a portion of the image of the environment during rendering. Alpha value mask information is communicated to the playback device to provide alpha values for use in image combining. Alpha values are communicated as pixel values in the encoded frame or as additional information. One or more mesh models and/or information on how to map image content to the one or more mesh models is communicated to the playback device for use in rendering image content recovered from a frame.

Claims

1. A playback method implemented by a content playback device, the method comprising:

receiving, at the content playback device, content including a first frame, said first frame including a first image captured by a first camera and first additional image content from an additional content source, said first additional image content occupying a portion of said first frame which is smaller than the first image and providing pixel values to replace pixel values of said first image during playback or be blended with pixel values of said first image during playback;

receiving, at the content playback device, a first mesh model of a first portion of an environment to which said first image is to be applied as a texture; and

generating, at the content playback device, a first output image to be displayed, said step of generating a first output image to be displayed including performing a rendering operation to generate said first output image, said rendering operation including generating said first output image using said first image as a first texture, said first additional image content as a second texture and said first mesh model as a model of an environment to which at least one of said first and second textures is applied, said generating a first output image including;

i) combining one pixel value of the first image included in the first frame with one pixel value of the first additional image content to generate one pixel value of said first output image or ii) replacing said one pixel value of the first image included in the first frame with said one pixel value of the first additional image content to generate said one pixel value of said first output image.

2. The playback method of claim 1,

wherein said first image is an image captured by said first camera with a fisheye lens, said first image occupying a non-rectangular portion of the first frame;

and

wherein said first additional image content is positioned in an edge corner of said first frame which is outside the portion of the first frame occupied by said first image captured by the first camera.

3. The playback method of claim 2, wherein generating a first output image to be displayed includes:

combining, by performing a blending operation, pixel values from said first image and pixel values from said first additional image content to generate pixel values of a portion of the generated output image.

4. The playback method of claim 3,

wherein said blending operation is an alpha blending operation, the method further comprising:

recovering an alpha value mask from said received content to be used in combining pixel values included in said first additional image content and said first image content, said alpha value mask including alpha values communicated;

i) as pixel values in said first frame or ii) in additional image information communicated with said first frame.

5. The playback method of claim 4, wherein said received content is encoded content, the method further comprising:

decoding said received content.

7. The playback method of claim 4, further comprising:

using the recovered alpha values during said combining step, wherein using the recovered alpha values includes multiplying a first individual pixel value in said first additional image content with a corresponding first additional image content alpha value obtained from said alpha value mask to generate a modified additional image pixel value.

8. The playback method of claim 7, wherein using the recovered values during the combining step further includes:

multiplying a first individual pixel value of the first image which is to be combined with said first individual image content with a corresponding first image content alpha value obtained from said alpha value mask to generate a modified first image pixel value; and

summing the modified additional image pixel value and modified first image pixel value to generate a pixel value of the first output image.

9. The playback method of claim 7, wherein said first additional image content alpha value and said first image content alpha value are communicated in said first frame as different pixel values.

10. The playback method of claim 9,

wherein said alpha value mask corresponds to a second additional image portion of the first frame which is equal to or smaller than a first additional image portion of the first frame used to communicate said first additional image content;

and

wherein first additional image content alpha values are communicated in said first frame as pixel values of a first type and said first image content alpha values are communicated as pixel values of a second type, at least one of said first and second types of pixel values being first color pixel values, with the other one of said first and second types of pixel values being pixel values of a second color or luminance pixel values, said second color being different from said first color.

11. The playback method of claim 7, wherein said alpha value mask includes alpha values which produce a greater level of transparency of said additional image content near outer edges of said first additional image content than at the center of said first additional image content in the generated image.

12. The playback method of claim 11, further comprising:

receiving first texture mapping information indicating how to apply said first image content to the first mesh model as the first texture; and

receiving second texture mapping information indicating how to apply said first additional image content to a second mesh model as the second texture or how to apply said first additional image content to the first mesh model as the second texture.

13. The playback method of claim 12, further comprising:

receiving location information indicating where said second mesh model is positioned in an environment which is modeled by said first mesh model; and

determining which portions of said first image content and said first additional image content are to be combined based on the received location information.

6. A playback method, the method comprising:

receiving content including a first frame, said first frame including a first image captured by a first camera and first additional image content from an additional content source, said first additional image content occupying a portion of said first frame which is smaller than the first image, said first image being an image captured by said first camera with a fisheye lens, said first image occupying a non-rectangular portion of the first frame, said first additional image content is positioned in an edge corner of said first frame which is outside the portion of the first frame occupied by said first image captured by the first camera;

receiving a first mesh model of a first portion of an environment to which said first image is to be applied as a texture;

recovering an alpha value mask from said received content to be used in combining pixel values included in said first additional image content and said first image content, said alpha value mask including alpha values communicated as pixel values in said first frame, said recovering the alpha value mask including recovering pixel values from a portion of said first frame used to communicate the alpha value mask; and

generating a first output image to be displayed, said step of generating a first output image to be displayed including performing a rendering operation to generate said first output image, said rendering operation including generating said first output image using said first image as a first texture, said first additional image content as a second texture and said first mesh model as a model of an environment to which at least one of said first and second textures is applied, generating the first output image to be displayed including combining pixel values from said first image and pixel values from said first additional image content to generate pixel values of a portion of the generated output image.

14. A content playback device, the content playback device comprising:

memory;

a processor configured to;

control the content playback device to receive content including a first frame, said first frame including a first image captured by a first camera and first additional image content from an additional content source, said first additional image content occupying a portion of said first frame which is smaller than the first image and providing pixel values to replace pixel values of said first image during playback or be blended with pixel values of said first image during playback;

control the content playback device to receive a first mesh model of a first portion of an environment to which said first image is to be applied as a texture; and

generate, at the content playback device, a first output image to be displayed, said step of generating a first output image to be displayed including performing a rendering operation to generate said first output image, said rendering operation including generating said first output image using said first image as a first texture, said first additional image content as a second texture and said first mesh model as a model of an environment to which at least one of said first and second textures is applied, said generating a first output image including;

i) combining one pixel value of the first image included in the first frame with one pixel value of the first additional image content to generate one pixel value of said first output image or ii) replacing said one pixel value of the first image included in the first frame with said one pixel value of the first additional image content to generate said one pixel value of said first output image; and

a display configured to display the first output image.

15. The content playback apparatus of claim 14, wherein said first additional image content is included in a first additional image portion of said first frame.

16. The content playback device of claim 14, wherein said first image is an image captured by said first camera with a fisheye lens, said first image occupying a non-rectangular portion of the first frame;

and wherein said first additional image content is positioned in an edge corner of said first frame which is outside the portion of the first frame occupied by said first image captured by the first camera.

17. The content playback device of claim 16, wherein the processor is further configured, as part of generating a first output image to be displayed, to:

combine pixel values from said first image and pixel values from said first additional image content to generate pixel values of a portion of the generated output image.

18. The content playback device of claim 14, wherein said display is a stereoscopic display configured to display left eye images on a first side of said display and right eye images on a second side of the display.

19. The content playback device of claim 18, where wherein said display is a head mounted display.

20. A non-transitory computer readable medium comprising computer executable instructions which when executed by a processor of a content playback device control the content playback device to:

receive, at the content playback device, content including a first frame, said first frame including a first image captured by a first camera and first additional image content from an additional content source, said first additional image content occupying a portion of said first frame which is smaller than the first image and providing pixel values to replace pixel values of said first image during playback or be blended with pixel values of said first image during playback;

receive, at the content playback device, a first mesh model of a first portion of an environment to which said first image is to be applied as a texture; and

generate, at the content playback device, a first output image to be displayed, said step of generating a first output image to be displayed including performing a rendering operation to generate said first output image, said rendering operation including generating said first output image using said first image as a first texture, said first additional image content as a second texture and said first mesh model as a model of an environment to which at least one of said first and second textures is applied, said generating a first output image including;

i) combining one pixel value of the first image included in the first frame with one pixel value of the first additional image content to generate one pixel value of said first output image or ii) replacing said one pixel value of the first image included in the first frame with said one pixel value of the first additional image content to generate said one pixel value of said first output image.