Methods and apparatus for detecting objects in proximity to a viewer and presenting visual representations of objects in a simulated environment

• US 9,836,845 B2
• Filed: 08/25/2015
• Issued: 12/05/2017
• Est. Priority Date: 08/25/2015
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, Moss, Alan McKay
• Primary Examiner(s): Mehmood, Jennifer
• Assistant Examiner(s): Azongha, Sardis F
• Application Number:
  US14/835,249
• Publication Number:
  US 20170061600A1
• Time in Patent Office: 833 Days
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

Methods and apparatus for determining location of objects surrounding a user of a 3D rendering and display system and indicating the objects to the user while the user views a simulated environment, e.g., on a headmounted display, are described. A sensor, e.g. camera, captures images or senses the physical environment where the user of the system is located. One or more objects in the physical environment are identified, e.g., by recognizing predetermined symbols on the objects and based on stored information indicating a mapping between different symbols and objects. The location of the objects relative to the user'"'"'s location in the physical environment is determined. A simulated environment, including content corresponding to a scene and visual representations of the one or more objects, is displayed. In some embodiments visual representation are displayed in the simulated environment at locations determined based on the location of the objects relative to the user.

Claims

1. A method of operating an interactive system including a head mounted cell phone, comprising:

operating a camera of the head mounted cell phone to capture an image of a physical environment where a user of said interactive system is physically located;

operating a processor in the cell phone to process the captured image to detect one or more objects in the physical environment where the user of said interactive system is physically located; and

displaying, on a display device of said head mounted cell phone, a simulated environment to said user, said displaying a simulated environment including displaying on the display device of said head mounted cell phone multiple frames between the capture of the image of the physical environment and a capture of another image of the physical environment, said displaying a simulated environment including displaying a visual representation of at least a first detected object as part of said simulated environment.

2. The method of claim 1, wherein said camera is positioned in said head mounted cell phone facing away from the user and with the display device of said head mounted cell phone facing the eyes of the user.

3. The method of claim 2, further comprising:

performing on a recurring basis, at a rate which is lower than a rate at which frames are displayed on the display of said head mounted cell phone, said steps of;

i) operating said camera of the head mounted cell phone to capture an image of a physical environment where a user of said interactive system is physically located and ii) operating said processor in the cell phone to detect one or more objects in the physical environment where the user of said interactive system is physically located, at least 30 frames being displayed between each capture of an image by the camera of the cell phone.

4. The method of claim 2, further comprising:

operating the processor in the cell phone to determine a location of the first object relative to the user in the physical environment; and

wherein displaying the visual representation of at least the first detected object includes presenting the visual representation of the first object in the simulated environment at a location determined based on the location of said first object relative to the user in the physical environment.

5. The method of claim 4, wherein operating the processor in the cell phone to determine the location of the first object relative to the user further includes determining a distance from the user to the first object.

6. The method of claim 2 wherein displaying, on a display device of said head mounted cell phone, a simulated environment to said user includes:

displaying different images on left and right sides of said display device of said head mounted cell phone.

7. The method of claim 1, wherein operating the processor in the cell phone to detect one or more objects in said physical environment includes:

recognizing a first predetermined symbol on at least said first object using an optical recognition technique, different predetermined symbols being used for different types of objects in said physical environment, each symbol having a predetermined shape, size or color; and

identifying said first object in said environment based on the recognized first predetermined symbol on said first object.

8. The method of claim 1,

wherein operating a processor in the cell phone to detect one or more objects in the physical environment includes detecting said first object and a second object, the method further comprising:

prior to said step of displaying a visual representation of the at least the first detected object, determining if the detected first and second objects are included in a list of objects that have been indicated by the user as being of interest, said determining indicating that said first detected object is included in said list while said second object is not included in said list; and

wherein said step of displaying a visual representation of at least the first detected object on the display includes excluding, from being displayed, visual representations of detected objects which are not on said list of objects that have been indicated by the user as being of interest, a visual representation of said second object not being displayed.

9. The method of claim 8, further comprising:

prior to said step of displaying a visual representation of the first detected object, determining if any parental control restrictions apply to said user, said determining indicating that a visual representation of said second object should not be displayed to the user due to the parental control restrictions; and

wherein said step of displaying, on the display of said head mounted cell phone, a visual representation of at least a first detected object as part of said simulated environment includes excluding said visual representation of the second detected object from being displayed due to said parental control restrictions.

10. The method of claim 1, further comprising:

receiving mapping information, from said user of said interactive system, indicating a mapping between each of said one or more objects and corresponding symbols, different symbols being used on different types of objects in said physical environment, each symbol having a predetermined shape, size or color; and

storing the received mapping information in memory.

11. The method of claim 1, further comprising:

receiving user information corresponding to said user of the interactive system, said user information including age of the user and optionally parental control information applicable to the user; and

storing the received user information in memory.

12. The method of claim 11, further comprising:

prior to said step of displaying a visual representation of the at least first detected object determining if said visual representation of the at least first detected object is allowed to be displayed based on the received user information.

13. An interactive system, comprising:

a head mount including a slot;

a cell phone inserted into said slot, the cell phone including;

i) a camera facing away from the head mount, ii) a display facing towards a position where a user'"'"'s eyes are located when the head mount is positioned on a head of a user and iii) a processor;

and wherein said processor of the cell phone is configured to;

process an image captured by the camera of said cell phone to detect one or more objects in a physical environment where a user of said interactive system is physically located; and

control displaying of a simulated environment to said user on said display, said processor being configured to control displaying of a visual representation of at least a first detected object on said display as part of being configured to control as part of displaying of the simulated environment to said user.

14. The system of claim 13, further comprising:

a sensor; and

a controller configured to control said sensor to capture information corresponding to the physical environment where the user is physically located.

15. The system of claim 14, wherein said sensor is included in said camera and wherein said controller is configured to control said camera to capture an image of the environment as part of being configured to control said sensor to capture information corresponding to the physical environment.

16. The system of claim 13, further comprising:

a location determinator configured to determine a location of the first detected object relative to the user in the physical environment; and

wherein said processor is further configured to present the visual representation of the first detected object in the simulated environment at a location determined based on the location of said first detected object relative to the user in the physical environment, as part of being configured to control displaying of the visual representation of at least the first detected object.

17. The system of claim 13, wherein said processor is further configured to:

recognize a first predetermined symbol on at least said first detected object using an optical recognition technique, different predetermined symbols being used for different types of objects in said physical environment, each symbol having a predetermined shape, size or color; and

identify said first detected object in said environment based on the recognized first predetermined symbol on said first detected object.

18. The system of claim 13, wherein said processor detects said first object and a second object;

wherein the system further comprises a determiner configured to determine if the first and second detected objects are included in a list of objects that have been indicated by the user as being of interest; and

wherein said processor is further configured to exclude, from being displayed, visual representations of objects which are not on said list of objects, said display controller being configured to exclude a visual representation of said second object from being displayed when said determiner determines that said second object is not included in said list of objects.

19. The system of claim 18, wherein said determiner is further configured to determine if any parental control restrictions apply to said user, said determiner determining that a visual representation of said second object should not be displayed to the user due to the parental control restrictions;

and

wherein said processor is further configured to exclude, from being displayed, said visual representation of the second detected object due to said parental control restrictions applicable to said user.

20. The system of claim 13, further comprising:

a receiver configured to receive mapping information, from said user of said interactive system, indicating a mapping between each of said one or more objects and corresponding symbols, different symbols being used on different types of objects in said physical environment, each symbol having a predetermined shape, size or color; and

a memory configured to store the received mapping information.

21. A non-transitory computer readable medium including processor executable instructions which when executed by a processor of a head mounted cell phone of an interactive system control the interactive system to:

operate a camera of the head mounted cell phone to capture an image of a physical environment where a user of said interactive system is physically located;

operate the processor in the cell phone to process the captured image to detect one or more objects in the physical environment where the user of said interactive system is physically located; and

display, on a display device of said head mounted cell phone, a simulated environment to said user, said displaying a simulated environment including displaying on the display device of said head mounted cell phone multiple frames between the capture of the image of the physical environment and a capture of another image of the physical environment, said displaying a simulated environment including displaying a visual representation of at least a first detected object as part of said simulated environment.