METHODS AND APPARATUS FOR DETECTING OBJECTS IN PROXIMITY TO A VIEWER AND PRESENTING VISUAL REPRESENTATIONS OF OBJECTS IN A SIMULATED ENVIRONMENT

• US 20170061600A1
• Filed: 08/25/2015
• Published: 03/02/2017
• Est. Priority Date: 08/25/2015
• Status: Active Grant

• Current Assignee: Nevermind Capital LLC
• Original Assignee: NextVR, Inc. (Apple Inc.)
• Inventors: Cole, David, Moss, Alan McKay
• Granted Patent: US 9,836,845 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/65 automatically by game devices or servers from real world data, e.g. measurement in live racing competition
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

Methods and apparatus for determining location of objects surrounding a user of a 3D rendering and display system and indicating the objects to the user while the user views a simulated environment, e.g., on a headmounted display, are described. A sensor, e.g. camera, captures images or senses the physical environment where the user of the system is located. One or more objects in the physical environment are identified, e.g., by recognizing predetermined symbols on the objects and based on stored information indicating a mapping between different symbols and objects. The location of the objects relative to the user'"'"'s location in the physical environment is determined. A simulated environment, including content corresponding to a scene and visual representations of the one or more objects, is displayed. In some embodiments visual representation are displayed in the simulated environment at locations determined based on the location of the objects relative to the user.

Claims

1. A method of operating an interactive system, comprising:

detecting one or more objects in a physical environment where a user of said interactive system is physically located; and

displaying a simulated environment to said user, displaying a simulated environment including displaying, on a display, a visual representation of at least a first detected object.

2. The method of claim 1, further comprising:

prior to said step of displaying a visual representation of the first detected object, operating a sensor to capture information corresponding to the physical environment where the user is physically located.

3. The method of claim 1, further comprising:

determining a location of the first object relative to the user in the physical environment; and

wherein displaying the visual representation of at least the first detected object includes presenting the visual representation of the first object in the simulated environment at a location determined based on the location of said first object relative to the user in the physical environment.

4. The method of claim 3, wherein determining a location of the first object relative to the user includes determining an angle between the first object and a forward viewing direction of the user in the physical environment.

5. The method of claim 3, wherein determining a location of the first object relative to the user further includes determining a distance from the user to the first object.

6. The method of claim 1, wherein detecting one or more objects in said physical environment includes:

recognizing a first predetermined symbol on at least said first object using an optical recognition technique, different predetermined symbols being used for different types of objects in said physical environment, each symbol having a predetermined shape, size or color; and

identifying said first object in said environment based on the recognized first predetermined symbol on said first object.

7. The method of claim 1,

wherein detecting one or more objects in the physical environment includes detecting said first object and a second object, the method further comprising:

prior to said step of displaying a visual representation of the at least the first detected object, determining if the detected first and second objects are included in a list of objects that have been indicated by the user as being of interest, said determining indicating that said first detected object is included in said list while said second object is not included in said list; and

wherein said step of displaying a visual representation of at least the detected first object on the display includes excluding, from being displayed, visual representations of objects which are not on said list of objects, a visual representation of said second object not being displayed.

8. The method of claim 7, further comprising:

prior to said step of displaying a visual representation of the first detected object, determining if any parental control restrictions apply to said user, said determining indicating that a visual representation of said second object should not be displayed to the user due to the parental control restrictions; and

wherein said step of displaying a visual representation of at least the first detected object on the display includes excluding said visual representation of the second detected object from being displayed due to said parental control restrictions.

9. The method of claim 1, further comprising:

receiving mapping information, from said user of said interactive system, indicating a mapping between each of said one or more objects and corresponding symbols, different symbols being used on different types of objects in said physical environment, each symbol having a predetermined shape, size or color; and

storing the received mapping information in memory.

10. The method of claim 1, further comprising:

receiving user information corresponding to said user of the interactive system, said user information including age of the user and optionally parental control information applicable to the user; and

storing the received user information in memory.

11. The method of claim 10, further comprising:

prior to said step of displaying a visual representation of the at least first detected object determining if said visual representation of the at least first detected object is allowed to be displayed based on the received user information.

12. An interactive system, comprising:

an object detector configured to detect one or more objects in a physical environment where a user of said interactive system is physically located;

a display device; and

a display controller configured to control displaying of a simulated environment to said user on said display device, said display controller being configured to control displaying of a visual representation of at least a first detected object on said display device as part of controlling displaying of the simulated environment to said user.

13. The system of claim 12, further comprising:

a sensor; and

a controller configured to control said sensor to capture information corresponding to the physical environment where the user is physically located.

14. The system of claim 13, wherein said sensor is a camera and wherein said controller is configured to control said camera to capture an image of the environment as part of being configured to control said sensor to capture information corresponding to the physical environment.

15. The system of claim 12, further comprising:

a location determinator configured to determine a location of the first object relative to the user in the physical environment; and

wherein said display controller is further configured to present the visual representation of the first object in the simulated environment at a location determined based on the location of said first object relative to the user in the physical environment, as part of being configured to control displaying of the visual representation of at least the first detected object.

16. The system of claim 12, wherein said object detector includes:

a symbol recognizer configured to recognize a first predetermined symbol on at least said first object using an optical recognition technique, different predetermined symbols being used for different types of objects in said physical environment, each symbol having a predetermined shape, size or color; and

an object identifier configured to identify said first object in said environment based on the recognized first predetermined symbol on said first object.

17. The system of claim 12, wherein said object detector detects said first object and a second object;

wherein the system further comprises a determiner configured to determine if the first and second detected objects are included in a list of objects that have been indicated by the user as being of interest; and

wherein said display controller is further configured to exclude, from being displayed, visual representations of objects which are not on said list of objects, said display controller being configured to exclude a visual representation of said second object from being displayed when said determiner determines that said second object is not included in said list of objects.

18. The system of claim 17, wherein said determiner is further configured to determine if any parental control restrictions apply to said user, said determiner determining that a visual representation of said second object should not be displayed to the user due to the parental control restrictions;

and

wherein said display controller is further configured to exclude, from being displayed, said visual representation of the second detected due to said parental control restrictions applicable to said user.

19. The system of claim 12, further comprising:

a receiver configured to receive mapping information, from said user of said interactive system, indicating a mapping between each of said one or more objects and corresponding symbols, different symbols being used on different types of objects in said physical environment, each symbol having a predetermined shape, size or color; and

a memory configured to store the received mapping information.

20. A non-transitory computer readable medium including processor executable instructions which when executed by a processor control an interactive system to:

detect one or more objects in a physical environment where a user of said interactive system is physically located; and

display a simulated environment to said user on a display device, displaying the simulated environment including displaying a visual representation of at least a first detected object on said display device.