Forward and backward looking vision system

• US 9,465,484 B1
• Filed: 03/11/2013
• Issued: 10/11/2016
• Est. Priority Date: 03/11/2013
• Status: Expired due to Fees

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Kamarshi, Vijay, Liu, Qiang
• Primary Examiner(s): Patel, Kumar
• Assistant Examiner(s): LAM, VINH TANG
• Application Number:
  US13/793,276
• Time in Patent Office: 1,310 Days
• Field of Search: 345173-184, 178 1801- 3004, 352114-115
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/215 comprising means for detecting acoustic signals, e.g. using a microphone
• CPC Class Codes: A63F 13/26 having at least one additional display device, e.g. on the game controller or outside a game booth
• CPC Class Codes: A63F 2003/0418 with a grid, e.g. 'Sudoku'-type games
• CPC Class Codes: A63F 2003/0428 Crosswords
• CPC Class Codes: A63F 2009/2461 Projection of a two-dimensional real image
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386

Abstract

A vision system associated with a projection system includes multiple optical pathways. For instance, when the projection system projects an image onto a generally vertical surface, the vision system may operate in a rear sensing mode, such as for detecting one or more gestures made by a user located behind the projection system. Alternatively, when the projection system projects the image onto a generally horizontal surface the vision system may operate in a front sensing mode for detecting gestures made by a user located in front of the projection system. One or more thresholds may be established for switching between the front sensing mode and the rear sensing mode based on orientation information. As another example, the vision system may be operated in both the front sensing mode and the rear sensing mode contemporaneously.

Claims

1. A system comprising:

a projector to project light corresponding to an image in a projection direction toward a display surface;

one or more processors;

one or more computer-readable media;

at least one light emitter configured to emit non-visible light;

a light sensor to detect the non-visible light; and

one or more modules maintained on the one or more computer-readable media which, when executed by the one or more processors, cause the one or more processors to perform operations comprising;

ascertaining whether the projector is (1) oriented so that the projection direction is within a first range of directions that includes a substantially horizontal direction or (2) oriented so that the projection direction is within a second range of directions that includes a substantially vertical direction;

based at least partly on a first determination that the projector is oriented so that the projection direction is within the first range of directions, emitting first non-visible light in a first direction away from the projection direction, receiving at least a first portion of the first non-visible light reflected from one or more first objects in an environment, and detecting at least one first gesture in a first area outside of the first range of directions relative to the projector based on first information generated from the first non-visible light reflected from the one or more first objects in the environment;

based at least partly on a second determination that the projector is oriented so that the projection direction is within the second range of directions, emitting second non-visible light in a second direction toward the projection direction, receiving at least a second portion of the second non-visible light reflected from one or more second objects in the environment, and detecting at least one second gesture in a second area within the second range of directions relative to the projector based on second information generated from the second non-visible light reflected from the one or more second objects in the environment.

2. The system as recited in claim 1, wherein the at least one light emitter comprises:

a first light emitter positioned to emit the second non-visible light in the second direction toward the projection direction; and

a second light emitter positioned to emit the first non-visible light in the first direction away from the projection direction.

3. The system as recited in claim 1, wherein the operations further comprise performing an operation based at least in part on at least one of the at least one first gesture or the at least one second gesture.

4. The system as recited in claim 1, wherein the projector shares an optical path with the second non-visible light reflected from the one or more second objects in the environment at a time at which the second non-visible light is emitted in the second direction toward the projection direction.

5. A method comprising:

using a projector to project an image in a projection direction onto a surface;

determining that the projection direction relative to the surface is in a substantially vertical direction;

based at least partly on determining that the projection direction relative to the surface is in the substantially vertical direction, emitting first non-visible light into a first gesture area toward the projection direction;

receiving a first reflected portion of the first non-visible light;

based at least partly on the first reflected portion of the first non-visible light, detecting one or more first gestures made in the first gesture area;

determining that the projection direction relative to the surface is in a substantially horizontal direction;

based at least partly on determining that the projection direction relative to the surface is in the substantially horizontal direction, emitting second non-visible light into a second gesture area away from the projection direction;

receiving a second reflected portion of the second non-visible light; and

based at least partly on the second reflected portion of the second non-visible light, detecting one or more second gestures made in the second gesture area.

6. The method as recited in claim 5, wherein the substantially horizontal direction is on an opposite side of the projector from the surface in a direction away from the projection direction.

7. The method as recited in claim 5, further comprising performing an operation based at least partly on at least one of the one or more first gestures or the one or more second gestures.

8. The method as recited in claim 5, wherein determining the projection direction is based at least in part on information from a sensor associated with the projector, wherein the sensor comprises at least one of:

an accelerometer;

or

a potentiometer.

9. The method as recited in claim 5, wherein:

the projector projects the image through at least one lens; and

a light sensor receives the first reflected portion of the first non-visible light through the at least one lens at a time at which the first non-visible light is emitted in the projection direction.

10. The method as recited in claim 5, wherein:

detecting the one or more first gestures made in the first gesture area comprises receiving the first reflected portion of the first non-visible light that is reflected from at least a first portion of a body of a user; and

detecting the one or more second gestures made in the second gesture area comprises receiving the second reflected portion of the second non-visible light that is reflected from at least a second portion of the body of the user.

11. The method as recited in claim 10, further comprising emitting the first non-visible light into the first gesture area and emitting the second non-visible light into the second gesture area concurrently.

12. The method as recited in claim 5, further comprising:

determining a change in the projection direction relative to the surface from the substantially vertical direction to the substantially horizontal direction; and

upon determining the change in the projection direction, detecting the one or more second gestures made in second gesture area.

13. The method as recited in claim 5, further comprising operating a camera to receive visible light from a direction that is determined based at least in part on the projection direction relative to the surface.

14. A system comprising:

a first lens to pass first projected content to and to receive first reflected non-visible light from a first field of view in a substantially vertical direction;

a second lens to pass second projected content to and to receive second reflected non-visible light from a second field of view in a substantially horizontal direction;

at least one light emitter to project light in a projection direction relative to a surface, the at least one light emitter to;

emit first non-visible light in the substantially vertical direction into the first field of view, the substantially vertical direction being the projection direction; and

emit second non-visible light in the substantially horizontal direction into the second field of view, the substantially horizontal direction being away from the projection direction;

at least one light sensor to receive the first reflected non-visible light through the first lens in the substantially vertical direction and to receive the second reflected non-visible light through the second lens in the substantially horizontal direction; and

a processor configured to receive information related to at least one of one or more first gestures detected based at least partly on the first reflected non-visible light or one or more second gestures detected based at least partly on the second reflected non-visible light.

15. The system as recited in claim 14, wherein the at least one light sensor comprises a single light sensor to obtain first images from the first reflected non-visible light received through the first lens, wherein the first images are interleaved with second images obtained from the second reflected non-visible light received through the second lens.

16. The system as recited in claim 14, wherein the processor is further configured to alternately flash the first non-visible light in the substantially vertical direction and the second non-visible light in the substantially horizontal direction.

17. The system as recited in claim 14, further comprising a projector to project the light corresponding to an image through at least one of the first lens or the second lens.

18. The system as recited in claim 14, wherein the projector is further configured to:

determining a change in the projection direction relative to the surface from the substantially vertical direction to the substantially horizontal direction; and

upon determining the change in the projection direction, detecting the one or more second gestures.

19. One or more non-transitory computer-readable media comprising instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising:

determining at least one of;

first orientation information corresponding to at least one of a projector, a projection display surface, or the projector relative to the projection display surface;

or

first presence information indicating a presence of at least a portion of a user within a field of view in a projection direction;

based at least in part on at least one of the first orientation information or the first presence information, emitting non-visible light in a first direction toward the projection direction of the projector;

determining at least one of;

second orientation information corresponding to at least one of the projector, the projection display surface, or the projector relative to the projection display surface;

or

second presence information indicating a presence of at least a portion of the user or another user within a second field of view away from the projection direction and

based at least in part on at least one of the second orientation information or the second presence information, emitting the non-visible light in a second direction away from the projection direction of the projector.

20. The one or more non-transitory computer-readable media as recited in claim 19, wherein the determining the first orientation information is based on at least one of:

a first orientation of an axis of projection of the projector;

a second orientation of the projection display surface that receives a projected image from the projector;

a size of the projected image;

or

a distance from the projector to the projection display surface.

21. The one or more non-transitory computer-readable media as recited in claim 19, the operations further comprising:

receiving, by a light sensor, a reflected portion of the non-visible light;

determining, at least in part from the reflected portion of the non-visible light, a gesture made by the user or the other user; and

performing an operation based at least in part on the gesture.

22. The one or more non-transitory computer-readable media as recited in claim 19, the operations further comprising operating at least one camera for determining the first presence information and the second presence information.

23. A method comprising:

using a projector to project an image in a projection direction;

determining a first orientation of a projection surface relative to the projector;

emitting first non-visible light in a first direction toward the projection direction if based at least partly on a determination that an orientation of the projection surface is substantially in the first orientation;

receiving, by at least one light sensor, a first reflected portion of the first non-visible light through a first lens;

determining a change from the first orientation of the projection surface relative to the projector to a second orientation of the projection surface relative to the projector;

emitting second non-visible light in a second direction away from the projection direction based at least in part on the change; and

receiving, by the at least one light sensor, the first reflected portion of the first non-visible light through the first lens and a second reflected portion of the second non-visible light through a second lens.

24. The method as recited in claim 23, further comprising detecting a gesture based at least in part on at least one of the first reflected portion of the first non-visible light or the second reflected portion of the second non-visible light.

25. The method as recited in claim 23, wherein using the projector to project the image further comprises using the projector to project the image through the first lens through which the first reflected portion of the first non-visible light is received.

26. The method as recited in claim 23, wherein determining the change from the first orientation to the second orientation comprises at least one of:

determining a first change in a distance from the projector to the projection surface;

or

determining a second change in an angle between the projector and the projection surface.