Hand-based interaction in virtually shared workspaces

• US 9,978,178 B1
• Filed: 10/25/2012
• Issued: 05/22/2018
• Est. Priority Date: 10/25/2012
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Shepard, Deborah May
• Primary Examiner(s): Yang, Ryan R
• Application Number:
  US13/660,832
• Time in Patent Office: 2,035 Days
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2219/024 Multi-user, collaborative environment

Abstract

When sharing visible content on multiple display surfaces associated with different users, the hand of a user may be detected and analyzed when placed over the user'"'"'s display surface. Characteristics of the user'"'"'s hand, including things such as position, orientation, color, shape, and texture, may be shared with other users. Based on these characteristics, a representation of the user'"'"'s hand may be depicted over the shared content on the display surfaces of the other users.

Claims

1. A computing device, comprising:

a processor;

an imaging device;

memory, accessible by the processor and storing instructions that are executable by the processor to perform acts comprising;

displaying content on a first display surface associated with a first user, wherein the content is displayed concurrently on one or more other display surfaces that are associated respectively with one or more other users;

receiving an image from the imaging device of an area that includes the first display surface, including the content displayed on the first display surface, and a first user hand positioned over at least a portion of the first display surface;

detecting one or more edges of the first display in the image received from the imaging device;

identifying coordinates of the first display surface based at least in part on the one or more edges;

determining a position of the first user hand relative to the first display surface, based at least in part on the coordinates;

determining the position of the first user hand relative to the content;

determining one or more first characteristics representative of the first user hand based at least in part on the image;

sending the one or more first characteristics representative of the first user hand for use in displaying a first virtual hand in conjunction with the content on the one or more other display surfaces;

receiving information indicating one or more second characteristics representative of a second user hand that is positioned over one of the one or more other display surfaces; and

displaying a second virtual hand representative of the second user hand in conjunction with the content on the first display surface, wherein the second virtual hand includes the one or more second characteristics representative of the second user hand.

2. The computing device of claim 1, wherein displaying the second virtual hand comprises superimposing the second virtual hand on the content.

3. The computing device of claim 1, wherein the second virtual hand comprises a photographic representation of the second user hand.

4. The computing device of claim 1, wherein the image comprises one or more of:

a depth map;

a two-dimensional greyscale image;

or

a two-dimensional color image.

5. The computing device of claim 1, wherein the one or more first characteristics of the first and second user hands comprise one or more visual characteristics of the first and second user hands and detected positions of the first and second user hands relative to the content.

6. The computing device of claim 1, wherein the one or more first characteristics including at least one skin tone.

7. A method, comprising:

causing display of visual content on first user display surface during a first period of time, the first user display surface associated with a first user;

causing display of the visual content on a second user display surface during the first period of time, the second user display surface associated with a second user;

receiving an image of an area that includes the first user display surface and a first user hand;

detecting one or more edges of the first user display surface in the image;

identifying coordinates of the first user display surface based at least in part on the one or more edges;

detecting that the first user hand is positioned over the first user display surface, based at least in part on the coordinates;

determining that the first user hand is positioned over at least a portion of the visible content displayed on the first user display surface, based at least in part on the coordinates;

determining one or more characteristics representative of a skin tone associated with the first user hand based at least in part on the image, wherein the one or more characteristics include a position of the first user hand relative to the visual content displayed on the first user display surface; and

displaying a virtual hand representative of the first user hand on the second user display surface at the position representative of the first user hand relative to the visual content.

8. The method of claim 7, wherein displaying the virtual hand comprises superimposing the virtual hand on the visual content on the second user display surface.

9. The method of claim 7, wherein displaying the virtual hand comprises displaying a photographic representation of the user hand on the second user display surface.

10. The method of claim 7, wherein the image comprises one or more of:

a depth map;

a two-dimensional greyscale image;

or

a two-dimensional color image.

11. The method of claim 7, wherein the one or more characteristics comprise one or more of the following:

one or more visual characteristics;

or

one or more color characteristics.

12. The method of claim 11, wherein the one or more visual characteristics comprise one or more shape characteristics.

13. A method comprising:

receiving content that is to be concurrently displayed on a first user display surface and a second user display surfaces that are associated respectively with different users;

displaying the content on the first user display surface;

receiving an image of a region that includes the first user display surface and a physical object that is positioned over the first user display surface;

detecting one or more edges of the first user display surface in the image;

identifying coordinates of the first user display surface based at least in part on the one or more edges;

determining that the physical object is positioned over at least a portion of the content, based at least in part on the coordinates;

determining one or more characteristics representative of at least one texture of the physical object that is positioned over the first user display surface based, at least in part, on the image; and

providing the one or more characteristics representative of the at least one texture of the physical object for use in displaying a virtual object representative of the physical object in conjunction with the concurrently displayed content on the second user display surface.

14. The method of claim 13, wherein the one or more characteristics of the physical object comprise a position of the physical object relative to the concurrently displayed content.

15. The method of claim 13, wherein the physical object comprises one or more of the following:

a user hand;

or

an article held by the user hand.

16. The method of claim 13, further comprising determining that the physical object is a user hand and determining a position of the hand relative to the first user display surface.

17. The method of claim 16, wherein the image comprises an optical image that indicates color information regarding the physical object.

18. The method of claim 16, wherein the image comprises a depth map that indicates three-dimensional information regarding the physical object.

19. A method comprising:

receiving content that is to be displayed concurrently on a first user display surface and a second user display surface, the first user display surface associated with a first location and the second user display surface associated with a second location;

displaying the content on the first user display surface;

detecting one or more features in the content displayed on the first user display surface;

identifying coordinates of the first user display surface based at least in part on the one or more features;

displaying the content on the second user display surface;

receiving information indicating one or more physical characteristics representative of a physical object positioned over the first display surface, wherein the physical object is detected based at least in part on the coordinates, wherein the physical object comprises one or more of a user hand or an article held by the user hand; and

displaying a virtual object representative of the physical object on the second user display surface, wherein the virtual object has the one or more characteristics representative of the physical object.

20. The method of claim 19, wherein displaying the virtual object comprises superimposing the virtual object on the concurrently displayed content on the second user display surface.

21. The method of claim 19, wherein:

the one or more characteristics indicate a position of the physical object relative to the displayed content; and

displaying the virtual object comprises displaying the virtual object at the indicated representative position relative to the concurrently displayed content.

22. The method of claim 19, wherein the one or more characteristics comprise an image of the physical object, the acts further comprising creating the virtual object based at least in part on the image of the physical object.

23. The method of claim 19, wherein the one or more characteristics comprise one or more shape characteristics of the physical object, the acts further comprising creating the virtual object based at least in part on the shape characteristics of the physical object.