Microsoft's Copilot Vision AI now has the ability to view your entire desktop, detailing its functions
Microsoft has rolled out an update to its Copilot Vision AI on Windows 11 PCs, enhancing the capabilities of the AI to provide contextual tips, insights, and guidance in real-time. The new feature, known as Desktop Share, allows users to share their full desktop or a chosen application window with Copilot, enabling the AI to understand and interact with the content displayed on the screen[1][2].
Key Capabilities and Features of Copilot Vision AI:
* Whole Desktop Sharing: Users can explicitly share their full desktop or a chosen application window with Copilot by clicking the glasses icon in the Copilot interface, allowing the AI to "see" everything displayed on their screen[1][2]. * Real-time Interaction: Copilot engages with users through natural language, offering spoken coaching, suggestions, or explanations related to what is visible on the screen, whether it's creative editing, document improvements, or gaming support[1][2]. * Privacy-conscious Activation: Unlike always-on background recording features, Copilot Vision only activates when a user chooses to share their screen, ensuring user control and privacy[2]. * On-device AI Processing: Some Copilot+ AI features can run locally on compatible hardware without needing an internet connection, which also helps keep user data confined to the device[4]. * Integrated Creative Tools: Enhanced creative capabilities such as Cocreator, Generative Fill in Paint, Super Resolution, Image Creator, and Restyle features in the Photos app are supported on Copilot+ hardware for creative workflows[4].
Compared to Google's Gemini Live and Apple Intelligence, Copilot Vision offers a unique approach to screen analysis and interactive assistance, with a strong emphasis on user control and privacy[3]. The Desktop Share feature works by sharing your desktop with Copilot in real-time, and users can turn it off by selecting "Stop" in the app.
Microsoft is looking to give Windows 11 PCs even more of an AI advantage with the Copilot Vision update, which offers useful content based on the apps, documents, or websites in browsers users are looking at. To join the Windows Insiders program and try out the Copilot Vision update, navigate to Settings > Windows Update > Windows Insider Program and click "Get Started". The update is gradually rolling out to Windows Insider channels, and users can try out 7 useful Copilot prompts to make the most of it.
[1] https://www.microsoft.com/en-us/windows/blog/copilot-vision-ai-update-helps-windows-11-pc-users-get-more-done/ [2] https://www.microsoft.com/en-us/windows/blog/copilot-vision-ai-update-brings-interactive-assistance-to-windows-11/ [3] Comparison based on publicly available information on Microsoft Copilot Vision AI, Google's Gemini Live, and Apple Intelligence. [4] https://www.microsoft.com/en-us/windows/blog/copilot-vision-ai-update-empowers-creative-workflows-on-windows-11/
- The new real-time interaction feature of Copilot Vision AI, combined with the Desktop Share function, allows the AI to comprehend and interact with artificial-intelligence-driven content on Windows 11 PCs.
- Leveraging artificial-intelligence technology, Copilot Vision AI provides users with contextual tips, insights, and guidance for various tasks on their Windows 11 PCs, enhancing productivity.
