Microsoft Warns Its Windows AI Feature Brings Data Theft and Malware Risks, and 'Occasionally May Hallucinate'
"Copilot Actions on Windows 11" is currently available in Insider builds (version 26220.7262) as part of Copilot Labs, according to a recent report, "and is off by default, requiring admin access to set it up."
But maybe it's off for a good reason...besides the fact that it can access any apps installed on your system:
In a support document, Microsoft admits that features like Copilot Actions introduce " novel security risks ." They warn about cross-prompt injection (XPIA), where malicious content in documents or UI elements can override the AI's instructions. The result? " Unintended actions like data exfiltration or malware installation ."
Yeah, you read that right. Microsoft is shipping a feature that could be tricked into installing malware on your system. Microsoft's own warning hits hard: "We recommend that you only enable this feature if you understand the security implications." When you try to enable these experimental features, Windows shows you a warning dialog that you have to acknowledge. ["This feature is still being tested and may impact the performance or security of your device."]
Even with these warnings, the level of access Copilot Actions demands is concerning. When you enable the feature, it gets read and write access to your Documents, Downloads, Desktop, Pictures, Videos, and Music folders... Microsoft says they are implementing safeguards. All actions are logged, users must approve data access requests, the feature operates in isolated workspaces, and the system uses audit logs to track activity.
But you are still giving an AI system that can "hallucinate and produce unexpected outputs" (Microsoft's words, not mine) full access to your personal files.
To address this, Ars Technica notes, Microsoft added this helpful warning to its support document this week. "As these capabilities are introduced, AI models still face functional limitations in terms of how they behave and occasionally may hallucinate and produce unexpected outputs."
But Microsoft didn't describe "what actions they should take to prevent their devices from being compromised. I asked Microsoft to provide these details, and the company declined..."
<a href="http://twitter.com/home?status=Microsoft+Warns+Its+Windows+AI+Feature+Brings+Data+Theft+and+Malware+Risks%2C+and+'Occasionally+May+Hallucinate'+%3A+https%3A%2F%2Ftech.slashdot.org%2Fstory%2F25%2F11%2F23%2F060221%2F%3Futm_source%3Dtwitter%26utm_medium%3Dtwitter" rel="nofollow"><img src="https://a.fsdn.com/sd/twitter_icon_large.png"></a>
<a href="http://www.facebook.com/sharer.php?u=https%3A%2F%2Ftech.slashdot.org%2Fstory%2F25%2F11%2F23%2F060221%2Fmicrosoft-warns-its-windows-ai-feature-brings-data-theft-and-malware-risks-and-occasionally-may-hallucinate%3Futm_source%3Dslashdot%26utm_medium%3Dfacebook" rel="nofollow"><img src="https://a.fsdn.com/sd/facebook_icon_large.png"></a>
at Slashdot.
Microsoft Warns Its Windows AI Feature Brings Data Theft and Malware Risks, and
"Copilot Actions on Windows 11" is currently available in Insider builds (version 26220.7262) as part of Copilot Labs, according to a recen...
Microsoft Warns Its Windows AI Feature Brings Data Theft and Malware Risks, and
"Copilot Actions on Windows 11" is currently available in Insider builds (version 26220.7262) as part of Copilot Labs, according to a recen...
