Resolved: Excessive Persistent VMMEM CPU utilization

By shauncassells in Microsoft Defender, Windows Insider on
8 comments

Resolution: Uninstall Windows Defender Application Guard UWP and Server.

I recently ran into an issue where every time I started my Work Windows 10 laptop the fan would kick on high.  Glancing quickly at Task Manager I could always see VMMEM and vmwp.exe (Virtual Machine Worker Process) chugging away and eating the CPU and battery.

Process Explorer - Sysinternals: www.sysinternals.com [I E\Shaun.CasseII File Options View Process Find Users Help Private Bytes LANorking Set Process System Idle Process vmmem E$Teams.exe vmwp.exe procexp64.exe System L CPU 48.34 20.06 10.97 5.28 3.53 2.31 60 K 1 ,991 ,040 K 412,312 K 17,188 K 36,280 K 232 K 8K 432,540 K 26,820 K 55,360 K 7,716K

What is VMMEM?

The vmmem process is a virtual process that the system synthesizes to represent the memory and CPU resources consumed by your virtual machines. In other words, if you see vmmem consuming a lot of memory and CPU resources, then that means your virtual machines are consuming a lot of memory and CPU resources.

From <https://devblogs.microsoft.com/oldnewthing/20180717-00/?p=99265>

Virtual Machines Running?

PROBLEMS OUTPUT øø:øe 9.e øø:øe 9.e oø:øe 9.e øø:øe øø:øe øø:øe øø:øe øø:øe DEBUG CONSOLE PS C: Wsers\shaun. cassells> get-vm TERMINAL State CPUUsage(X) MemoryAssigned(m) Uptime Status Operating Operating Operating Operating Operating Operating Operating Operating Operating Operating Version I EDemo I EDemo I EDemo I E Demo IEDemo-191ø-1Eø1 1 EDemo-191ø-1EØ2 1 EDemo-191ø-cmø1 1 EDemo-191e-Dcø1 1 EDemo-191ø-pceoø1 IEDemo-191ø-pceøø2 -191ø-Re1 -191e-pceøø3 -191ø-pceøø4 -191ø-pceøøs Off Off Off Off Off Off Off Off Off Off øø:øe:øø øø:øe:øø normally normally normally normally normally normally normally normally normally normally

Very odd as I have ZERO Hyper-V machines running

Windows Sandbox?

Hmm perhaps it is Windows Sandbox?

  1. appwiz.cpl
  2. Turn Windows Features On and Off (upper left)
  3. Windows Sandbox

Nope that is disabled as well

Windows Features Turn Windows features on or off To turn a feature on, select its check box. To turn a feature off, clear its check box. A filled box means that only part of the feature is turned on. CJ j Telnet Client Client Virtual Machine Platform Windows Defender Application Guard Windows Hypervisor Platform Windows Identity Foundation 3.5 Windows PowerSheII 2.0 Windows Process Activation Service Windows Pro'ected File System "Indows Sandbcy Windows Subs Enables the de Windows TIFF IF' ter Work Folders Client uired run Windows Cancel

Well what is causing the issue?  Hmm maybe it is a performance issue?

Maybe I am having a Performance Issue?

Side Note: reporting on counters is easy but not very good at root cause.  See day job.

  1. Admin CMD prompt
  2. Perfmon / report

Hmm looks like I have a bad service.  MSDTC is failed.  Well that could cause some memory issues and VMMEM is perhaps related.  Lets fix that.  What you are supposed to do is dig through the EventView (great powershell queries exist for that)… or you could try the oldest trick in the book!

https://support.microsoft.com/en-us/help/916926/you-may-receive-error-code-1073737712-when-you-try-to-start-the-distri

Fix:

msdtc -resetlog

Cool well I fixed something that was broken.  That’s nice. Still have the high process usage. 

Nuclear Option – Hard core root cause

Shutdown the Hyper-V service

Net stop vmcompute

Hyper-V Host compute Service properties (Local computer) General On Display name ; Host Compute support for running V'/indow-s Containers path to C N exe Startup type status ; Running You the start parameters that apply when you start the Start parameters:

This will tell you all dependent processes. 

Oh look VMMEM utilization is at dead stop!  What is the root cause?

Windows Defender Application Guard

Overview of WDAG: https://docs.microsoft.com/en-us/windows/security/threat-protection/windows-defender-application-guard/wd-app-guard-overview

Hardware isolation diagram
Windows Features Turn Windows features on or off To turn a feature on, select its check box. To turn a feature off, clear its check box. A filled box means that only part of the feature is turned on. CJ j Telnet Client Client Virtual Machine Platform Windows Defender Aoohcatlcn Enables platform support for virtual machines In Ows entity oun anon Windows PowerSheII 2.0 Windows Process Activation Service Windows Projected File System Windows Sandbox Windows Subsystem for Linux Windows TIFF IFiIter Work Folders Client Cancel

Turning this off required a reboot and did resolve the VMMEM process running at startup.

Next up.  Why?

Looks like it was a UWP application

Windows Defender Application Guard Companion – https://www.microsoft.com/en-us/p/windows-defender-application-guard-companion/9n8gnlc8z9c8?activetab=pivot:overviewtab

Microsoft Store Home Gaming Entertainment Productivity Deals This product is installed. Windows Defender Application Guard Companion Microsoft Corporation Security > PC protection Share 17 Windows Defender Application Guard helps protect your device from advanced attacks by opening untrusted websites in an isolated Microsoft Edge browsing window. Using a unique hardware-based isolation approach, Application Guard opens untrusted websites p Search $3 Wish list More EVERYONE ESRB Overview System Requirements Revi ews Related Available on

Another interesting thread

https://answers.microsoft.com/en-us/edge/forum/edge_other-edge_win10/windows-defender-application-guard-fails-to-load/fec9111c-e795-4a7b-b22b-bbe7f2a84d22?auth=1

Summary

Windows Defender Application Guard (WDAG) was absolutely the root cause. However, just disabling is only the first part of research. Anyone know a way to enable the feature without the CPU and battery drain?

Published by shauncassells

Shaun Cassells, Senior Solutions Engineer, 1E Shaun is a Microsoft MVP Windows Insider MVP and executive at Central Texas Systems Management User Group (CTSMUG). Shaun is a frequent speaker at conferences like MMS, Gartner, TechEd Europe, TechEd North America, IT Dev Connections, System Center Rallies, and Systems Management User Groups (SMUGs). Shaun is an award winning blogger syndicated from www.shauncassells.com. Shaun specializes in analysis, optimization, and design of Tactical Solutions to Strategic Business Goals. Prior to joining 1E in 2010, he worked for a Global 100 company as the Configuration Manager Service Owner and Architect. Recently as a Principle Consultant at 1E, he accomplished design, review, and improvements to a variety of Configuration Manager environments from the very small to sites with a half million seats. He provided leadership that created a patented automated application rationalization and usage based OSD mapping solution in use by multiple fortune 500 companies. These days you can find Shaun traveling to help discover, review, and improve business productivity across the world.

8 comments on “Resolved: Excessive Persistent VMMEM CPU utilization”

  3. I was excited when I found this post. I was experiencing exactly the same: high CPU usage (20% or more) from VMMEM. Unfortunately, Windows Defender Application Guard was already turned off on my machine. I’d recently installed Docker plus turned on WSL (linux) on my Win 10 machine. I tried reversing all that. WSL removed, no Docker services running. VMMEM continued to churn the CPU. This was with no virtual machines running. I finally unchecked Hyper-V in “Turn Windows features on or off”, rebooted and that got rid of VMMEM. Posting my fix here in case someone else has similar experience and finds your post as I did.

    Liked by 1 person

    Reply

      1. Hey Shaun,
        I have reported this on feedback hub, as it is becomming very inconvenient for me. I’ve attached perf data so hopefully this will help towards the general problems in reaching a resolution.

        Thanks!

        ———————————————————————————————————————————-

        The brief description in my report is here, hope it helps:

        Hi,
        I experience this problem several times per wee. My laptop runs very slowly, the fan ramps up to full and it starts consuming large amounts of CPU, RAM and especially disk, it often becomes totally unresponsive for up to 30 seconds. Killing Docker Desktop and Hyper-V has no effect. A reboot is the only reliable solution as I’ve not been able to find the root cause after 30 mins and just don’t have the time to properly investigate and fix it. The reboots are quite disruptive as I have all sorts of documents open, Teams, Visual Studio, VS code, WSL, and lots of other tools too – it can take half an hour to save them all, restart and get them all back (and even longer for my concentration!)

        I Have just noticed that Task Manager is telling me vmmem is using 100% disk and no RAM today!

        Thanks,
        ———————————————————————————————————————————-

        Like

      2. In windows feedback hub (winkey+F) when you submit this bug or the one you have in the past. Please click the share button to your bug and send it to me. I can have Microsoft Engineering take a look at it.

        Like

  5. I have the same issue and I found the root cause.

    The root cause of my case is DockerDesktopVM. When I opened hyper-V manager I saw DockerDesktopVM is running. Then, I connected to this VM and saw “write failed” message repeating indefinitely. Looks like this VM is trying to write something and continuously failed.

    How I solve this?
    I open DockerDesktop application, and then DockerDesktopVM stop trying to write something. After that, I turned off DockerDesktop application, and then DockerDesktopVM shut down by itself. Finally, vmmem is gone.

    Like

    Reply

Leave a comment