122 votesDevin Rider commented
The issue you’re describing has to do with how SQL Server manages the internal backing tables for memory optimized tables with large object and off row data. In down level editions of SQL Server, the engine does not correctly drop these internal tables when the user alters these columns. This issue has been addressed in the latest version of SQL Server.
The mitigation is to create a new database, migrate the data from the old database to the new database, and drop the old database. When you drop the old database, the leaked internal tables are freed, and the memory reclaimed. We also recommend that you upgrade to the latest version of SQL Server as otherwise, the leak can re-occur if you alter the columns again.
This KB is not very informative and says this only applies for Linux, but as you’ve demonstrated the issue also applies to SQL Server running on Windows: