AFAIK the node_modules should not be needed as they are precompiled into the vendor.js file.
I removed them from my project by taking the folder out of the publishOptions include list in project.json. However the site only works then if the asp-prerender tag is also removed from the home page markup.

Also tried adding the aspnet-prerendering module to the webpack.config.vendor.js list but then the compiled javascript didn't run.