Possibility to cache folders in build directory

Maybe I am missing something but is it possible to somehow cache specific folders in my build directory?

I have the problem that I want to keep node_modules and bower_components folder but it always gets removed:

Running on runner-56b3fc8c-project-4-concurrent-0 via next...
Fetching changes...
Removing bower_components/
Removing node_modules/