Run unity once in build process?
When a build starts Unity processes all the in built packages:
...
Start importing Assets using Guid(00000000000000001000000000000000) Importer(-1,00000000000000000000000000000000)
Done importing asset: 'Assets' (target hash: '1b379d2fcd0ecfe174cd8a6e24ec1d15') in 0.030469 seconds
Start importing Packages/com.unity.test-framework using Guid(6023f2b823fc19b4e4a90875281ff117) Importer(-1,00000000000000000000000000000000)
Done importing asset: 'Packages/com.unity.test-framework' (target hash: '9523e83a7376204932cb42ad1f394a6c') in 0.002278 seconds
Start importing Packages/com.unity.modules.imgui using Guid(c060426bfd6e82575228df6656368eaa) Importer(-1,00000000000000000000000000000000)
Done importing asset: 'Packages/com.unity.modules.imgui' (target hash: '2f6c4e7ea8aac208b68c05aed0d9a48f') in 0.002142 seconds
Start importing Packages/com.unity.modules.animation using Guid(1158e311a3101950348dcecb1bebc42d) Importer(-1,00000000000000000000000000000000)
Done importing asset: 'Packages/com.unity.modules.animation' (target hash: '92a791fe33e8254f70bff5b81a3558ac') in 0.002303 seconds
Start importing Packages/com.unity.modules.xr using Guid(515638b803bef8599dbd6d5c8bdaa53e) Importer(-1,00000000000000000000000000000000)
...
In theory if Unity was run once during the docker build, all those packages will have already been processed and build times should be shorter.