cuda::ref::operator += and -= produce errors

I'm getting the following error:

/mnt/home/mmorales/Devel/qmcpack2ccq/extern/boost_multi/multi/memory/adaptors/cuda/ptr.hpp(575): error: initial value of reference to non-const must be an lvalue
          detected during:
            instantiation of "boost::multi::memory::cuda::ref<T> &boost::multi::memory::cuda::ref<T>::operator+=(O &&) && [with T=std::complex<double>, O=qmcplusplus::afqmc::ComplexType &, <unnamed>=std::complex<double> &]"

The error comes from the device code implementation of operator+= and operator-=. If I comment out the device path (including the #if ...), the code compiles correctly. The original code is in host code, but somehow the device path is being compiled too.