Reference material includes
If implementing multigrid in MPI, keep in mind that a well-tuned 3D multigrid code will spending most of its running time at the finest level mesh, so you'll want to be able to ascribe computational costs on a per-level basis. At some point, the mesh will become sufficiently coarse that you will need to perform all computation on a single core. There simply isn't enough work to keep processors busy doing computation and communication overheads will be too high to be able to reduce the running time via parallelism.
More generally, as you recurse to coarser levels, you'll eventually want to perform the computation on the host, reducing the number of threads as the mesh gets coarser. The reason why is that as the levels get coarser, there will be less and less computational work per core, and data motion overhead can become a problem. This strategy makes the communication patterns more interesting; when you coarsen a mesh you must send it to the CPU, which will work at the next coarser level, and similarly, when you perform coarse grid correction you'll distribute the data to the GPU before projecting the coarsened data onto a finer mesh.
Maintained by [Tue Jan 20 22:11:27 PST 2015]