boomerAMG on GPU for SolidMechanicsLagrangianSSLE

[castelletto1]

The purpose of this PR is to enable boomerAMG-preconditioning on GPU for an elasticity problem. The linear algebra interface is set to hypre by default. As a model problem the following simple cantilivered cube problem has been added:

SSLE-QS-cantileveredCube.xml

The mesh consists of a unit cube that is discretized with a regular 10x10x10 Cartesian mesh.

GEOS-DEV/LvArray#213

The Hypre build options are in the corresponding thirdPartyLibs PR:
GEOS-DEV/thirdPartyLibs#132

[oseikuffuor1]

All, here are some recommendations for building and running with hypre on the GPU.
Configure Options:

1. --with-cuda
2. --enable-cusparse (this is optional and allows one to use cusparse for certain operations. You can ignore this for now)
3. --enable-debug (Helpful for debugging issues down the road. Can be ommitted if the integration is stable)
4. Need to set the environment variable HYPRE_CUDA_SM to match the SM (streaming multiprocessor) for the hardware. Default is 60, but change to 70 if running on lassen (V100 systems) for example.
5. See https://hypre.readthedocs.io/en/latest/ch-misc.html for more details about building hypre for GPU support.

Enabling GPU support

1. In user code, first call HYPRE_Init(); to initialize some GPU libraries prior to calling any other hypre functions
   2a. Set the HYPRE_ExecutionPolicy variable to device: HYPRE_ExecutionPolicy default_exec_policy = HYPRE_EXEC_DEVICE;
   2b. Set the default policy handle: hypre_HandleDefaultExecPolicy(hypre_handle()) = default_exec_policy; (this tells hypre to do AMG setup on device. Clearly you can combine step 2 and just set the handle to HYPRE_EXEC_DEVICE)
2. Call HYPRE_Finalize(); at the end (before MPI_Finalize is called)
3. See ij.c or ij_assembly.c in test directory of hypre for additional insight.

Runtime solver options:

• Relaxation options: Use options 18 or 7.
• Coarsening options: Only PMIS is supported
• Interpolation options: Use options 3, 6, 14 or 15.
  Once everything is up and running, there may be other options to tune for performance. I have omitted them for now.

There's one minor edit from our discussion yesterday. I believe there were two scenarios:

1. Assemble linear system on host and pass to hypre
2. Assemble linear system on device and pass to hypre

Since geosx does not rely on unified memory, I am leaning towards option 2, as long as the data given to hypre is a consistent parcsr matrix data. I had mentioned that passing hypre host data should work, but this is only true with unified memory. Without unified memory, option 1 could be realized by moving the assembled matrix to device and calling hypre then. If you have questions about the linear system matrix setup on device, let me know and we can chat again. Of course let me know also if you have additional questions about these notes.

[andrea-franceschini]

Does hypre provide a block Jacobi preconditioner?

[oseikuffuor1]

Does hypre provide a block Jacobi preconditioner?

@AF1990 are these blocks per processor or per unknowns? We do not have a BJ preconditioned for the unknown version, but for the per processor blocks, we have the BJ ILU preconditioner.

[andrea-franceschini]

@AF1990 are these blocks per processor or per unknowns? We do not have a BJ preconditioned for the unknown version, but for the per processor blocks, we have the BJ ILU preconditioner.

I am interested in the unknown version of the BJ preconditioner. I know that the processor version is already available.

[rrsettgast]

@oseikuffuor1 I think this is setup per your instructions, but execution on Lassen using nvprof doesn't show a cuda kernel so it must still be executing on host. Any suggestions?

[rrsettgast]

@corbett5 @klevzoff I won't be addressing this since hypre will be fixing their global dog index types soon.

[klevzoff]

Did you mean for this to be a parallel kernel launch? Or what's the purpose of using 2D arrays for srcIndices and srcValues?

[rrsettgast]

There are a bunch of host functions in this for loop. They are from hypre, so we don't have control over them. It is pretty strange, but the underlying hypre functions are all host, but if running on device, they take device pointers.

[klevzoff]

Reverted back to Trilinos, intentional?

(We should look into changing the way this file is handled... it's getting out of hand, especially with more macros added)

[rrsettgast]

This seems to be a general issue. I must have built on lassen, and accidentally committed this file.