error: Opaque pointers are only supported in -opaque-pointers mode

[hearnsj]

error: Opaque pointers are only supported in -opaque-pointers mode (Producer: 'LLVM17.0.0git' Reader: 'LLVM 15.0.7jl')

This error seen when running on a system with ROCm 6.1.0 with MI210 GPUs
I guess it is to be expected as mixing LLVM versions

Setting export JULIA_LLVM_ARGS="-opaque-pointers=1" then gives:
ERROR: Taking the type of an opaque pointer is illegal

[hearnsj]

julia> versioninfo()
Julia Version 1.10.2
Commit bd47eca2c8a (2024-03-01 10:14 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Linux (x86_64-linux-gnu)
CPU: 128 × AMD EPYC 7763 64-Core Processor
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, znver3)
Threads: 1 default, 0 interactive, 1 GC (on 128 virtual cores)
julia> using AMDGPU

julia> AMDGPU.devices()
┌────┬────────────────────┬────────────────────────┬───────────┬────────────┐
│ Id │ Name │ GCN arch │ Wavefront │ Memory │
├────┼────────────────────┼────────────────────────┼───────────┼────────────┤
│ 1 │ AMD Instinct MI210 │ gfx90a:sramecc+:xnack- │ 64 │ 63.984 GiB │
│ 2 │ AMD Instinct MI210 │ gfx90a:sramecc+:xnack- │ 64 │ 63.984 GiB │
│ 3 │ AMD Instinct MI210 │ gfx90a:sramecc+:xnack- │ 64 │ 63.984 GiB │
│ 4 │ AMD Instinct MI210 │ gfx90a:sramecc+:xnack- │ 64 │ 63.984 GiB │
│ 5 │ AMD Instinct MI210 │ gfx90a:sramecc+:xnack- │ 64 │ 63.984 GiB │
│ 6 │ AMD Instinct MI210 │ gfx90a:sramecc+:xnack- │ 64 │ 63.984 GiB │
│ 7 │ AMD Instinct MI210 │ gfx90a:sramecc+:xnack- │ 64 │ 63.984 GiB │
│ 8 │ AMD Instinct MI210 │ gfx90a:sramecc+:xnack- │ 64 │ 63.984 GiB │
└────┴────────────────────┴────────────────────────┴───────────┴────────────┘

[hearnsj]

Duh.. I may be reporting this issue
JuliaGPU/GPUCompiler.jl#511

[pxl-th]

Can you show the output of AMDGPU.versioninfo()?
You should be using our downgraded device libraries that use regular pointers for Julia kernel compilation

[hearnsj]

julia> AMDGPU.versioninfo()
ROCm provided by: system
[+] HSA Runtime v1.13.0
@ /opt/rocm-6.1.0/lib/libhsa-runtime64.so
[+] ld.lld
@ /opt/rocm-6.1.0/llvm/bin/ld.lld
[+] ROCm-Device-Libs
@ /opt/rocm/amdgcn/bitcode
[+] HIP Runtime v6.601.40091
@ /opt/rocm-6.1.0/lib/libamdhip64.so
[+] rocBLAS v4.1.0
@ /opt/rocm-6.1.0/lib/librocblas.so.4
[+] rocSOLVER v3.25.0
@ /opt/rocm-6.1.0/lib/librocsolver.so
[+] rocALUTION
@ /opt/rocm-6.1.0/lib/librocalution.so
[+] rocSPARSE
@ /opt/rocm-6.1.0/lib/librocsparse.so.1
[+] rocRAND v2.10.5
@ /opt/rocm-6.1.0/lib/librocrand.so
[+] rocFFT v1.0.21
@ /opt/rocm-6.1.0/lib/librocfft.so
[+] MIOpen v3.1.0
@ /opt/rocm-6.1.0/lib/libMIOpen.so

HIP Devices [8]
1. HIPDevice(name="AMD Instinct MI210", id=1)
2. HIPDevice(name="AMD Instinct MI210", id=2)
3. HIPDevice(name="AMD Instinct MI210", id=3)
4. HIPDevice(name="AMD Instinct MI210", id=4)
5. HIPDevice(name="AMD Instinct MI210", id=5)
6. HIPDevice(name="AMD Instinct MI210", id=6)
7. HIPDevice(name="AMD Instinct MI210", id=7)
8. HIPDevice(name="AMD Instinct MI210", id=8)

I am trying to run the code from this discourse thread
https://discourse.julialang.org/t/benchmark-parallelstencil-implicitglobalgrid-performance-on-cluster/113023/2
https://github.com/svretina/ScalarWave3D.jl

[hearnsj]

module load rocm amdclang

$ env | grep rocm
CXX=/opt/rocm-6.1.0/llvm/bin/amdclang++
CPLUS_INCLUDE_PATH=/opt/rocm-6.1.0/include
F77=/opt/rocm-6.1.0/llvm/bin/amdflang
LOADEDMODULES=rocm/6.1.0:amdclang/17.0-6.1.0
F90=/opt/rocm-6.1.0/llvm/bin/amdflang
INCLUDE=/opt/rocm-6.1.0/include
LD_RUN_PATH=/opt/rocm-6.1.0/llvm/lib
LMOD_FAMILY_GPUSDK=rocm
ROCM_PATH=/opt/rocm-6.1.0/
LD_LIBRARY_PATH=/opt/rocm-6.1.0/lib64:/opt/rocm-6.1.0/lib:/opt/rocm-6.1.0/llvm/lib
FC=/opt/rocm-6.1.0/llvm/bin/amdflang
PATH=/opt/rocm-6.1.0/bin:/opt/rocm-6.1.0/llvm/bin:/datasets/teams/hackathon-testing/hearns/.juliaup/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
CC=/opt/rocm-6.1.0/llvm/bin/amdclang
LMFILES=/etc/lmod/modules/ROCm/rocm/6.1.0.lua:/etc/lmod/modules/ROCm/amdclang/17.0-6.1.0.lua
C_INCLUDE_PATH=/opt/rocm-6.1.0/include
CPATH=/opt/rocm-6.1.0/include:/opt/rocm-6.1.0/llvm/include

[pxl-th]

Ah, for some reason it is using device libraries from system-wide installation.

What version of AMDGPU.jl are you using? Try 0.8.11, it should automatically use patched device libraries

[hearnsj]

Thankyou. Updated to AMDGPU@0.8.11

Now I get an error which is specific to the code. Please close this issue!
julia> ScalarWave3D.Run.cartesian3D(L, ncells, tf, cfl, save_every=1, res="test")
ERROR: Scalar indexing is disallowed.
Invocation of setindex! resulted in scalar indexing of a GPU array.
This is typically caused by calling an iterating implementation of a method.
Such implementations do not execute on the GPU, but very slowly on the CPU,
and therefore should be avoided.

If you want to allow scalar iteration, use allowscalar or @allowscalar
to enable scalar iteration globally or for the operations in question.

[pxl-th]

It's hard to say what exactly causing this without full stack trace.
Most likely that package is using some operation that is incompatible with GPU and needs to be adapted.
I notice it is using CUDA directly, so AMDGPU.jl package may be missing some functionality

But to be sure, full stacktrace would be helpful ;)

[hearnsj]

julia> ScalarWave3D.Run.cartesian3D(L, ncells, tf, cfl, save_every=1, res="test")
ERROR: Scalar indexing is disallowed.
Invocation of setindex! resulted in scalar indexing of a GPU array.
This is typically caused by calling an iterating implementation of a method.
Such implementations do not execute on the GPU, but very slowly on the CPU,
and therefore should be avoided.

If you want to allow scalar iteration, use allowscalar or @allowscalar
to enable scalar iteration globally or for the operations in question.
Stacktrace:
[1] error(s::String)
@ Base ./error.jl:35
[2] errorscalar(op::String)
@ GPUArraysCore ~/.julia/packages/GPUArraysCore/GMsgk/src/GPUArraysCore.jl:155
[3] _assertscalar(op::String, behavior::GPUArraysCore.ScalarIndexing)
@ GPUArraysCore ~/.julia/packages/GPUArraysCore/GMsgk/src/GPUArraysCore.jl:128
[4] assertscalar(op::String)
@ GPUArraysCore ~/.julia/packages/GPUArraysCore/GMsgk/src/GPUArraysCore.jl:116
[5] setindex!
@ ~/.julia/packages/GPUArrays/OKkAu/src/host/indexing.jl:56 [inlined]
[6] scalar_setindex!(::AMDGPU.ROCArray{Float64, 3, AMDGPU.Runtime.Mem.HIPBuffer}, ::Float64, ::Int64, ::Vararg{Int64})
@ GPUArrays ~/.julia/packages/GPUArrays/OKkAu/src/host/indexing.jl:40
[7] _setindex!
@ ~/.julia/packages/GPUArrays/OKkAu/src/host/indexing.jl:24 [inlined]
[8] setindex!
@ ~/.julia/packages/GPUArrays/OKkAu/src/host/indexing.jl:22 [inlined]
[9] macro expansion
@ ~/ScalarWave3D.jl/src/Run.jl:72 [inlined]
[10] macro expansion
@ ~/.julia/packages/Polyester/Ho6h3/src/closure.jl:525 [inlined]
[11] sample_statevector!(statevector::AMDGPU.ROCArray{…}, grid::Vector{…}, paramsID::StaticArraysCore.SVector{…}, t::Float64)
@ ScalarWave3D.Run ~/ScalarWave3D.jl/src/Run.jl:70
[12] cartesian3D(L::Int64, ncells::Int64, tf::Float64, cfl::Float64; save_every::Int64, res::String)
@ ScalarWave3D.Run ~/ScalarWave3D.jl/src/Run.jl:36
[13] top-level scope
@ REPL[16]:1
[14] top-level scope
@ ~/.julia/packages/AMDGPU/gtxsf/src/tls.jl:200

[pxl-th]

Well, it looks like the code is written only for CPU, e.g.:
https://github.com/svretina/ScalarWave3D.jl/blob/b1670797e88c42cf1b48511ceeb7796879ca26a2/src/Run.jl#L70
which does element wise indexing on the statevector, but this is generally very slow to do on GPU.
To adapt the code for GPU, you'd need to write GPU kernels or use higher-level array operations.

[hearnsj]

Thankyou! Please close the issue.