How can I verify if ROCm is properly utilized by my application?
I'm trying to determine if ROCm is working correctly with my application. I see some GPU activity, but it's not significant. When I start the server, I expect to see three INFO lines, but I only see two, and there's no GPU info included. Here's what my server start looks like:
2024-01-25T17:24:09.377143Z INFO tabby::serve: crates/tabby/src/serve.rs:114: Starting server, this might take a few minutes...
2024-01-25T17:24:10.056685Z INFO tabby::routes: crates/tabby/src/routes/mod.rs:35: Listening at 0.0.0.0:8080I've checked with rocminfo and radeontop, and I'm seeing about 5% GPU utilization. I'm concerned that I might be using amdgpu packaged in the kernel and not amd-dkms. I'm also on a newer version of ROCm compared to what my application supports, which could be causing issues. How can I confirm that ROCm is being used effectively, and what typical GPU utilization should I expect for inference tasks?
Stephen
Asked on Jan 25, 2024
If you're not seeing the printouts starting with ggml, it's likely that ROCm is not being used by your application. When you pass the --device rocm flag, you should see specific printouts indicating success, or an error message if ROCm could not be found. Since you're not getting either of those, there might be an issue. It's also worth noting that Debian tends to ship with outdated packages, which could contribute to the problem. Make sure that your version of ROCm is compatible with the application you're using, and check if ROCm support is included by default in your current Debian version. You may need to investigate further to ensure that ROCm is set up correctly and that your application is configured to use it.