troubleshooting

Provider troubleshooting

command not found: golemsp run

Problem

Upon executing the golemsp run command, it is observed that the process terminates immediately, showing the error:

command not found: golemsp run

Solution

The issue generally arises when the golemsp executable is not present in the system's PATH. To rectify this:

  1. First, determine which shell you are using:
echo $SHELL

  1. Based on the shell you're using, update the corresponding configuration file to add the path where golemsp is located:

    • For Bash:
    echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
    • For Zsh:
    echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.zshrc

  2. After updating the configuration file, you should update the current terminal session:

export PATH="$HOME/.local/bin:$PATH"
  1. Try running the golemsp run command again.

child provider exited too early: Ok(ExitStatus(unix_wait_status(256)))

Problem

When executing the golemsp run command, it terminates shortly after and displays the following error messages:

[2023-03-14T16:17:24Z ERROR golemsp::service] child provider exited too early: Ok(ExitStatus(unix_wait_status(256)))
[2023-03-14T16:17:24Z WARN  golemsp::service] provider exited with: Custom { kind: Other, error: "process exited too early" }

Solution

The underlying issue is that Golem's provider needs the Linux KVM module to instantiate VMs for task computation.

If you're using docker

Mount the KVM module within your container with the following configuration:

volumes:
     - /dev/kvm:/dev/kvm

If you're using a VM

Ensure that the nested virtualization flag for the VM is toggled on. For platforms like Proxmox, you may need to enable additional CPU flags to make it work.

If you're using Virtualbox

Virtualbox is not compatible with Golem. Consider switching to Vmware Workstation Player or another supported platform.

Could not access KVM kernel module: No such file or directory

Problem

When executing the golemsp run command, it terminates shortly after and displays the following error messages:

Could not access KVM kernel module: No such file or directory
    vmrt: failed to initialize kvm: No such file or directory
    [2023-05-13 18:43:48.469882 +00:00] ERROR [/home/runner/.cargo/git/checkouts/ya-runtime-sdk-9ad26604fa07f4ec/f542b92/ya-runtime-sdk/src/runner.rs:66] Runtime panic: panicked at 'Failed to start runtime: Connection refused (os error 111)', runtime/src/main.rs:362:14

Solution

The underlying issue is that Golem's provider needs the Linux KVM module to instantiate VMs for task computation.

If you're using docker

Mount the KVM module within your container with the following configuration:

volumes:
     - /dev/kvm:/dev/kvm

If you're using a VM

Ensure that the nested virtualization flag for the VM is toggled on. For platforms like Proxmox, you may need to enable additional CPU flags to make it work.

If you're using Virtualbox

Virtualbox is not compatible with Golem. Consider switching to Vmware Workstation Player or another supported platform.

vmrt: failed to initialize kvm: No such file or directory

Problem

When executing the golemsp run command, it terminates shortly after and displays the following error messages:

Could not access KVM kernel module: No such file or directory
    vmrt: failed to initialize kvm: No such file or directory
    [2023-05-13 18:43:48.469882 +00:00] ERROR [/home/runner/.cargo/git/checkouts/ya-runtime-sdk-9ad26604fa07f4ec/f542b92/ya-runtime-sdk/src/runner.rs:66] Runtime panic: panicked at 'Failed to start runtime: Connection refused (os error 111)', runtime/src/main.rs:362:14

Solution

The underlying issue is that Golem's provider needs the Linux KVM module to instantiate VMs for task computation.

If you're using docker

Mount the KVM module within your container with the following configuration:

volumes:
     - /dev/kvm:/dev/kvm

If you're using a VM

Ensure that the nested virtualization flag for the VM is toggled on. For platforms like Proxmox, you may need to enable additional CPU flags to make it work.

If you're using Virtualbox

Virtualbox is not compatible with Golem. Consider switching to Vmware Workstation Player or another supported platform.

unsupported virtualization type: XEN

Problem

The golemsp run command exits shortly after running it.

Solution

Your system is unfortunately not supported, as we do not support the XEN hypervisor.

No file descriptors available

Problem

The golemsp run command exits shortly after running it and you might see a log like

thread 'actix-rt|system:0|arbiter:231' panicked at 'called Result::unwrap() on an Err value: Os { code: 24, kind: Uncategorized, message: "No file descriptors available" }', /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-server-2.2.0/src/worker.rs:425:30
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
thread 'main' panicked at 'called Result::unwrap() on an Err value: RecvError', /home/runner/.cargo/registry/src/github.com-1ecc6299db9ec823/actix-rt-2.8.0/src/arbiter.rs:156:25

Solution

When running Golem on systems with a large number of cores and threads, you may encounter an issue where the available file descriptors on your system get exhausted. This can lead to unexpected errors or interruptions in Golem's operation.

To solve this issue, increase the file descriptors' limit before launching Golem.

  1. Open a terminal window.
  2. Increase the file descriptors limit by running the following command: ulimit -n 65535
  3. In the same terminal window, launch the provider again using golemsp run

yagna is already running

Problem

When attempting to run the commands yagna service run or golemsp run, you receive an error message that reads:

Error: yagna is already running

Solution

This indicates that the Yagna service, which could be acting as a provider or requestor daemon, is already active on your system.

To resolve the issue:

  1. If your system supports the killall command, execute:

    killall yagna

    1.1 If killall isn't supported or you'd prefer a manual method:

    ps aux | grep yagna

    From the output, note the PID (Process ID) of the yagna service.

    1.2 Terminate the identified service with:

    kill -9 <PID>

  2. Once you've terminated the running instance(s), you can start Yagna or the provider as you normally would.

I am not receiving any tasks

Problem

You've set up as a provider on the Golem Network, but you're not receiving any tasks. This can be an effect of multiple causes:

  1. Pricing Configuration: If your pricing is set too high relative to other providers, requestors might bypass your offer. You can compare your pricing with other top earners on the network by visiting the stats page.

  2. Network Demand: The number of tasks available might be low due to a disparity between supply (number of providers) and demand (number of requestors). This is a typical marketplace dynamic where, at times, supply may exceed demand.

Solution

  1. Adjust Pricing: Review the current pricing trends on the stats page and adjust your pricing competitively.

  2. Stay Updated: Monitor the Golem Network for any updates or changes that might impact the distribution of tasks on the network.