Headless Arch Linux [Mining to F@H]

[Asgaroth]

Hi All,

My conscience finally got the better of me and I've decided to switch from mining to F@H

My Hardware:

• Motherboard: Asus B250 Mining Expert - specs here
• CPU: Intel Core i7 6700K - specs here
• Memory: Corsair Vengeance LPX 16GB Kit - specs here
• USB Risers x1 -> x16 - specs here
• GPU: nVidia: Zotac GeForce 1070 Ti AMP Extreme (6 of these) - product page here
• GPU: AMD: PowerColor Red Devil RX580 8GB Golden Sample (7 of these) - product page here

I am currently testing this out on Windows 10 but would prefer to switch to Linux. I am used to Manjaro Linux so am thinking of switching to Arch Linux for this box, it will be dedicated to F@H.

I understand that:

• I will not be able to drive all the GPU's with the 4 core 8 thread CPU that I have here, I'm currently driving 6 1070 ti's with the CPU at 85% utilization on Windows 10, GPU's are each at 80% utilization with temps of 45 - 50 degrees celcius
• I wont get max PPD due to the reduction in PCIe lanes to the GPU's. I've read that the 1070 Ti should be getting in the range of 800K PPD, when running on Linux, assuming a 20% reduction, I'm guessing these should be pushing around 680K PPD each

What I would like to know from those that have gone before me is:

1. Anyone have any pointers on an initial headless setup for a F@H system
2. Any BIOS tweaks recommended? (I'll try to see if I can get these risers operating at Gen-3 speeds, currently testing at Gen - 2)
3. With a headless setup, can you still use nvidia-settings to set power-limit, core/memory over-clocks?
4. Do the Arch users use the AUR package for F@H or do you decompress the deb directly from Standford?

Any tips/tricks/suggestions/criticisms/pointers/gotcha's are greatly appreciated.

EDIT: I'll be documenting my setup and configuration below in the hopes it helps someone else looking to go down this route.

[Asgaroth]

Arch Linux Installation

NOTE: Work In Progress

Details:

• Disto Used: Arch Linux 2018.12.01
• Boot Method: UEFI

Assumptions:

1. You've created a USB boot flash drive of the ISO image and booted your system to the Arch prompt.
2. Single hard disk drive, in my case I have a 500GB disk referenced as sda.
3. IP address is given out by DHCP and is a static lease (does not change)

References:

• Arch Wiki Installation Guide (here)

Steps:

1. Setup your keyboard mapping.

   Code: Select all

   loadkeys uk

2. Enable NTP

   Code: Select all

   timedatectl set-ntp true

3. Partition the drive.
   NOTE: Use the GPT partition scheme for UEFI booting, my drive was already configured with the GPT partition scheme so I didn't need to change it.
   We will be creating the following partition layout.
   1. EFI System Partition
   2. Linux Swap Partition
   3. Linux Filesystem Partition

   Code: Select all

   gdisk /dev/sda
     n
     1
     enter
     +550M
     ef00
     
     n
     2
     enter
     +64G
     8200
     
     n
     3
     enter
     enter
     8300
     
     w
     y
   

4. Create the file systems:

   Code: Select all

   mkfs.fat -F32 /dev/sda1
   mkswap /dev/sda2
   swapon /dev/sda2
   mkfs.ext4 /dev/sda3
   

5. Mount the "root" file system:

   Code: Select all

   mount /dev/sda3 /mnt

6. Setup your local mirror list:
   • Go to the Pacman Mirror List Generator, make your selections, and generate the mirror list.
   • Copy the generated mirror list and paste it into the mirror list on the Arch boot stick

     Code: Select all

     vi /etc/pacman.d/mirrorlist

7. Enable the multilib repository:
   • Edit pacman.conf

     Code: Select all

     vi /etc/pacman.conf 

   • Uncomment the following two lines

     Code: Select all

     [multilib]
     Include = /etc/pacman.d/mirrorlist
     

8. Install the base system

   Code: Select all

   pacstrap /mnt base base-devel linux linux-firmware

9. Generate the file system table

   Code: Select all

   genfstab -U -p /mnt >> /mnt/etc/fstab

10. Change root to the mounted filesystem we just boot-strapped

    Code: Select all

    arch-chroot /mnt

11. Setup your locale (Note: I'm in Ireland so modify for where you are)

    Code: Select all

    sed -i -e 's|^#\(en_IE.UTF-8 UTF-8.*\)|\1|' /etc/locale.gen
    locale-gen
    echo "LANG=en_IE.UTF-8" > /etc/locale.conf
    export LANG=en_IE.UTF-8
    

12. Setup your timezone (I'm in Ireland so modify for where you are)

    Code: Select all

    ln -sf /usr/share/zoneinfo/Europe/Dublin /etc/localtime
    hwclock --systohc --utc
    

13. Setup the keymap for your installed system

    Code: Select all

    echo "KEYMAP=uk" > /etc/vconsole.conf
    

14. Setup your system's hostname

    Code: Select all

    echo "thanatos" > /etc/hostname
    

15. Setup your hosts file.
    • NOTE:Change the address for your host below to match your own static lease address

    Code: Select all

    cat << __EOFL __ > /etc/hosts
    127.0.0.1	localhost
    ::1		   localhost
    172.16.17.4 thanatos.localdomain	thanatos
    __EOFL__
    

16. Get your network interface name

    Code: Select all

    # ip link show
    1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
        link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
        link/ether b0:6e:bf:5f:a0:7c brd ff:ff:ff:ff:ff:ff
    

17. Setup a systemd unit file to enable DHCP on this interface.
    • NOTE: This wont activate on the initial boot, we will enable the systemd-network unit file as a post-install task.

    Code: Select all

    cat << __EOFL__ > /etc/systemd/network/20-wired.network
    [Match]
    Name=enp0s31f6
    
    [Network]
    LinkLocalAddressing=no
    IPv6AcceptRA=no
    DHCP=ipv4
    
    [DHCP]
    UseDomains=yes
    __EOFL__
    

18. Enable the multilib repository in your locally installed system (See step 7 above)
19. Perform a system update.

    Code: Select all

    pacman -Syu

20. Set the root user's password

    Code: Select all

    passwd root

21. Create a "normal" user account for yourself (optional)

    Code: Select all

    useradd -mg users -G wheel,storage,power -s /bin/bash username
    passwd username
    

22. Install the sudo application

    Code: Select all

    pacman -S sudo

23. Enable users in the wheel group to have sudo access

    Code: Select all

    sed -i -e 's|^# \(%wheel ALL=(ALL) ALL\)|\1|' /etc/sudoers
    

24. Install the GRUB bootloader and UEFI related packages

    Code: Select all

    pacman -S grub efibootmgr dosfstools os-prober mtools
    

25. Install the GRUB bootloader with UEFI support

    Code: Select all

    mkdir /boot/EFI
    mount /dev/sda1 /boot/EFI
    grub-install --target=x86_64-efi --bootloader-id=grub_uefi --recheck
    

26. Install the Intel micro-code update package
    • We do this before generating the GRUB boot configuration file as GRUB will detect this is installed and enable it on the kernel boot options.
    • For AMD CPU's install amd-ucode

    Code: Select all

    pacman -S intel-ucode
    

27. Generate the grub boot configuration

    Code: Select all

    grub-mkconfig -o /boot/grub/grub.cfg

28. Exit out of the Arch chrooted environment

    Code: Select all

    exit

29. Unmount file systems

    Code: Select all

    umount -R /mnt

30. Reboot the system
    • NOTE: You may need to remove your USB boot stick if your BIOS is configured to boot off of USB first.

    Code: Select all

    reboot

[Asgaroth]

Post Install Tasks

NOTE: Work In Progress

After your first reboot login as root at the system console and perform the following steps

1. We need to enable the systemd network/resolver unit files so that systemd will manage our network configuration.

   Code: Select all

   systemctl enable systemd-networkd
   systemctl enable systemd-resolved
   

   NOTE:
   • Ensure /etc/resolv.conf is a symlink to /run/systemd/resolve/resolv.conf
2. Reboot the host to enable the network.

   Code: Select all

   reboot

3. Enable NTP

   Code: Select all

   timedatectl set-ntp true

4. Install OpenSSH and start/enable it.
   • After the startup of ssh, I assume, from here on out, you are logged in as a "normal" user

   Code: Select all

   pacman -S openssh
   systemctl start sshd
   systemctl enable sshd
   

5. Install nVidia related drivers and packages

   Code: Select all

   sudo pacman -S libglvnd nvidia cuda

6. Blacklist the OpenSource nouveau driver

   Code: Select all

   sudo cat << __EOFL__ > /etc/modprobe.d/blacklist.conf
   blacklist nouveau
   blacklist rivafb
   blacklist nvidiafb
   blacklist rivatv
   blacklist nv
   blacklist uvcvideo
   __EOFL__
   

7. Install the GIT application

   Code: Select all

   sudo pacman -S git

8. Install the YAY package (This enables us to install packages from the AUR)

   Code: Select all

   cd ~
   mkdir git_repos
   cd git_repos
   git clone https://aur.archlinux.org/yay.git
   cd yay
   makepkg -si
   

9. Install the OpenCL libraries for AMD GPU's

   Code: Select all

   yay -S opencl-amd

10. Reboot to disable the blacklisted drivers and activate the nvidia drivers

    Code: Select all

    sudo reboot

General System Maintenance

• Updating the system (while enforcing package list re-sync with mirrors)

  Code: Select all

  # pacman -Syyuu

• Updating packages from AUR

  Code: Select all

  yay -Su

• Updating keyrings

  Code: Select all

  pacman -Sy archlinux-keyring

[Asgaroth]

F@H Client Installation and Initial Configuration

NOTE: Work In Progress

References:

• Folding@Home Arch Wiki

1. Install foldingathome package

   Code: Select all

   yay -S foldingathome

2. Make sure FAHClient can see your configuration

   Code: Select all

   cd /opt/fah
   sudo /opt/fah/FAHClient --info
   
   11:49:01:INFO(1):Read GPUs.txt
   ***************************** Folding@home Client ******************************
            Website: https://foldingathome.org/
          Copyright: (c) 2009-2018 foldingathome.org
             Author: Joseph Coffland <[email protected]>
               Args: --info
   ************************************ Build *************************************
            Version: 7.5.1
               Date: May 11 2018
               Time: 19:59:04
         Repository: Git
           Revision: 4705bf53c635f88b8fe85af7675557e15d491ff0
             Branch: master
           Compiler: GNU 6.3.0 20170516
            Options: -std=gnu++98 -O3 -funroll-loops
           Platform: linux2 4.14.0-3-amd64
               Bits: 64
               Mode: Release
   ************************************ System ************************************
                CPU: Intel(R) Core(TM) i7-6700K CPU @ 4.00GHz
             CPU ID: GenuineIntel Family 6 Model 94 Stepping 3
               CPUs: 8
             Memory: 15.55GiB
        Free Memory: 10.93GiB
            Threads: POSIX_THREADS
         OS Version: 4.19
        Has Battery: false
         On Battery: false
         UTC Offset: 0
                PID: 21978
                CWD: /opt/fah
                 OS: Linux 4.19.8-arch1-1-ARCH x86_64
            OS Arch: AMD64
               GPUs: 13
              GPU 0: Bus:1 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1070 Ti] 8186
              GPU 1: Bus:4 Slot:0 Func:0 AMD:5 Ellesmere XT [Radeon RX
                     470/480/570/580]
              GPU 2: Bus:6 Slot:0 Func:0 AMD:5 Ellesmere XT [Radeon RX
                     470/480/570/580]
              GPU 3: Bus:8 Slot:0 Func:0 AMD:5 Ellesmere XT [Radeon RX
                     470/480/570/580]
              GPU 4: Bus:9 Slot:0 Func:0 AMD:5 Ellesmere XT [Radeon RX
                     470/480/570/580]
              GPU 5: Bus:10 Slot:0 Func:0 AMD:5 Ellesmere XT [Radeon RX
                     470/480/570/580]
              GPU 6: Bus:11 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1070 Ti] 8186
              GPU 7: Bus:12 Slot:0 Func:0 AMD:5 Ellesmere XT [Radeon RX
                     470/480/570/580]
              GPU 8: Bus:13 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1070 Ti] 8186
              GPU 9: Bus:14 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1070 Ti] 8186
             GPU 10: Bus:15 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1070 Ti] 8186
             GPU 11: Bus:16 Slot:0 Func:0 AMD:5 Ellesmere XT [Radeon RX
                     470/480/570/580]
             GPU 12: Bus:17 Slot:0 Func:0 NVIDIA:7 GP104 [GeForce GTX 1070 Ti] 8186
      CUDA Device 0: Platform:0 Device:0 Bus:1 Slot:0 Compute:6.1 Driver:10.0
      CUDA Device 1: Platform:0 Device:1 Bus:11 Slot:0 Compute:6.1 Driver:10.0
      CUDA Device 2: Platform:0 Device:2 Bus:13 Slot:0 Compute:6.1 Driver:10.0
      CUDA Device 3: Platform:0 Device:3 Bus:14 Slot:0 Compute:6.1 Driver:10.0
      CUDA Device 4: Platform:0 Device:4 Bus:15 Slot:0 Compute:6.1 Driver:10.0
      CUDA Device 5: Platform:0 Device:5 Bus:17 Slot:0 Compute:6.1 Driver:10.0
    OpenCL Device 0: Platform:0 Device:0 Bus:4 Slot:0 Compute:1.2 Driver:2671.3
    OpenCL Device 1: Platform:0 Device:1 Bus:6 Slot:0 Compute:1.2 Driver:2671.3
    OpenCL Device 2: Platform:0 Device:2 Bus:8 Slot:0 Compute:1.2 Driver:2671.3
    OpenCL Device 3: Platform:0 Device:3 Bus:9 Slot:0 Compute:1.2 Driver:2671.3
    OpenCL Device 4: Platform:0 Device:4 Bus:10 Slot:0 Compute:1.2 Driver:2671.3
    OpenCL Device 5: Platform:0 Device:5 Bus:12 Slot:0 Compute:1.2 Driver:2671.3
    OpenCL Device 6: Platform:0 Device:6 Bus:16 Slot:0 Compute:1.2 Driver:2671.3
    OpenCL Device 7: Platform:1 Device:0 Bus:1 Slot:0 Compute:1.2 Driver:415.22
    OpenCL Device 8: Platform:1 Device:1 Bus:11 Slot:0 Compute:1.2 Driver:415.22
    OpenCL Device 9: Platform:1 Device:2 Bus:13 Slot:0 Compute:1.2 Driver:415.22
   OpenCL Device 10: Platform:1 Device:3 Bus:14 Slot:0 Compute:1.2 Driver:415.22
   OpenCL Device 11: Platform:1 Device:4 Bus:15 Slot:0 Compute:1.2 Driver:415.22
   OpenCL Device 12: Platform:1 Device:5 Bus:17 Slot:0 Compute:1.2 Driver:415.22
   ********************************************************************************
   

3. Generate an empty initial configuration

   Code: Select all

   cd /opt/fah
   sudo ./FAHClient --configure
   

4. Make a backup of the config created.

   Code: Select all

   sudo cp /opt/fah/config.xml /opt/fah/config.xml.bak

5. Edit the configuration and allow remote access
   • Manually editing the configuration file is not recommended, however, it did work for me to get started.
   • Substitute your passcode/team/username in the example config.
   • The address "172.16.17.5" in the example below is the address of the machine where you will be connecting from (Using the F@H client)
   • I've disabled remote authentication here, it's not recommended, so enable it if you require it

   Code: Select all

   sudo vi /opt/fah/config.xml

   Code: Select all

   <config>
     <!-- Folding Slot Configuration -->
     <smp v='false'/>
   
     <!-- HTTP Server -->
     <allow v='127.0.0.1 172.16.17.5'/>
   
     <!-- Remote Command Server -->
     <command-allow-no-pass v='127.0.0.1 172.16.17.5'/>
   
     <!-- Slot Control -->
     <power v='full'/>
   
     <!-- User Information -->
     <passkey v='YOUR_PASSKEY'/>
     <team v='YOUR_TEAM'/>
     <user v='YOUR_USERNAME'/>
   
     <!-- Web Server -->
     <web-allow v='127.0.0.1 172.16.17.5'/>
   </config>
   

6. Start the client

   Code: Select all

   sudo systemctl start foldingathome

7. Enable the client to start at boot

   Code: Select all

   sudo systemctl enable foldingathome

8. Now you can head over to your "management" host, start the F@H client and connect to your headless F@H rig.

[Asgaroth]

Well, one thing I've noticed since switching from Windows 10 to Arch Linux is that my estimated PPD went from 1.2M to 4.3M! That is a massive jump with no hardware changes at all.



I know this is just an estimation on the client and is bound to fluctuate from day to day, but its been running for 15 hours now and it is estimating 4.3M PPD, that roughly equates to around 710K PPD per card, not bad going, I'll keep an eye on it and see how it goes.

Now to do more reading around to see if there are any client tweaks I can do.

I'll update these posts if I need to tweak some other settings (eg: overclocking, I suspect I'll need a virtual X server for that, WIP)

[foldy]

Windows bottlenecks on x1 risers for FAH so this is why Linux gives you the PPD booster.

Be sure to disable the FAH CPU slot because all CPU is needed to feed the GPUs.

Thank you for sharing your Arch Linux experience

[ProDigit]

I've noticed, that my Core i7 4500u processor got very slow results running the client at -1 or 3 or 4 CPU cores under Windows.
I only have GalliumOS as alternate OS, and haven't been able to run the client on it. Only from Google Chrome browser.
But in Windows, I found the best results with the client were 2 CPU cores on my system, even if the CPU has 4 threads.

The difference between -1 (automatic), 3, and 2 cores was between 1400-1700 credits, and 4500 credits.
The Core i7 4500u is a 2 core 4 thread system.

Load at 2 and 3 cores is identical (86% CPU).
I'm not sure if 4 cores brought it up to 100%, but either way, performance is lowered when more than 2 cores are selected.
In my case, 2 cores also allow the fan to run at mid-RPM range, rather than full speed.