Setting up a LAN cache on TrueNAS SCALE

Tech Homelab
1

I’m writing this guide so that others can benefit from the research and work I’ve done to get a LAN cache working on TrueNAS SCALE.

Motivation

My personal motivation for this project was that I am working on migrating away from Windows to Linux and dual-booting my desktop. There might be times where I want to play a game on either system, but having a single disk to share between them is known to cause a lot of issues due to the following:

  • NTFS is not officially supported for Steam on Linux and is actively discouraged by Valve.
  • Sharing NTFS disks / Steam game installs between systems corrupts the game files, corrupts the shader caches for both operating systems, and requires repairing / reinstalling games every time you boot into one OS from the other.

To avoid these problems altogether, I’m going to maintain separate game disks for each OS - which leads into the problem that I’ll need to download each game twice, one for each OS, which will be an undue burden on my internet. Even with a gigabit connection with my ISP, doubling the bandwidth usage for game downloads is going to slow things down a lot.

Goals

  1. Have a working autonomous LAN cache for Steam games
  2. Cache should work for both Windows and Linux systems
  3. Desktop should not having any DNS impact from using the cache DNS

lancache and steam-lancache-prefill

lancache-monolith (or lancache as I’ll be calling it) is an app that acts as a middle man between your devices and the CDN’s over the internet where you get files from. lancache works by replacing the DNS records of your devices so that your devices pull files from the LAN cache if they exist, and otherwise gets those files from the original CDN over the internet. The goal is so that you only need to download something like a game once, and then have multiple devices pull those same files over LAN instead of over the internet. You can find out more about it on their website here.

steam-lancache-prefill is an app that lets you preload the LAN cache with games belonging to your Steam account. This is great if you want to precache games on a server so that your client devices get the files extremely quickly, and we can set it up to run on a schedule so the caching is completely invisible to both us and our client devices. You can read more about it on their GitHub page here, or their wiki page here.

Use the TrueNAS lancache App

  1. Change TrueNAS SCALE’s WebUI ports
    1. In the WebUI for TrueNAS SCALE, go to System → General Settings
    2. Press the Settings button for the GUI section
    3. Change the HTTP port to something besides 80 (I used 8080)
    4. Change the HTTPS port to something besides 443 (I used 4433)
    5. Save
    6. When the WebUI reloads, log back in
  2. Create dataset
    1. Use the following settings:
      • Exec → On or Inherit (On)
      • ACL Type → Off
      • ACL Mode → Discard (this should be greyed out once you turn off ACL Type)
      1. I choose to enable atime and dedup, but these not necessary unless you plan to do the following:
        • Download copies of games for multiple operating systems
        • Have more than one user use the Steam Prefill tool
    2. The disks you should use depends on your LAN speeds:
      1. WiFi 5 or 1Gbps ethernet: HDD’s, or at most SATA SSD’s
        1. You can supplement an HDD pool with L2ARC VDEV and/or a SLOG VDEV to improve both read and write speeds. L2ARC will probably make the most difference as a SLOG will be limited by your internet speeds and the speed of each CDN you download files from rather than disk write speeds for the pool.
      2. Wifi 6 & 7 or 2.5 - 10 Gbps ethernet: SATA SSD’s
      3. >10Gbps ethernet: NVMe SSD’s
    3. After creating the dataset, open the shell (or SSH into the TrueNAS server) and create a new directory like so: sudo mkdir /mnt/GameCache/logs
  3. Set up lancache app
    1. In the TrueNAS WebUI, go to Apps
    2. Search for lancache and choose to install it
      1. LanCache Monolithic Configuration
        • Upstream DNS → whatever DNS you want to use, defaults to Google DNS
        • Cache Memory Size (in MB) → Leave at default (500)
        • Cache Disk Size (in GB): Whatever cache size you want, should not be higher than the capacity of the dataset.
      2. Network Configuration
        • For both HTTP Port and HTTPS Port set the Port Bind Mode to Publish port on the host for external access
        • Port Number for HTTP Port: 80
        • Port Number for HTTPS Port: 443
      3. Storage Configuration
        1. LanCache Monolithic Cache Storage
          • Type: Host Path (Path that already exists on the system)
          • Host Path: Choose the dataset you created at the beginning
        2. LanCache Monolithic Logs Storage
          • Type: Host Path (Path that already exists on the system)
          • Host Path: Choose the path to the logs directory you created (in my case that is `/mnt/GameCache/logs)
      4. Resources Configuration: set however many CPU cores and however much RAM you can afford to allocate. In your personal home probably don’t need more than 4 cores and 8 GB (8192 MB) of RAM
  4. Set your DNS to your TrueNAS Server’s IP
    1. Windows:
      1. Open the Settings app, go to Network & Internet
      2. Click Wifi if you’re using Wifi otherwise click Ethernet
      • Wifi:
        1. Manage known networks → Click the network you use at home
        2. Click the Edit button next to DNS Server Assignment
        3. Change it from Automatic (DHCP) to Manual
        4. Turn on IPv4
        5. Type in the IP address of your TrueNAS server in the Preferred DNS field
        6. Click Save
      • Ethernet:
        1. Click the Edit button next to DNS server assignment
        2. Change it from Automatic (DHCP) to Manual
        3. Turn on IPv4
        4. Type in the IP address of your TrueNAS server in the Preferred DNS field
        5. Click Save
  5. Flush current DNS
    • Windows: ipconfig /flushdns
    • Linux (systemd): sudo resolvectl flush-caches
    • Validate new DNS is correct (any OS): nslookup lancache.steamcontent.com
      • This should list your TrueNAS server IP
      • Can test for any typical CDN’s used for different services such as Blizzard or Xbox
  6. Clear Steam download cache: Open Settings → Downloads → Clear cache
  7. Test single game caching
    1. Pick a game from your Steam library
      • Ideally not already installed
      • Ideally very small (<5GB)
    2. Install the game, measure the network speeds in the Steam downloads page including the peak download speeds
    3. Uninstall the game from Steam
    4. Reinstall the game, monitor the network speeds again. They should be a lot higher than they originally were as the cache should now have that game’s files available locally. I saw some games go from 100 MB/s (0.8Gbps) before caching to 450MB/s (3.6Gbps) after caching.

Manual setup with Docker Containers

  1. Follow the steps 1 and 2 from the app version above
  2. Set up lancache-dns container
    • sudo docker run --restart unless-stopped --name lancache-dns --detach -p 53:53/udp -e USE_GENERIC_CACHE=true -e LANCACHE_IP=$HOST_IP -e UPSTREAM_DNS=8.8.8.8 lancachenet/lancache-dns:latest
    • I set the upstream DNS to Google’s DNS at 8.8.8.8 but you can set it to anything, even your own router’s DNS server
  3. Set up lancache
    • sudo docker run --restart unless-stopped --name lancache --detach -v /mnt/GameCache:/data/cache -v /mnt/GameCache/logs:/data/logs -p 80:80 -p 443:443 lancachenet/monolithic:latest
    • Like the app-based install, we are forwarding ports 80and 443 of the container to the same ports on the host. This allows the lancache-dns to access the actual lancache app.
  4. Continue on step 4 from the app version above

Setting up steam-lancache-prefill

This step is optional and requires to log into your Steam account with this app. This app, just like Steam, doesn’t save your password and uses your login access token that isn’t saved to disk. If you don’t trust this, skip this section, but as per Steam Prefill’s FAQ page it should be completely secure.

Although I’d normally suggest using containers, this app has some configuration files that you can only modify if it’s not in a container (or at least it’s a lot harder to modify with the container), and the container method causes a lot of old container volume data to build up over time which takes up space.

Normal Prefill Setup with Accessible Configuration Files

  1. Open the TrueNAS shell (or an SSH session to it)
  2. Navigate to your dataset: cd /mnt/GameCache
  3. Make a new directory for this code: sudo mkdir SteamPrefill && cd SteamPrefill
  4. Install the app
    1. Run the command sudo curl -o update.sh --location "https://raw.githubusercontent.com/tpill90/steam-lancache-prefill/master/scripts/update.sh"
    2. Then run sudo chmod +x update.sh
    3. Then run sudo ./update.sh
    4. Finally, run chmod +x ./SteamPrefill
  5. Now you can use the app by running the command sudo ./SteamPrefill !

Prefill with Containers

  1. Open the TrueNAS shell (or an SSH session to it)
  2. Run the following command to download the container image and run it: sudo docker run -it --rm --net=host --volume ~/.config/SteamPrefill:/Config tpill90/steam-lancache-prefill:latest

You’ll be using that exact command every time you want to use the prefill app through a container. I recommend making an alias for it so you don’t have to type this whole thing put every time by doing the following:

  1. Open the TrueNAS shell (or an SSH session to it)
  2. Run this command to open the bash shell’s default environment config file: nano ~/.bashrc
  3. Scroll all the way down and make a new line, pasting in this snippet: alias PrefillContainer='sudo docker run -it --rm --net=host --volume ~/.config/SteamPrefill:/Config tpill90/steam-lancache-prefill:latest'
  4. Press CTRL+X to close the file, and when prompted to save the file just hit ENTER.
  5. Do one of the following to update the shell environment:
    1. Exit and re-open the shell
    2. If using SSH, exit and re-connect over the SSH
    3. Run the command source ~/.bashrc
  6. Run the command PrefillContainer and you should see the regular output of the app

Prefill Usage

Regardless of how your set up the prefill app, you will use the following options to actually do the prefilling.

  1. Run your prefill command with the added option at the end select-apps to get an interactive window to select games to add to the cache manually.
    1. You’ll be asked to sign into Steam and use Steam Guard if you have it enabled (which you really should).
  2. To actually load the cache with your selected games, run the original command with the prefill option (instead of select-apps)
  3. If you want to prefill without manually selecting games, you have some options:
    1. Using prefill --all will preload ALL games that are in your library. BIG WARNING: it really means every single game in your library - not just the installed ones. This is most likely not what you want and will take up a massive amount of space on your server with little to no use for most of the cached games.
    2. Using prefill --recent will grab all games you played in the last 2 weeks, arguably the most useful option here.
    3. Using prefill --recently-purchased will grab all of the games you purchased in the last 2 weeks.
    4. Using prefill --top N will grab the most popular games by player count as measured in the last 2 weeks. N is an optional number to say to grab the top N games, with a value range of 1-100. By default this value is 50 if you don’t provide N
  4. All of these prefill options can also take the --os argument that takes values of windows, linux, and macos. You can select multiple operating systems by just separating the values by a space like --os windows linux

In my case, I ran prefill --os windows linux --recent --force to grab my recent games for both Windows and Linux.

Actually Using Prefill

From here you have two options:

  1. Use the select-apps option to choose what games to cache, or prefill --all to just cache every game you own. You ideally should only need to run this step again when you want to add/remove a game from the cache.
  2. If you used the non-container version, you can edit the /mnt/GameCache/SteamPrefill/Config/selectedAppsToPrefill.json file to add in all the games you want to prefill by their game ID’s.
    1. You can get the game’s ID by checking SteamDB, or otherwise right-clicking on the game in your Steam library → Properties → Updates tab and tpying in the same ID as listed for App ID

Once you’ve selected your apps one way or another, you’ll want to actually prefill them. If you used select-apps it’ll prompt you to prefill after you confirm your selections. Either after this or if you want to manually prefill the games, you’ll have to run the app using prefill with whatever options you need.

Now we can do this manually or have this work as we download games from Steam, but we can also update the cache automatically on a schedule. This is great because when we do want to download games or updates, they’ll already be on the cache and we can start gaming sooner!

  1. In the TrueNAS WebUI, go to System → Advanced Settings
  2. Under the Cron Jobs section, press the Add button
    • Description: Steam Prefill Job
    • Command:
      • If you used the non-container version: sudo /mnt/GameCache/SteamPrefill/SteamPrefill prefill
      • If you used the container version: sudo docker run -it --rm --net=host --volume ~/.config/SteamPrefill:/Config tpill90/steam-lancache-prefill:latest prefill
    • Run As User: root
    • Schedule: Whatever works best for you, I do 7AM in the morning (Custom, keep all at default except change Hours to 7) so I can have the cache grab any updates before I start my day.
      You can always set up the schedule to be more or less frequent. If you need help understanding how to set up the schedules, use https://crontab.guru to create the cron schedule you’re looking for, or consider making multiple cron jobs.

If you’re like me and are using multiple operating systems, you just need to list both operating systems when you use the --os option in the same command like prefill --os windows linux

FAQ and other information

  • Why does the app in the TrueNAS app tab only need one container, but the manual setup requires two containers?
    • The app available in the TrueNAS app tab combines both the lancache container that handles the caching, and the lancache-dns container that routes DNS requests to your lancache container instead of the CDN for the services over the internet.
  • Why do I have to change the ports for the TrueNAS WebUI?
    • lancache is made up of two services (lancache and lancache-dns) that need to communicate to each other. For some reason with the app version, you can’t change the ports that they use to communicate to each other correctly, meaning that if HTTP/HTTPS aren’t set to 80/443 respectively then the lancache will not work correctly. I couldn’t find any options inside the container to fix this at this time
    • This should not be required for the manual Docker container version, but you’ll have to change some of the container commands that I frankly don’t care to test out. I recommend changing the WebUI ports as it makes setup easier.
  • Why do you recommend using the Steam Prefill as a non-container?
    • First, this means you’re not clogging up your server’s space with old Docker container volumes
    • Second, if you’re like me and live in a house with more than one person who uses Steam, you can set up Steam Prefill for each user separately so everyone can have their own cache. This also works great as ZFS with deduplication enabled will decrease the space used for games that multiple users have. Note that ZFS dedup does not guarantee perfect space reduction of 1/N for N number of users, as ZFS operates using data blocks for deduping data - at best you’ll see only 1 copy of the data actually take up space, and at worst you’ll see N number of copies if the data blocks don’t line up perfectly.
    • You can read on how to do this in the Steam Prefill’s FAQ page
  • Does prefilling Steam games for multiple operating systems take up more space?
    • If there is a separate version of the game for a different OS (like Windows and MacOS builds that are fundamentally built differently) then you’ll effectively take up to N times space (N being the number of operating systems you download a version of the game for)
    • You can greatly mitigate this by using ZFS’ deduplication feature - read the above Why do you recommend using the Steam Prefill as a non-container? FAQ on how ZFS dedup works, but for this case it’ll also depend on how similar the same files are between the different OS versions of a given game ontop of how the blocks align in ZFS.

Closing

This setup should cover all the requirements I set out to meet at the beginning of this task, and should hopefully help some others in accomplishing something similar for their homelabs. Most of these steps should work on other operating systems, and I hope that this guide helps make that process easier as well.

There are definitely some improvements that we can make for this setup (like not having to change the TrueNAS WeBUI ports so the containers work right) but I’m very happy with what this guide accomplishes.

This thread is for people to post suggestions & improvements, ask questions, and get help with their own setups. So go on an build your own LAN cache and let us know how it goes!

2 Likes