How To Host Your Own APT Mirror

In this post I will describe the necessary steps to host your own APT mirror that you can use in your local network to speed up package installations and updates, and save bandwidth.

Requirements

  • A Debian server/VM
  • About ~750GB of HDD space (more if you want to host multiple architectures and/or more repositories)

Installation

Installing the system itself is outside the scope of this post, you can refer to the post linked above for a guide on installing Debian remotely, you can use a different method if you prefer, as long as you have a functioning system in the end.

Setting up apt-mirror

After you installed Debian and did the initial hardening/configuration according to your needs, you can start with installing the dependencies.

  1. Install apt-mirrorapt install apt-mirror apt-transport-https ca-certificates
  2. Configure apt-mirror:
    1. Open the configuration file /etc/apt/mirror.list using your favorite editor and paste the following:
      ############# config ##################
              
       ## This option sets the base path for apt-mirror to use.
       set base_path    /var/apt-mirror
              
       ## The options below allow you to change the locations
       ## that are usually located in the base_path.
       # set mirror_path  $base_path/mirror
       # set skel_path    $base_path/skel
       # set var_path     $base_path/var
       # set cleanscript $var_path/clean.sh
              
       ## You can change the default architecture using this option.
       # set defaultarch  <running host architecture>
              
       ## You can use the options below to run a script
       ## after the mirroring finished.
       # set postmirror_script $var_path/postmirror.sh
       # set run_postmirror 0
              
       ## The number of threads to use for downloading packages. 
       set nthreads 8
              
       ## Packages with a tilde (~) in their version are pre-release,
       ## we don't want them in the mirror.
       set _tilde 0
              
       ############# end config ##############
              
       ##
       # Debian v12 Bookworm
       ##
              
       # Debian Bookworm
       deb https://deb.debian.org/debian bookworm main contrib non-free non-free-firmware
       deb-src https://deb.debian.org/debian bookworm main contrib non-free non-free-firmware
       deb-amd64 https://deb.debian.org/debian bookworm main contrib non-free non-free-firmware
       # deb-arm64 https://deb.debian.org/debian bookworm main contrib non-free non-free-firmware
              
       # Debian Bookworm Updates
       deb https://deb.debian.org/debian bookworm-updates main contrib non-free non-free-firmware
       deb-src https://deb.debian.org/debian bookworm-updates main contrib non-free non-free-firmware
       deb-amd64 https://deb.debian.org/debian bookworm-updates main contrib non-free non-free-firmware
       # deb-arm64 https://deb.debian.org/debian bookworm-updates main contrib non-free non-free-firmware
              
       # Debian Bookworm Backports
       deb https://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware
       deb-src https://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware
       deb-amd64 https://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware
       # deb-arm64 https://deb.debian.org/debian bookworm-backports main contrib non-free non-free-firmware
              
       # Debian Bookworm Security
       deb https://deb.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware
       deb-src https://deb.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware
       deb-amd64 https://deb.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware
       # deb-arm64 https://deb.debian.org/debian-security bookworm-security main contrib non-free non-free-firmware
              
       ##
       # Other Stuff
       ##
              
       # Clean Scripts
       clean https://deb.debian.org/debian-security
      
    2. Make sure to update the configuration according to your needs, especially the base_path and nthreads options, in case you want/need to use a different storage path or more threads to sync the mirror.
    3. In case you want to also mirror arm64 packages, you can uncomment the deb-arm64 statements in the repo list.
  3. Add a CRON Job for the mirroring:
    1. Use crontab -e to open the crontab editor using your favorite editor
    2. Add: 0 1 * * * /usr/bin/apt-mirror > /var/log/apt-mirror.log
    3. This will run every day at 1:00, you can of course change this as you want.
  4. Run the first mirroring (manually)
    1. I’d recommend using screen for this (or any other similar tool you prefer): screen -S first-mirror
    2. Run apt-mirror and wait until the execution finishes
Regarding HTTPS

There are ongoing discussions on various websites about whether APT should use HTTPS/TLS by default (currently, it does not). I used HTTPS URLs for the mirror in this guide to provide a secure default, but the webserver of the mirror itself (which is assumed to run in the local network) is running with HTTP in order to keep the guide simple. You can find some examples of configuring NGINX for TLS on this blog (for example you could search for 443 to find config examples).

Setting up NGINX

Now that the mirror content is synced, you need a way to fetch the mirrored content from your server. NGINX will be used in this guide together with the fancyindex module in order to render nice looking indexes.

  1. Install NGINX: apt install nginx libnginx-mod-http-fancyindex
  2. Configure NGINX
    1. Open the default config at /etc/nginx/sites-enabled/default using your favorite editor
    2. Paste the following:
      server {
       listen 80 default_server;
       server_name _;
      
       access_log /var/log/nginx/mirror.access.log;
       error_log  /var/log/nginx/mirror.error.log;
      
       server_name_in_redirect off;
      
       autoindex off;
       server_tokens off;
      
       root /var/www/html;
      
       location /debian {
           alias /var/apt-mirror/mirror/deb.debian.org/debian;
      
           fancyindex on;
           fancyindex_exact_size off;
           fancyindex_header /head.html;
           fancyindex_footer /foot.html;
       }
      
       location /debian-security {
           alias /var/apt-mirror/mirror/deb.debian.org/debian-security;
      
           fancyindex on;
           fancyindex_exact_size off;
           fancyindex_header /head.html;
           fancyindex_footer /foot.html;
       }
      
       error_page 400 401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 421 422 423 424 425 426 428 429 431 451 500 501 502 503 504 505 506 507 508 510 511 /error.html;
      
       location /error.html {
           root /var/www/html;
           internal;
       }
      }
      
    3. Don’t forget to update the paths of the alias params in case you changed the path in the apt-mirror config
  3. Add the HTML files (I’ll provide unstyled examples):
    1. Add /var/www/html/index.html (replace MIRROR_DOMAIN with the actual DNS name (or IP(?)) of the mirror:
        <!DOCTYPE html>
        <html lang="en">
        <head>
            <title>APT Mirror</title>
        </head>
        <body>
        <h1>APT Mirror</h1>
        <p>This is an APT mirror.</p>
        <h2>Currently hosting:</h2>
        <ul>
            <li>
                <a href="/debian">/debian</a>
                <ul>
                    <li>bookworm</li>
                    <li>bookworm-updates</li>
                    <li>bookworm-backports</li>
                </ul>
            </li>
            <li>
                <a href="/debian-security">/debian-security</a>
                <ul>
                    <li>bookworm-security</li>
                </ul>
            </li>
        </ul>
        <h2>Usage</h2>
        <p>Select the repos you need and add them to your <code>/etc/apt/sources.list</code>:</p>
        <details>
            <summary>Debian v12 (bookworm)</summary>
            <pre>
        # Debian Bookworm
        deb http://MIRROR_DOMAIN/debian bookworm main contrib non-free non-free-firmware
        deb-src http://MIRROR_DOMAIN/debian bookworm main contrib non-free non-free-firmware
              
        # Debian Bookworm Updates
        deb http://MIRROR_DOMAIN/debian bookworm-updates main contrib non-free non-free-firmware
        deb-src http://MIRROR_DOMAIN/debian bookworm-updates main contrib non-free non-free-firmware
              
        # Debian Bookworm Security
        deb http://MIRROR_DOMAIN/debian-security bookworm-security main contrib non-free non-free-firmware
        deb-src http://MIRROR_DOMAIN/debian-security bookworm-security main contrib non-free non-free-firmware
              
        # Debian Bookworm Backports
        deb http://MIRROR_DOMAIN/debian bookworm-backports main
        deb-src http://MIRROR_DOMAIN/debian bookworm-backports main
                </pre>
        </details>
        </body>
        </html>
      
    2. Add /var/www/html/error.html:
        <!DOCTYPE html>
        <html lang="en">
        <head>
            <title>APT Mirror</title>
        </head>
        <body>
        <h1>Error!</h1>
        <p class="lead">An error occurred.</p>
        <p><a href="/">Main Page</a>
        </body>
        </html>
      
    3. Add /var/www/html/head.html:
        <html lang="en">
        <head>
            <title>APT Mirror</title>
        </head>
        <body>
        <h1>APT Mirror</h1>
        <p>Welcome to the APT Mirror.</p>
        <h1>
      
      • This gets cut off at the open <h1> because the fancyindex module will render the title of the folder it’s showing after that and close it.
    4. Add /var/www/html/foot.html:
        <hr />
        <p>Thanks for using the APT Mirror.</p>
        </body>
        </html>
      
    5. Remove /var/www/html/index.nginx-debian.html in case it exists
    6. Ensure the files are not world-writable and belong to either root or www-data
  4. Confirm that the NGINX config is valid by running: nginx -t
  5. Restart NGINX: systemctl restart nginx

You should now be able to open http://MIRROR_DOMAIN/ (changed to your actual DNS name) in a browser and also view the repo contents. I added example usage instructions into the example index.html, so you can copy the example sources.list contents from there.

The next step would be adding styles to the mirror pages so it looks better, and of course add more/different repos/architectures in case you need them.

I hope this post was useful. If you have any suggestions or found errors please let me know!

rtw89 A repo for the newest Realtek rtw89 codes. This repo now contains the code for the Realtek RTW8922AE, which is a Wifi 7 device. It has been tested using a Wifi 6 AP as I do not have access to a Wifi 7 model. The driver works very well. This repo is current with rtw-next up to April 3, 2024. This branch was created from the version merged into the wireless-next repo, which is in the 5.16 kernel. IF YOU USE DRIVERS FROM THIS REPO FOR KERNELS 5.16+, YOU MUST BLACKLIST THE KERNEL VERSIONS!!!! FAILING TO DO THIS WILL RESULT IN ALL MANNER OF STRANGE ERRORS. This code will build on any kernel 6.10 and newer as long as the distro has not modified any of the kernel APIs. IF YOU RUN UBUNTU, YOU CAN BE ASSURED THAT THE APIs HAVE CHANGED. NO, I WILL NOT MODIFY THE SOURCE FOR YOU. YOU ARE ON YOUR OWN!!!!! Note that if you use this driver on kernels older than 5.15, the enhanced features of wifi 5 and wifi 6 are greatly crippled as the kernel does hot have the capability to support the new packet widths and speeds. If you use such a kernel, you might as well have an 802.11n (wifi 4) device. This repository includes drivers for the following cards: Realtek 8851BE, 8852AE, 8852BE, 8852CE, and 8922AE. If you are looking for a driver for chips such as RTL8188EE, RTL8192CE, RTL8192CU, RTL8192DE, RTL8192EE, RTL8192SE, RTL8723AE, or RTL8723BE, these should be provided by your kernel. If not, then you should go to the Backports Project (https://backports.wiki.kernel.org/index.php/Main_Page) to obtain the necessary code. If you have an RTW8822B{E,U,S}, RTW8822C{E,U,S}, RTW8723D{E,U,S}, or RTW8821C{E,U,S}, then you should use the drivers at https://github.com/lwfinger/rtw88.git. Installation instruction Requirements You will need to install "make", "gcc", "kernel headers", "kernel build essentials", and "git". For Ubuntu: You can install them with the following command sudo apt-get update sudo apt-get install make gcc linux-headers-$(uname -r) build-essential git Users of Debian, Ubuntu, and similar (Mint etc) may want to scroll down and follow the DKMS instructions at the end of this document instead. For Fedora: You can install them with the following command sudo dnf install kernel-headers kernel-devel sudo dnf group install "C Development Tools and Libraries" For openSUSE: Install necessary headers with sudo zypper install make gcc kernel-devel kernel-default-devel git libopenssl-devel For Arch: After installing the necessary kernel headers and base-devel, git clone https://aur.archlinux.org/rtw89-dkms-git.git cd rtw89-dkms-git makepkg -sri If any of the packages above are not found check if your distro installs them like that. Installation For all distros: git clone https://github.com/lwfinger/rtw89.git cd rtw89 make sudo make install Installation with module signing for SecureBoot For all distros: git clone https://github.com/lwfinger/rtw89.git cd rtw89 make sudo make sign-install You will be prompted with a password, please keep it in mind and use it in the next steps. Reboot to activate the new installed module. In the MOK management screen: Select "Enroll key" and enroll the key created by above sign-install step When prompted, enter the password you entered when create sign key. If you enter wrong password, your computer won't be bootable. In this case, use the BOOT menu from your BIOS, to boot into your OS then do below steps: sudo mokutil --reset Restart your computer Use BOOT menu from BIOS to boot into your OS In the MOK management screen, select reset MOK list Reboot then retry from the step to make sign-install How to unload/reload a Kernel module sudo modprobe -rv rtw_8852ae sudo modprobe -rv rtw89core #These two statements unload the module Due to the behavior of the modprobe utility, it takes both to unload. sudo modprobe -v rtw_8852ae #This loads the module A single modprobe call will reload the module. Uninstall drivers For all distros: sudo make uninstall Problem with recovery after sleep or hibernation Some BIOSs have trouble changing the power state from D3hot to D0. If you have this problem, then sudo cp suspend_rtw89 /usr/lib/systemd/system-sleep/. That script will unload the driver before sleep or hibernation, and reload it following resumption. Option configuration IMPORTANT: If you have an HP or Lenovo laptop, Their BIOS does not handle the PCIe interface correctly. To compensate, run the following command: sudo cp 70-rtw89.conf /etc/modprobe.d/. Then unload the drivers and reload. You should see the options appended to the end of the rtw89_pci or rtw89pci load line. If it turns out that your system needs one of the other configuration options, then do the following: sudo nano /etc/modprobe.d/<dev_name>.conf There, enter the line below: options <driver_name> <<driver_option_name>>=<value> The available options for rtw89pci are disable_clkreq, disable_aspm_l1, and disable_aspm_l1ss. The available options for rtw89core are debug_mask, and disable_ps_mode. If after rebooting the wifi still doesn't work, it might mean that it was not loaded. To fix that, you will have to manually rebuild initramfs. To do that, execute one of the two commands, depending on how old/new your system is. mkinitrd # If you're running an older system dracut -f --regenerate-all # If you're running a newer system After rebuilding initramfs, reboot your computer and check if the wifi works properly now. Normally, none of these will be needed; however, if you are getting firmware errors, one or both of the disable_aspm_* options may help. They are needed when a buggy BIOS fails to implement the PCI specs correctly. When your kernel changes, then you need to do the following: cd ~/rtw89 git pull make clean make sudo make install ;or sudo make sign-install Remember, this MUST be done whenever you get a new kernel - no exceptions. These drivers will not build for kernels older than 5.8. If you must use an older kernel, submit a GitHub issue with a listing of the build errors, but be aware that doing so will cripple your device. Without the errors, the issue will be ignored. I am not a mind reader. When you have problems where the driver builds and loads correctly, but fails to work, a GitHub issue is NOT the best place to report it. I have no idea of the internal workings of any of the chips, and the Realtek engineers who do will not read these issues. To reach them, send E-mail to linux-wireless@vger.kernel.org. Include a detailed description of any messages in the kernel logs and any steps that you have taken to analyze or fix the problem. If your description is not complete, you are unlikely to get any satisfaction. One other thing - your mail MUST be plain test. HTML mail is rejected. DKMS packaging for debian and derivatives DKMS is commonly used on debian and derivatives, like ubuntu, to streamline building extra kernel modules. By following the instructions below and installing the resulting package, the rtw89 driver will automatically rebuild on kernel updates. Secure boot signing will happen automatically as well, as long as the dkms signing key (usually located at /var/lib/dkms/mok.key) is enrolled. See your distro's secure boot documentation for more details. Prerequisites: sudo apt install dh-sequence-dkms debhelper build-essential devscripts git-build-recipe This workflow uses devscripts, which has quite a few perl dependencies. You may wish to build inside a chroot to avoid unnecessary clutter on your system. The debian wiki page for chroot has simple instructions for debian, which you can adapt to other distros as needed by changing the release codename and mirror url. If you do, make sure to install the package on your host system, as it will fail if you try to install inside the chroot. Build and installation # If you've already built as above clean up your workspace or check one out specially (otherwise some temp files can end up in your package) git clean -xfd git deborig HEAD dpkg-buildpackage -us -uc sudo apt install ../rtw89-dkms_1.0.2-3_all.deb This will install the package, and build the module for your currently active kernel. You should then be able to modprobe as above. It will also load automatically on boot. A note regarding firmware Firmware from userspace is required to use this driver. This package will attempt to pull the firmware in automatically as a Recommends. However, if your distro does not provide one of firmware-realtek >= 20230117-1 or linux-firmware >= 20220329.git681281e4-0ubuntu3.10, the driver will fail to load, and dmesg will show an error about a specific missing firmware file. In this case, you can download the firmware files directly from https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git/tree/rtw89. 将上述内容翻译成中文
09-25
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值