Tonight’s Outage

     Tonight’s outage was not the result of a hardware or software error, rather the result of an operator error.  I had built a new kernel and had intended to try it on my workstation before deployment but I also had a window open on the main file server because that is where I store and distribute kernels from and also where I have the configuration files.  I went to reboot my workstation but was in the wrong terminal and rebooted the server instead.  And because I hadn’t shut the virtual machines on it down properly, it did not come up cleanly, in particular the kernel NFS server was snarled and restarting it did not correct, so a second reboot was necessary.

     We will be performing a kernel upgrade to 6.1.9 this Friday, not because there are any obvious issues for 6.1.7, operator errors aside, it has been very stable, but because I made an error and misconfigured it.  I’ve corrected this on the web server which is most sensitive to this but really need to fix it on all machines.  And since 6.1.9 does have some minor fixes might as well get that in place.

    I am most looking forward to the release of 6.2, because it has some fixes that largely recover the performance lost to the various security work-arounds for the Intel Skylake chips and two of our physical servers are based upon this architecture.

Kernel Upgrade

     I made an error when I configured the last kernel.  While 6.1.7 does appear to be stable AND it appears to have fixed the long standing NFS bug for which I enabled the extra debugging, I accidentally compiled it with premption which I do not want on a server as it adds additional context switching overhead and decreases overall efficiency.  Thus I am going to be making a new kernel at least for the web server tonight (which is most affected) and will be doing a kernel upgrade just to fix this on the rest of the servers next Friday.  In the meantime, things may at times get a little slow.

Default PHP is now 8.0

     We have changed the default PHP from version 7.4 to version 8.0 now.

     Everything from 5.6 to 8.2 is available, anything older than 8.0 is no longer receiving security updates so should be avoided IF POSSIBLE but some older applications may not work with 8.0.

     It is best to update those applications if possible, but if not you can use an .htaccess file to override the default PHP version.  See:

     https://www.eskimo.com/support/override-php-version/

Kernel Upgrades Jan 20 11PM PST (GMT-0800)

     We will be upgrading to the 6.1.7 kernel this evening at 11pm.  Because KASAN caused issues with some of our servers, some would not boot with it, some were slower, we will only be putting it on two NFS servers that have been problematic.  I believe however that 6.1.7 has already addressed the bug because I found a patch in the changelog that addresses exactly the issue we’ve been experiencing, a use after free in nfsd.

     We will be rebooting centos7 and scientific7 earlier in the afternoon because of difficulties in upgrading those kernels that requires some extra processes.

     Tonight will affect all services, if all goes well we should be done by 11:30 and no service should be out more than 10 minutes EXCEPT for Yacy.  Yacy rebuilds it’s database upon reboot and this takes 30-45 minutes.

     This will also affect all of our fediverse servers, https://friendica.eskimo.com/, https://hubzilla.eskimo.com/, https://nextcloud.eskimo.com/ (currently unfederated owing to a plugin problem), and https://yacy.eskimo.com/.