Mxlinux is now upgraded to Mxlinux-21.

     The following describes what I am working on to address slow web response.  If you like tech details read on, else just know I’m working on a solution.

     I am in the process of acquiring components for a new web server.  The existing server is no longer able to handle the peak loads.  This came about rather suddenly and is a combination of increased traffic and a larger mix of applications verses flat-files being served.  Applications require CPU to execute something to create a page or possibly an interactive situation such as connecting to a shell server via the web.  In particular the popularity of friendica and nextcloud has increased, and nextcloud has become ever more bloated with each new version.

     There is a big plus to this, it has generated new customer interest and with that the money is available for new hardware.  But we were already running some fairly high performance hardware so now really reaching for stratospheric performance.  It’s easy to get a lot of threads at a low CPU speed or a few threads at a high speed but both at once are a challenge.

     The existing web server is on an i7-6850k based system which is a 6-core 12-thread CPU, and it is maxed out at 128GB of RAM, that’s as much as that platform can support.  Only the database is on SSD which means any fork/exec that isn’t cached involves disk access.  When the machine gets busy everything gets blown out of cache because there isn’t enough RAM and that’s when performance really suffers.  The machine is shared with Mint, Debian, and Ubuntu shell servers, those and the web server are all virtual machines on the same physical box.

     To address this I am building a new server that will be based on an i9-10940x which is a 14 core 28 thread CPU, most importantly it is capable of addressing 256GB of RAM.  The shell servers will remain on the existing machine and the web server will be on bare metal on the new server so it will have 4x the RAM available as the existing machine.  Additionally, not only the database but also system programs will be on SSD in a RAID-1 configuration so that fork/exec operations will be much faster.  The i9-10940x is rated at 3.6 Ghz but can be clocked at 5Ghz IF the heat can be adequately removed, so finding an adequate cooling solution is the main challenge.  I’m going to start with a Noctua 15 dual fan and replace the fans with some noisy but very high CFM.  The co-lo already sounds like you’re standing behind a 747 during take-off so noisy fans not a huge concern.  If that does not suffice may have to go liquid but I really prefer to avoid that since maintenance of liquid cooled systems tends to be a major headache.  So we’ll have more than twice the CPU cycles, 4x the memory, and much faster disk infrastructure.

     One thing making this take longer is I have to get rid of two old Sparc machines in order to make the power budget this will require available.  At the co-lo facility I have one 20 AMP circuit and converting the old antique free radius to the new version is some learning curve that is straining my 63 year old brain but making slow progress.

     You can login to any of our shell servers without a password by setting up ssh keys.  To do this, from a terminal on your machine type:

     ssh-copyid username@host.eskimo.com

     Where username is your login, and host is the shell server you want to connect to, for example, if I wanted to set them up for ubuntu I would type:

     ssh-copyid nanook@ubuntu.eskimo.com

     It will then prompt for your password.  After you type it you can then ssh username@ubuntu.eskimo.com or whichever host(s) you set up, and you should never need to type a password again.

     This prevents two potential issues, one it will prevent someone who installs a keyboard logger on your machine from obtaining authentication information for your accounts here.  Two, it will prevent you from getting locked out from too many mistyped passwords.

     Second bit of news, I will be attempting an upgrade of mxlinux from 19 to 21 this evening.  The dpkg program in mxlinux 19 is so old it will no longer install a modern kernel.  The machine will need to be rebooted during this procedure and may be unavailable for some time if things go wrong.  Since mxlinux does not include an updater program, this is, like debian, a very manual process fraught with potential errors.

     Our client mail server is having some issues with NFS connectivity to Ubuntu.  I am going to reboot it in hopes of clearing this problem however, NFS reconnecting is not 100% after a reboot in Linux so I will have to manually check all the shell servers which may take about twenty minutes after the reboot.

     A while back vnc and rdp no longer functioned on the various Redhat based servers, centos7, centos8, centos-stream, fedora, and scientific7.

     This was caused by an update the xrdp that required a newer version of openssl but they did not provider a corresponding upgraded openssl.

     That has since been resolved and these remote access protocols are now available on all of these machines.

     In addition, I have installed mosh on all of these machines.  Mosh is a remote access protocol designed to work around intermittent connectivity such as that provided by a cell phone.

     Our web server was unavailable or intermittently available this evening for about 1/2 hour owing to some hackers in France first trying to exploit a buffer overflow that existed in Apache 2.4.50, and when that failed just a high traffic denial of service attack that ran the server load over 250 briefly at which point it responded too slowly and most browsers timed out.  I had to set the timeout in Firefox to 60 seconds so I could connect and see what was going on with the server.

     I have since blocked all of the address space this attack was coming from, an address block in Paris France used for overflow belonging to OVH hosting, a hacker and spammer haven, 54.36.148.0/22 and compiled and installed Apache 2.4.51 which fixes this exploit (also required recompiling the apr and apr-util libraries as those installed by Ubuntu 20.04 were too old and lacked a function needed by Apache httpd 2.4.51).

     Debian is now, to the best of my knowledge, fully restored to health.  Most of the software that was previously online is there, there are a few packages I removed because I felt they hindered security without providing much use for remote users.  If there is something missing you need still let me know.

     What was formerly called pine, a mail program, is now alpine.  It is the same program but the University of Washington changed the name and license.

     What was formerly called pico, a small text editor used by default with pine, is now called nano.  It is the same program but the University of Washington changed the name and license.

     X-forwarding now works again.

     VNC and RDP now work again BUT I was only able to get them to work with the slick greeter and that greeter puts the Desktop selection on a pulldown menu that doesn’t have room for all of the Desktops installed, so if you use VNC or RDP you will not be able to choose any desktop.  X2go does not use the greeter so  you can choose any desktop with X2go.

     I discovered in the process that the original problem was caused by a conflict between ufw and firewalld.  Both were installed prior to the upgrade to Bullseye and both used to be mutually compatible, both entered rules into iptables and you just got the sum total of those rules.  Now however if you try to enable both it locks up the machine and it will not boot properly.  Firewalld no longer works by itself either, it does not enter the rules you specify into iptables, in short it is just plain broken so I am now using ufw exclusively for firewall control.

    I’ve made a backup in it’s fixed state so we should be able to recover from future maladies.

I found what was causing the boot problems, firewalld and ufw were simultaneously installed.  In earlier revisions these played good together but now they lock up when both installed.  And firewalld by itself does not seem to function anymore, does not add anything into iptables.

VNC is just giving a black screen.  Installed same as I had it before so having a hard time trying to determine why.

     I was trying to get vnc working and for some reason the firewall was blocking it.  I had ufw allow vnc but still couldn’t connect to it off of the machine.  So I turned off firewalld and the machine locked up.  Obviously this is not correct behavior and I am still troubleshooting.  Availability of debian is likely to be intermittent tonight and random crashes are quite probable during the troubleshooting phase.

     Debian is back up except I do not have all the graphical access working again and much software is not installed.

     I would appreciate your help.  Installing all the software previously installed is not workable as it causes systemd to get stuck in a loop and not boot properly.  I ended up re-installing a second time after working all night to recover it.  So I only want to install things people actually use.

     To that end, I’m asking if you are a regular Debian user, please login to debian.eskimo.com and check if software is present that you need, ESPECIALLY if you had cron jobs there (which I have not restored yet).  If software is missing, please use the ticket system (website https://www.eskimo.com/ under Support pulldown, Tickets) to initiate a ticket so I know what needs to be re-installed and/or fixed.

     There will be periodic downtime during this process in order to make frequent backups to essentially checkpoint the system so I don’t end up having to do yet another full re-install.

My apologies for kicking everyone off of Ubuntu.  Went to reboot Debian and accidentally rebooted the wrong server.