Regular members have noticed that the performance of "The Coffeeshop" web site has been less that they would wish recently, with connection and server unavailability issues.
Comment has been made that the information I have posted about the problems has been incomplete, and that I have failed to give timescales and pointers to when and how the matter would be resolved. Those comments were correct. I was investigating intermittent problems that left no obvious evidence in the system log files (so that I could not predict when or how the issue wouldbe found and resolved) and were of such a character that they could have been caused by third parties who could have gained further knowledge of the server from what I posted, to the detriment of a quick solution.
I have excellent news.
Actions I taken have elimitated the connection and server unavailability issue. These things are never quite as clear cut as I would like though, as only time will provide the full confirmation.
More technical?We use "virtual hosting" for the Coffeeshop, which means that a single web server looks after a whole series of domains. And one of those other domains was running a piece of commercial (sold) software which had a weakness that could be exploited when it was run on our particular configuration. Being a very widely used piece of software, there was a build up over time of knowledge of the exploit, and I'm reasonanly convinced that the majority of access failures were caused not by someone looking to "hit" the coffesshop but by general noise and perhaps automata that were aware of it.
The effect of the exploit is characterised by the server starting additional threads / copies of itself, which lead to a very poor performance. A "heartbeat" script which I installed a while back allowed the host computer running the server to 'clean up' as things went awry, but such a script had to clear out the good threads with the bad, thus dropping connections, and it then had to do housekeeping before it restarted - thus the 'server unavailable'.
The software that includes the exploit is no longer running on this computer in the same way, although the heartbeat script is. And the heartbeat script includes a log file so that I can get back and see when if and when it has had to act. Indeed - these are the log files I was able to use earlier to report, with good authority, on server stats which were far poorer that I wished but far better than I saw suggested in various quarters.
One of the big issues that comes up when you're doing technical support work such as this on a server / website is what your menbers, and other visitors who may not be fully in support of you, think and do while it's going on. Even with as much information provided in public as I felt reasonable and prudent - bearing in mind I didn't know when the issue would be fixed and I could have been giving vital data to someone orchestrating the events - there ends up being elements of customer service, firefighting, crowd control of an impatient crowd. Call them whichever you prefer. And I am sure you have seen some evidence. What you didn't see in full was a variety of changes to traffic patterns, which lead me to the conclusion (in hindsight) that although the initial problems were as described above, I was not being helped in resolving it due to the red herrings being thrown up by a tiny minority - a handful - with other agenda than helping get the robusteness back.
If you have any questions, please feel free to p.m. or email me (
graham@wellho.net), and I'll do my best to answer you as fully as I can.
ConclusionI think we're well back on track! The server looks robust. And a couple of other loopholes have been tightened which help to ensure our ongoing integrity and security. "The Coffeeshop" and its team of moderators look forward to the continued active support of members.
