The frequency of the 504 errors is not going to be entirely based on the number of users concurrently on the site, but also on what it is that they are doing.
From what I can tell (and with the help of a little Googling) the 504 error is reported by the Nginex webserver not getting a response back from the PHP-based forum software within a certain time. There could be a number of reasons for this... While the "spec" of the server (RAM, CPU etc) do have an effect on this, the architecture of the underlying software can mean that having multiple CPU's might just result in having any additional CPU's sitting idle, and if heaps of RAM is available, only that RAM up to a certain limit is actually used.
While I find the occasional 504 error annoying, I think we should remember how bad it was when the forum went away entirely... Lets face it - we can all log in, we can all post, we can all respond, we can all enjoy our hobby - Martin's doing a great job.
I have been in the situation where the failure of an IT system that I was responsible for took down an entire business and believe me it's not fun (and we have way more than the number of users this forum does to worry about), so I feel for Martin very deeply and I'm absolutely certain it will come right in the end.