When you have decreased FPS in game the client processing the packets slower as well, it was like this on retail too. Write a macro with /ping commands in Macro section and run it in an empty room with high fps, and on siege / city / crowded place, there will be a notable differnece. Otherwise under normal cirumstances and high fps I have 31-50 ms on coax cable net from Hungary using WTFast. Without WTFast I have issues, but that's because my ISP is crappy.
If you have this regularly maybe the routing our ISP choosing under heavy load (evenings) is not optimal, try a gaming proxy, there are many out there with free trials. Otherwise I'm not experiencing this / also there would be many topic if it was a server issue imo. Once my ISP was a dick on retail a few years ago and had similar issue like you did, alternative routing helped.
Put here a tracert output when you start to have heavy latency. Based on the custom start command the client is using to start the game IP is 51.91.6.184.