How much CPU and RAM usage are you seeing? On my machines I see 5% or less CPU and between 25 and 60MB RAM. The RAM changes up and down over time and the memory usage is about the same as Windows Media Player (and much less than Dropbox!).
Keep in mind this is a .NET application. The code is managed so it is really up to Windows to handle memory usage. However, if the memory usage is higher than what I just mentioned please let us know.
BTW, the check box for turning the Visualizer off is at the bottom of the Settings tab.