The down side is that there is increased end-to-end latency due to the requirement to translate between the time and frequency domains (and back again).
Two common methods of performing this task are the Overlap And Add or the Overlap And Save.
These algorithms are summarized in the following document : http://www.numerix-dsp.com/tutorials/DSP/FrequencyDomainFiltering.pdf
If you have found this solution useful then please do hit the Google (+1) button so that others may be able to find it as well.
Numerix-DSP Libraries : http://www.numerix-dsp.com/eval/