I agree. You have to read the book and take into account when it was written. And it probably is more sexist than racist.
A lot of people spend too much time looking for symbols, portents and signs in literature when, sometimes, the book is simply a damn good read.
Probably the underlying reason for the accusation is the strong similarity between LOTR and Wagner's Ring Cycle. The real reason for such a similarity is probably not so much an empathy with Wagner's racist views - but in the fact that both works were born from the same Norse myths and legends.