Caste leaders won’t be allowed to visit Ramnad memorial