Muslims had agreed to temple on Babri site: Vishnu Hari Dalmia, Former VHP chief