Housing minister contradicts PM and insists landlords are working peopl