Fed Libs not to blame in Vic: Frydenberg