Senior Liberal says turmoil in Canberra could hurt cause in Victoria