America's ambassador to the UN, Nikki Haley, said the meeting showed Iran that "the world will be watching" its actions.
But Russia's ambassador, Vassily Nebenzia, said the US was "abusing the platform of the Security Council".
Envoys from several other countries including China expressed reservations about whether the council was the right place to debate the issue.
While Russia believes the UN Security Council has no business getting involved, British ambassador Matthew Rycroft said it was "right and proper - indeed, our responsibility to assess whether a situation like this could become a threat to international peace and security".
At least 21 people have been killed in Iran after anti-government protests broke out in late December. Economic factors, such as increasing inflation and high unemployment among the young, are also causing concern.
Up to 42,000 people have taken part in the protests. But tens of thousands have also reportedly taken part in pro-government rallies in recent days.
Donald Trump has tweeted his support for the anti-government protesters, warning Iran that the "world is watching".
In comments that echoed the President's, Ms Haley said the world should "applaud" the protesters' "courage", adding: "The Iranian regime is now on notice: The world will be watching what you do."
The Netherlands hoped the meeting would "work as a preventive measure to avoid further escalation of violence", Dutch Ambassador Karel van Oosterom said.
:: What does Iran's unrest mean for oil prices?
Under its charter, the UN empowers the council to "investigate any dispute, or any situation which might lead to international friction".
But France's ambassador, Francois Delattre, urged caution, saying the council should act "with all the vigilance required, guarding against any exploitation of this crisis, which would only reinforce the extremes".
Iran's foreign minister, Javad Zarif, described the decision to call the meeting as "another FP (foreign policy) blunder for the Trump administration", adding that the council had "rebuffed the US' naked attempt to hijack its mandate".
Iran is not a member of the Security Council but its ambassador, Gholamali Khoshroo, was invited to participate.
Mr Khoshroo claimed the protests had received "direct encouragement by foreign forces including by the president of the United States", something the Trump administration has denied.