BOSTON -- With the newest vacancy on the U.S. Supreme Court raising questions about the future of legal precedents like the right to an abortion, Associate Justice Stephen Breyer emphasized repeatedly Thursday that the strength of the country's legal system is rooted in stability and precedents should be overturned only in the rarest of cases.

Breyer, who was nominated to the nation's highest court by President Bill Clinton in 1994, was not talking specifically about Roe v. Wade or any other specific past decision of the Supreme Court. But he warned that judges ought to be careful when tinkering with precedent, even if they don't agree with the original decision.

"Just be sure it's very rare," he said.

The 82-year-old San Francisco native took part Thursday in an hour-long conversation about the court hosted by the Edward M. Kennedy Institute for the United States Senate, and moderated by Ari Melber, an attorney and host of The Beat with Ari Melber on MSNBC.

Breyer is a graduate of and former professor at Harvard Law School, and worked for the U.S. Senate Judiciary Committee when it was chaired by the late Massachusetts Sen. Edward Kennedy.

Melber brought up Breyer's dissent in a 5-4 decision in 2019 that overruled a past precedent that had allowed states to be sued in courts of other states. In his opinion, Breyer wrote, "Today's decision can only cause one to wonder which cases the court will overrule next."

Asked what people should take away from that dissent about his view on the law, Breyer said, "That it has to stay pretty stable."

Immediately after the death of Justice Ruth Bader Ginsburg last month, President Donald Trump nominated Notre Dame law professor Amy Coney Barrett to succeed Ginsburg, who was regarded by liberals and Democrats as a champion for civil and women's rights, including the right to an abortion.

U.S. Sen. Elizabeth Warren as recently as Thursday in an op/ed published in The Cut made the case that Barrett's nomination is a clear attempt by Trump and conservative Republicans to roll back Roe v. Wade, and advocates in Massachusetts have urged an immediate strengthening of state abortion access laws in the wake of Barrett's selection.

Barrett has been critical in the past of the abortion law decision, but Vice President Mike Pence said during a debate Wednesday night he would not presume to know how she would rule if a challenge to Roe reached the Supreme Court.

Breyer said he has heard cases when he disagreed with the original ruling and was tempted to overturn it, but resisted. He brought up the case of Brown v. Board of Education partially overturning the doctrine of "separate but equal" established in Plessy v. Ferguson as one of those "rare" cases where overruling precedent was warranted.

"Were they right to overrule that? Of course, they were right. It was bringing in a bit of justice. It was bringing in some justice into an area where there was no justice and that's what the court is about," Breyer said. "But that is unusual. Don't generalize from the fact that you can do it once you can do it all the time. Rare."

Breyer steered clear during the conversation of politics, including questions posed by people watching the Zoom talk about the effectiveness of the Senate confirmation process and whether Trump was right to release a list of judges and lawyers from which he would nominate Supreme Court justices, or if Joe Biden should do the same.

Ginsburg's last public statement shared by her family asked that her replacement not be named until after the election, which has not been honored by Trump or Senate Republicans. Asked whether who their successor might be is something justices think about as they near retirement," Breyer said, "Sure, people think about it."

"Of course, from time to time you think about it. It's part of the aging process and it's inevitable. We stay out of politics and really sometimes it's really hard to just stay out, but the more the political fray is hot and intense the more we stay out of it," Breyer said.

Breyer also said that too often the media puts justices into ideological boxes to fit a narrative when, in reality, 40 percent of decisions made by the Supreme Court are unanimous and even in the more divided cases the voting blocs don't always break down the way people think they might.

"If you want to be a politician go into politics. Judging is a different job," Breyer said.