Even in college, 6-foot-3 Jerry Richardson had an imposing presence.

He was captain of Wofford’s football team, a star receiver who also kicked the ball and returned punts. He racked up honors on campus as easily as he broke tackles on the field.

“He was sort of bigger than life,” recalls fellow student Carroll Gray.

Richardson remained an imposing figure through a pro football championship, a thriving business career and ultimately as the man who brought the NFL to Charlotte. In many ways he was as striking as the 13-foot bronze likeness outside Bank of America Stadium.

But in less than 48 hours last weekend, the image that Richardson had so methodically built and carefully guarded crashed amid accusations of inappropriate behavior, secret settlements and a racial slur.

What began with revelations in Sports Illustrated ended with Richardson’s decision to cede control of the organization and, at season’s close, sell the team he’d built from a dream.

“There will be an awful lot of good things that he’s done,” says Richard Vinroot, mayor when the NFL came to Charlotte. “But these other things are certainly going to cloud that legacy. For most people the bad will be much easier to remember than the good.”

It’s been a stunning fall for a man who rose so high. The man obsessed with control found himself buffeted by stories and social media posts he was powerless to stop. The accusations scarred his personal reputation as surely as criminal charges against former players once marred the image of his team.

Richardson’s fall comes at a time when similar accusations have toppled giants of the arts and media and brought public recognition to the women who broke their long-held silences.

Some say he fell victim to changing times and workplace mores.

Felix Sabates, a luxury car dealer and longtime fixture of Charlotte’s professional sports scene, draws a distinction between sexual assault and harassment and “talking trash.”

“In my opinion, if this would have happened 30 years ago, people would have laughed about it and kept going,” he says. “Today you can’t do that.”

But Sabates adds: “There’s a big difference between sexual harassment and stupidity.”

Several female employees told Sports Illustrated that Richardson asked them if he could personally shave their legs. Some spoke of backrubs “that lingered too long or went too low.” Several recounted “the seatbelt maneuver,” where he would reach across female passengers to fasten their seatbelts and brush their chests in the process.

SI reported at least four former employees got “significant” financial settlements in exchange for their silence. Richardson apparently wanted to keep the payments a secret. He never told the NFL or his ownership partners about them.

Football to fast food

At 81, with silver hair and customary dark coat and the impeccable knot of his tie, Richardson has a commanding bearing. Even in his stadium box, his gaze is stern. Around the office he was called “Mister.”

He seldom betrays emotion unless his own character or integrity is challenged.

“You would think he was a retired general … from the demeanor and disciplined approach from everything from dress to posture to business,” says Max Muhleman, who helped Richardson build a stadium as architect of permanent seat licenses – the fees paid for the right to buy seats. “There’s a lot of Marine in him for a guy who wasn’t in the Marines.”

Richardson is a native of Spring Hope, a small town near Rocky Mount. After graduating from Wofford College in Spartanburg, he was drafted in the 13th round by the Baltimore Colts and was Colts Rookie of the Year in 1959. In the NFL championship, he caught a touchdown pass from the legendary Johnny Unitas.

A salary dispute led to a decision to go into business. When the Colts offered $250 less than he sought, he called on an old Wofford teammate who’d entered the fast-food business. Richardson invested half of his modest championship bonus and together they opened the first Hardee’s in Spartanburg. Business grew.

Richardson would later run what was once the nation’s fourth-largest food empire, with 5 million customers a day at nearly 500 Hardee’s, 1,300 Denny’s and 215 Quincy’s Steak Houses. Revenue was in the billions.

Richardson was not a hands-off owner. He would sometimes arrive at one of his Hardee’s franchises at 5 a.m. and pitch in making biscuits, pouring coffee or working the drive-through window.

Racial suit

One allegation in the Sports Illustrated story was that Richardson used a racial slur against an African-American scout, who left the team this year with what was reported to be a confidential settlement. It was a surprising suggestion for a man known for the courtly manners of a Southern gentleman and whose star quarterback, like many teammates, is African-American.

It wasn’t the first hint of racial trouble.

In Spartanburg, Richardson had a reputation as a racial moderate. In the 1960s, his Hardee’s restaurants peacefully integrated. A generation later, Richardson broke with most of the local business ranks when he supported the candidacy of James Talley, a longtime friend who became the city’s first black mayor.

But Richardson’s chain of restaurants ran into legal problems over alleged mistreatment of black customers.

In July 1993, as head of Flagstar – then the corporate name of his restaurants – he signed a “Fair Share” agreement with the national NAACP to spend about $1 billion to significantly increase the numbers of African-American managers, franchise owners and contractors.

At the same time, Richardson was still fighting allegations that hundreds of Denny’s restaurants in seven states systematically discriminated against black customers.

Among those joining a class action suit were six black Secret Service agents who in April 1993 were refused service at a Maryland Denny’s on their way to a nearby event for President Bill Clinton. The group was not fully served while their white counterparts at a nearby table were. Richardson, who argued that any instances of discrimination were isolated, nevertheless agreed to settle.

All this was occurring as Richardson was fighting to bring the NFL to Charlotte. In October 1993, three months after signing his agreement with the NAACP to racially diversify his company, Richardson was awarded a franchise.

“I think the owners were impressed by how Jerry acted immediately to resolve the Denny’s problem from a P.R. standpoint and a practical standpoint,” then-NFL Commissioner Pete Rozelle said, according to The New York Times.

In 1994, Flagstar agreed to pay $54 million to settle what the federal officials described at the time as the largest case ever under the public accommodations section of the Civil Rights Act.

For Washington, D.C, attorney John Relman, the Denny’s case ended with a question mark: Was Richardson’s decision to settle driven more by his desire to right a wrong or to acquire a football team?

Relman, who represented the Secret Service agents, believes one of the current allegations against Richardson may provide an answer.

“People just don’t start using racial slurs when they’re 80,” he says.

Rare display

Richardson’s quest for a team began in 1987 with a visit to Hugh McColl Jr. The two had a long relationship. McColl, chairman of what was then NCNB, loaned Richardson $25,000 when he opened a Charlotte Hardee’s in the early 1960s.

But the NFL was a different story.

“I thought he was crazy,” McColl later recalled. “We had to overcome a huge amount of skepticism about whether we could build a stadium or sell the tickets.”

Since winning the franchise, Richardson has worked incessantly to promote a wholesome image for his team. Before making Cam Newton the first overall pick in the 2011 NFL draft, Richardson shared his opinion with the quarterback on the subject of body art, a story he told to talk show host Charlie Rose and which struck some as being culturally insensitive.

“I said, ‘Do you have any tattoos?’ He said, ‘No sir, I don’t have any.’ I said, ‘Do you have any piercings?’ He said, ‘No sir, I don’t have any.’ And I said, ‘We want to keep it that way.’ ”

Despite his hands-on efforts, Richardson – and his team – have endured personal and professional bumps.

On the night of the 2009 Super Bowl, after years of cardiac problems, he underwent a heart transplant.

Later that year he fired his son Mark as president of the team and his son Jon as stadium president with little public explanation as to why.

In 2014, 15 years after releasing receiver Rae Carruth after he was charged with conspiring to murder his pregnant girlfriend, Richardson faced more off-the-field trouble. Star defensive end Greg Hardy was charged with domestic violence.

Richardson was criticized over his handling of the Hardy arrest and taken to task by an Observer editorial the same night he received the Echo Foundation’s “Echo Award Against Indifference.” Richardson broke down during his remarks.

“I stand firmly against domestic violence, plain and simple,” he said. “To those who would suggest we’ve been too slow to act, I ask that you ... not be too quick to judge. Over the course of 20 years, we have worked extremely hard to build an organization of integrity.”

It was a rare display of emotion for a man known for stoicism. More typical was the intensity and directness that marked most interactions.

Muhleman remembers the first time he met Richardson. He started making small talk when Richardson interrupted.

“ ‘I want to tell you how we operate,’ ” he recalls Richardson saying. “ ‘We always say what we’re going to do. We’re on time. We believe in teamwork’.”

Five years ago, when NFL owners and players were hashing out a new collective bargaining agreement, Richardson reportedly insulted Peyton Manning, then one of the league’s biggest stars and a player representative. “Do I need to help you read a revenue chart son?” Richardson said. “Do I need to help break that down for you because I don’t know if you know how to read that?”

So insulting and condescending was Richardson, one newspaper reported, that some owners later apologized to Manning.

Vinroot, the former mayor, calls Richardson “a no-nonsense guy.”

“He’s not a lot of fun and games, I wouldn’t call him charming either,” Vinroot says. “But he was a decent man.”

Former state Rep. Charles Jeter saw Richardson’s impact when he went to Raleigh in 2013 to lobby for state money for stadium improvements. The Panthers would get $87.5 million from the city but none from the state.

“He was almost treated as if he were a conquering hero,” Jeter says “Jerry didn’t get what he wanted in large part because they weren’t giving money to a billionaire.”

Richardson has been known for his privacy. He rarely talks with reporters or mingles with fans. One exception is at Art’s Barbecue and Deli on East Morehead Street, where he goes for the grilled cheese and baked beans.

“He’s always got a smile on,” says owner Danny Katopodis. “He has no problem speaking to anybody. He loves fans. He loves just regular people.”

A former Art’s employee says whenever Richardson brought another NFL owner or network executive to the restaurant, he always introduced his famous guests to the staff. He gave Art’s staff tickets and field passes to last Sunday’s game.

Katopodis doesn’t know what to think about the allegations against Richardson.

“I’m not passing judgment on anybody,” he says. “He’s a fantastic man. He’s always been very generous to us here.”

Reporter Cassie Cope and Researcher Maria David contributed.