While Papa John’s continues to distance itself from founder John Schnatter, even deciding to evict him from company headquarters in Louisville, Kentucky, the beleaguered figure is fighting back. In a letter to directors, dated Saturday and reviewed by The Wall Street Journal, Schnatter expressed regret over his decision to step down as executive chairman on July 11, saying, “The board asked me to step down as chairman without apparently doing any investigation. I agreed, though today I believe it was a mistake to do so. I will not allow either my good name or the good name of the company I founded and love to be unfairly tainted.”

He added: “I have checked with corporate governance experts who tell me that this was not a proper action by the board.”

READ MORE: Sponsors bail as fallout continues for Papa John’s.

Schnatter, who remains a board member and, as of March owned 29 percent of Papa John’s shares, which measures to about $500 million, said he’s not racist. Schnatter’s resignation stemmed from details of a May conference call with marketing agency Laundry Service. The session was reportedly arranged between executives and the agency to prevent future public-relations snafus. In the letter, Schnatter agreed the call was intended to prepare him for difficult questions he might face from reporters. The agency asked him if he was racist.

“I then said something on the order of, Colonel Sanders used the word ‘N,’ [I actually used the word], that I would never use that word, and Papa John’s doesn’t use that word,” Schnatter wrote in the letter. He also said Laundry Service suggested the company retain rapper Kanye West to appear with Schnatter in marketing. He said he resisted since West used the “N-word” in some of his music.

“I am confident that an examination of the facts will bear out what I have written in this letter and show that once again our company has demonstrated that it does not know how to handle a crisis based on misinformation,” he said in the letter.

“I will not allow either my good name or the good name of the company I founded and love to be unfairly tainted.” — John Schnatter, Papa John’s founder.

CNBC is also reporting that Schnatter said Laundry Service was fired by Papa John’s the day after the call, but still owed the agency about $1.3 million. He wrote Laundry Service requested $6 million “because they claimed some of their people had been offended by what I had said.” He added that an attorney threatened a smear campaign if the pizza chain didn’t comply. Papa John’s, Schnatter said, ultimately paid the agency $2.5 million.

According to The Wall Street Journal, Schnatter visited corporate headquarters Tuesday to “talk informally to employees and received hugs from people as he walked the halls.”

The board can’t remove Schnatter as director, leaving the decision up to shareholders at the next annual meeting in May to re-elect him or not. “Those individuals were acting on rumor and innuendo, without any investigation—let alone a third-party investigation of the facts,” Schnatter said in the letter.

People familiar with his thinking, as reported in the article, said Schnatter regrets his decision not just to resign as chairman, but also his agreeing to exit as CEO in December in the wake of his NFL comments over anthem protests hurting Papa John’s sales. He has hired Los Angeles trial attorney Patricia Glaser to aid in his efforts.

Glaser penned a separate letter July 15 saying Papa John’s “board had no authority to remove Mr. Schnatter as a director” and asked for “an independent investigation and fully inform itself as to what actually occurred.”

She said failure to do so would render individual board members liable for all resulting harm to Schnatter and Papa John’s.

Olivia Kirtley, Papa John’s lead independent director, sent Schnatter a letter last week saying the board wanted to discuss his full resignation at the meeting that took place Sunday, The Wall Street Journal said.

After the meeting, Papa John’s sent a release announcing the creation of a special committee of the board of directors, consisting of all of the independent directors, to “evaluate and take action with respect to all of the company’s relationships and arrangements” with Schnatter. The committee approved and directed the company to terminate Schnatter’s founder agreement, which defined his role with Papa John’s “as advertising and brand spokesperson for the company.” It also approved and directed the company to terminate a sublease agreement granting Schnatter the right to use certain office space at the company’s corporate headquarters in Louisville, Kentucky, and “specifically requested” that Schnatter cease all media appearances, and not make any further statements to the media regarding the company, its business or employees.

Additionally, on Tuesday, Papa John’s said it appointed international law firm Akin Gump Strauss Hauer & Feld LLP “to oversee an audit and investigation of the company. As previously announced, the audit and investigation will examine all of the existing processes, policies and systems related to diversity and inclusion, supplier and vendor engagement and Papa John’s culture.”

“We’re focused on creating the right future for Papa John’s,” Kirtley said in a statement. “The special committee of the board is committed to this thorough audit and investigation and to taking deliberate actions to rebuild trust at Papa John’s and to ensure that this company is driven by the values of diversity, equity, inclusion and respect.”