Don Johnson on Gambling with U.S. Presidents and Smoking Weed in the White House

Don Johnson has had interesting friendships with several U.S. presidents, and he has done some interesting things in their presence!

During a Wednesday, June 4, appearance on Jimmy Kimmel Live, the 75-year-old Miami Vice alum shared that he has visited the White House under every president since the late Jimmy Carter was in office.

The conversation started when host Jimmy Kimmel presented Johnson with a photo of himself with Carter, Chuck Leavell and Dickey Betts from 1975 at an Allman Brothers concert.

“I don’t remember that photo, but we were all stoned,” Johnson said about the picture, which appeared to show Betts smoking.

The actor was quick to clarify that he was not speaking about Carter, saying, “Well I don’t know about him. I don’t want to cast any aspersions on the former president. God rest his soul.”

However, Johnson clearly had a close bond with Carter. After working with the Allman Brothers on a concert during his 1976 presidential campaign, he said that they were invited to the White House.

“Then the fun began,” he said, prompting Kimmel to point out that “nobody has fun” at the White House.

He then walked that back, saying, “I have heard Willie Nelson had fun there,” referencing the fact that the country music star smoked marijuana with Carter’s son on the roof of the building in the ’70s.

“Is that the kind of fun you had there?” he asked, clarifying, “Did you smoke at the White House?”

Appearing slightly flustered, Johnson replied, “I don’t know how to answer this. Well, yes … I guess I did.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

Another president that the star had a close bond with was the late George H.W. Bush. Johnson recalled representing the United States at the 1992 Winter Olympics in Albertville with the former president, whom he described as “fantastic.”

“He was a great guy,” Johnson gushed.

Kimmel questioned if Bush was a stoner, which Johnson denied. However, he said that the president had another playful vice — gambling.

“He wanted to gamble when we played golf out at Camp David. And so I took his money,” he recalled to laughter and applause from the audience. “I’ll tell you something else: He’s a trash talker, and he ran into a buzzsaw because so am I.”

While he did win money from the world leader, Johnson clarified that “it wasn’t that much.”

“Eighteen or 20 bucks,” he told Kimmel, adding, “But that’s the best 18 or 20 bucks you’ll ever get.”

The PEOPLE App is now available in the Apple App Store! Download it now for the most binge-worthy celeb content, exclusive video clips, astrology updates and more!

In his autobiography, Nelson wrote that he smoked with an employee at the White House. However, in 2020, Carter clarified that it was actually with his son James Earl “Chip” Carter III.

“When Willie Nelson wrote his autobiography, he confessed that he smoked pot in the White House one night when he was spending the night with me,” the former president said in the documentary Jimmy Carter: Rock & Roll President. “And he says that his companion that shared the pot with him was one of the servants in the White House. That is not exactly true — it actually was one of my sons, which he didn’t want to categorize as a pot-smoker like him.”