Japanese authorities told the Olympic and Paralympic committees it was "highly unlikely" that entry to the country could be guaranteed.
Organisers said the move now gives "clarity" to ticket holders and helps ensure "a safe and secure Games for all participants and the Japanese public".
The Games are due to begin on 23 July.
The Paralympics follow the Olympics a month later, from 24 August.
Organisers said the "challenging" Covid-19 situation in Japan and many other countries, global travel restrictions and emergence of variant strains of the virus had led to the decision and that ticket holders would be refunded.
The Olympics were postponed by a year in March last year because of the growing spread of coronavirus across the world.
'Difficult decisions need to be made'
It is the first time in the event's history it has been postponed, with more than 11,000 athletes from about 200 countries scheduled to take part in 2020.
Thomas Bach, president of the International Olympic Committee, said the move is a "great sacrifice for everybody".
"We share the disappointment of all enthusiastic Olympic fans from around the world, and of course the families and friends of the athletes, who were planning to come to the Games," he said. "For this I am truly sorry.
"Every decision has to respect the principle of safety first. I know that our Japanese partners and friends did not reach this conclusion lightly.
"We stand shoulder-to-shoulder at the side of our Japanese partners and friends, without any kind of reservation, to make the Olympic and Paralympic Games Tokyo 2020 a great success."
International Paralympic Committee president Andrew Parsons said "difficult decisions" had to be made with safety the "top priority".
"It goes without saying that in an ideal world we would prefer to have international spectators at the Games," he said.
"But at the moment we must acknowledge that due to the global pandemic we are not living in an ideal world."
'Very sad news' - reaction
The British Olympic Association said that while it is "a very disappointing situation", it highlights the "determination to stage" a safe event during the pandemic.
"This is very sad news, not only for British fans but particularly for the family and friends of athletes," a BOA statement continued.
The exclusion of international fans comes as another major financial blow to the Tokyo Games.
Costs for the Games have increased by $2.8bn (£2.1bn) because of measures needed to prevent the spread of Covid-19 but organisers have consistently ruled out a delay.
Earlier this year, Japanese prime minister Yoshihide Suga said the Games would be "safe and secure" and could serve as a "symbol of global solidarity".
However, a poll at that time by national broadcaster NHK showed the majority of the Japanese general public oppose holding the Games in 2021, favouring a further delay or outright cancellation of the event.
Japan has also encountered problems unrelated to the pandemic, with the head of the Tokyo Olympics organising committee Yoshiro Mori resigning after he was criticised for making "inappropriate" remarks about women.
The Tokyo Games' creative chief then also resigned after suggesting a female comedian could appear as an "Olympig" at the opening ceremony.
'A dream for nearly 40 years' - fans with tickets speak to BBC Sport
Alex Blaney said it is "a sad day", and he is "devastated" not to be able to help his 74-year-old father fulfil a long-held ambition.
"When I was seven in 1984, we were watching Daley Thompson's high jump and my dad turned to me and said he would 'love to go to an overseas Olympics'," Blaney said.
"I've remembered that dream of his for nearly 40 years. We've got 30 tickets for all of us to make the dream come true. This was probably the last chance we had, as he now has dementia. Maybe we might make Paris
, but it won't be the same."
Kevin Grant said attending the Paralympics was something for his autistic son - a wheelchair basketballer who is "fascinated with Japan" - to "look forward to and plan for" after a difficult year.
"He will be devastated, I am sure, but as he struggles to express feelings it will be hard to know the true impact," Grant said.
Gill Ludlow said her husband and son - who got the tickets as a birthday gift - had been set to go on "a trip of a lifetime", but they "totally understand" the decision.
Philip Davis organised his tickets in 2019, "before the word Covid was even known", and said the announcement "isn't a surprise", although he has "tried to stay positive" and hopeful.
"I personally have been a bit on the fence about it but would've been happy to go nonetheless," he said. "I'll definitely still watch from home though."
Hannah Chambers, who was set to attend with her boyfriend and hoping to watch a friend who is in contention to compete in Tokyo, said: "It's disappointing but understandable given that the world is in turmoil."
John Riggs said "planning starts for Paris" after the announcement, adding: "We appreciate that there are more important things going on in the world at the moment, as there probably always are, but it still leaves you disappointed."
For Dej Mahoney it will be the first Games since 1992 that he has missed.
After watching Jamaican sprinter Usain Bolt win eight gold medals over the years, he said he will "revel in those memories and not feel too hard done by".
