Witnesses told of a gruesome scene in which the man was hit by a car, then hacked with cleavers and his body dumped in the middle of the road in Woolwich, southeast London.
The two suspects in the killing were injured in a confrontation with police and have been taken to two hospitals, where they are being treated.
CNN affiliate ITN aired a video showing a man with bloody hands and holding a meat cleaver, who says, "We swear by almighty Allah we will never stop fighting you."
The man, who is black and appears to have a London accent, carries on: "The only reasons we killed this man this is because Muslims are dying daily. This British soldier is an eye for an eye a tooth for tooth.
"We apologize that women had to see this today but in our lands our women have to see the same. You people will never be safe. Remove your government. They don't care about you."
The victim is believed to be a serving soldier who was based at a nearby barracks, Nick Raynsford, a member of Parliament, told CNN.
The soldier had apparently been on duty in central London and was returning to the Royal Artillery Barracks in Woolwich when he was attacked, Raynsford said.
Speaking in Paris, Prime Minister David Cameron said there were "strong indications" that the man's killing was a terrorist incident.
He said he would be returning early from the official trip to handle the situation and will be back in London Wednesday night.
"It is the most appalling crime," he said, speaking alongside French President Francois Hollande, who pledged solidarity with Britain in the face of terrorist threats.
Cameron declined to confirm whether the man killed was a serving soldier, while Britain's defense ministry said it was investigating to see whether that's the case. But the prime minister vowed the United Kingdom would stand firm in the face of threats to its security and "will not buckle."
UK Home Secretary Theresa May led a meeting of the country's civil emergency committee, known as COBRA, on Wednesday evening, and Cameron will chair another such meeting on Thursday.
Afterward, authorities increased security around Woolwich and all military barracks in London, a British government source said.
British Defense Secretary Philip Hammond said the killing was a "very shocking incident" and that the UK takes the safety of its troops "very seriously," as he headed into the COBRA meeting.
London Mayor Boris Johnson tweeted: "This afternoon's attack in Woolwich is a sickening deluded and unforgivable act of violence. My thoughts are with the victim and his family."
That sentiment was echoed by many others -- such as May, the Home Secretary, who described it as a "a sickening and barbaric attack," and Labour Party leader Ed Millband, who predicted the "whole country will be horrified" by what he called the "appalling events."
Commander Simon Letchworth told reporters in Woolwich that local officers had responded when an assault was reported at 2:20 p.m. Wednesday and a firearms unit was called to the scene. British police do not usually carry firearms.
"Two men, who we believe from early reports to have been carrying weapons, were shot by police," he said. "They have both been taken to separate London hospitals and are receiving treatment for their injuries."
'They were hacking at this poor guy'
Eyewitnesses have given gruesome details of the killing.
One, Michael Atlee told CNN he and friends heard gunshots and when they turned the corner saw a pool of blood on the sidewalk and a trail of blood leading into the road.
The man killed appeared to be wearing a T-shirt for Help for Heroes, a charity that helps military veterans, he said.
Another witness, who gave his name only as James, told London's LBC 97.3 radio station that he saw two men standing by the victim on the floor.
At first he thought they were trying to help the man but then saw two meat cleavers, like a butcher would have.
"They were hacking at this poor guy, literally," he told the radio station, as if they were trying to remove his organs.
"These two guys were crazed. They were just not there. They were just animals."
The men appeared to want to be filmed, he said, saying that one of the attackers went over to a bus and asked people to take photos of him as if he wanted to be on TV.
He said the two men had rushed at the police when they arrived, at which point shots were fired and both men went down.
Lauren Collins told CNN: "I still am quite shaken at what I've seen -- I've seen a victim of an awful attack, and I've seen a body of a young man."
Another witness, Woolwich Theatre owner Adrian Green, said the incident was "shocking" but did not reflect the true nature of the area.
Raynsford said the military based at the centuries-old barracks had a close relationship with the local community. Woolwich is a mixed, multicultural area, he added.
By Laura Smith-Spark
CNN's Mariano Castillo, Joshua Levs, Atika Shubert, Stephanie Halasz, Kim Chakanetsa and Pierre Meilhan contributed to this report.
The-CNN-Wire™ & © 2013 Cable News Network, Inc., a Time Warner Company. All rights reserved.