(CNN) -- President Barack Obama declined Wednesday to join congressional voices calling for an investigation into why the FBI did not notify the White House and other political leaders sooner about the investigation into the affair that led to the resignation of the director of the CIA.
"I am withholding judgment with respect to how the entire process surrounding Gen. Petraeus came up," Obama told reporters at the White House. "We don't have all the information yet, but I want to say I have a lot of confidence generally in the FBI. So I'm going to wait and see."
Obama said he agreed with David Petraeus' decision to resign after acknowledging an affair, but praised his service to the country.
"I want to emphasize that, from my perspective, at least, he has provided this country an extraordinary service," Obama said.
He also said he had seen no evidence of any potentially damaging breach in national security stemming from the affair.
"I have no evidence at this point from what I've seen that classified information was disclosed that in any way would have had a negative impact on our national security," Obama said.
In Congress, lawmakers in both parties have complained about not having been notified sooner of the investigation that led to Petraeus' resignation or about potential security breaches.
Sen. Lindsey Graham, R-South Carolina, said he was increasingly concerned about the potential fallout from the Petraeus affair and any national security implications, including possible links to the September 11 attack on the U.S. Consulate in Benghazi, Libya, that killed four Americans.
Graham has called for a special select committee to investigate the attack. "The goal of this investigation is to have professional staff that hears everyone testify, the same set of senators who hears everyone testify, so we can segregate out the weird from the national security," Graham said. "And there is beginning to be a national security component to the human failing that I want to know about."
House Intelligence Committee Chairman Mike Rogers, R-Michigan, and ranking member Dutch Ruppersberger, D-Maryland, were expected to ask about the resignation and whether Congress should have been notified sooner during private briefings with CIA and FBI officials.
FBI Director Robert Mueller joined Deputy Director Sean Joyce and acting CIA Director Mike Morell in briefing the lawmakers.
Senate Intelligence Committee Chairwoman Dianne Feinstein, D-California, said Tuesday that she had "many questions about the nature of the FBI investigation, how it was instituted."
"And we'll be asking those questions," she said.
But the committee's ranking Republican member, Sen. Saxby Chambliss of Georgia, said Wednesday the group would not look into questions about the FBI investigation and how congressional leaders learned about it until after the bureau concludes its work.
Secretary of Defense Leon Panetta on Wednesday defended his request to withhold the nomination of Gen. John Allen to become NATO commander pending the investigation into his contacts with Jill Kelley, whose complaints about anonymous, harassing e-mails led to the discovery of Petraeus' affair with Paula Broadwell, his biographer.
Broadwell's government security clearance has been suspended pending the outcome of ongoing investigations, two U.S. officials with direct knowledge of the move told CNN Wednesday.
Defense officials announced Tuesday that the FBI had referred information to them indicating Allen may have exchanged potentially inappropriate e-mails with Kelley, who was a volunteer at MacDill Air Force Base.
Allen, who was once stationed at the base, has denied wrongdoing, a senior defense official said.
Her access to MacDill without an escort has been suspended, a Defense Department official said Wednesday.
Kelley had been given special access to the base because of her position as a booster and promoter of programs to help U.S. troops, the official said.
A U.S. official familiar with the e-mails Allen sent to Kelley described them as warranting the investigation.
"If they got out, John Allen would be very embarrassed by them," said the official, who added that there was no evidence of physical contact between the two.
The official said that the e-mails under investigation -- which have been called "potentially inappropriate" -- are from Allen.
A second U.S. official, who has had the e-mails described to him, characterized their content as "sexy," but could not say whether they "crossed the line."
They "are not the type of e-mails you would want your wife to read," the second official said.
But a senior official close to Allen told CNN on Tuesday that the e-mails contained nothing pointing to sex or anything of a romantic nature. Allen may have said, informally, "thanks sweetheart" in an e-mail, the official close to Allen said.
"Anyone who knows him knows his style; he has a habit of replying to every single e-mail (he is sent). Kelley would e-mail his business and personal accounts," the official said.
It will be up to the Defense Department's inspector general to decide if the e-mails' content represents conduct unbecoming an officer, said a third source, a senior U.S. official.
Allen has yet to be questioned by Defense Department inspector general staff, but that could be completed in days or weeks, a U.S. official with knowledge of the investigation said.
The anonymous e-mail to Allen that allegedly came from Broadwell was sent after May, perhaps in June, the official said.
The e-mail, which had also been sent to a number of other officers, bore the handle "kelleypatrol -- or something similar," the official said.
He described the e-mail as "a warning that Kelley was a seductress or something along those lines" and said it was vaguely threatening, but above all weird.
"Allen did not know it was (from) Broadwell," the official said.
The official also said it was unclear when Kelley went to the FBI or whether Allen's warning to her was the trigger, but that Allen saw nothing in the e-mail's wording to warrant referral by him to the FBI.
Kelley's version differs from one offered by the senior official close to Allen, who said it was Allen who received an anonymous e-mail about Kelley, and tipped her off that someone was threatening her.
One of the sources familiar with Kelley said she first mentioned the alleged harassment in a casual conversation with an FBI agent she knew socially. She did not seek him out specifically for action on the matter, but he was happy to help, the source said. The source added that Kelley did not know at first that the e-mails led to Petraeus.
A source familiar with Kelley's version of events said the anonymous e-mails traced to Broadwell began in June. It wasn't until two months later that the FBI told Kelley who had sent the e-mails, said the source, adding that Kelley does not know Broadwell.
Kelley, 37, and her husband have released a statement saying they have been friends with Petraeus and his family for more than five years and asked for privacy.
The move to delay Allen's nomination was "a prudent measure until we can determine what the facts are, and we will," Panetta told reporters Wednesday. "No one should leap to any conclusions."
He said Allen "certainly has my continued confidence to lead our forces," a view shared by Obama, White House spokesman Jay Carney said.
That Allen remains in command in Afghanistan suggests that there is no criminal issue, a U.S. official told CNN. But the official said the Defense Department's inspector general could still find evidence of criminal conduct.
Kelley has not responded publicly to the latest news.
Both Allen and Petraeus appear to know Kelley's sister, Natalie Khawam. The men wrote letters in support of the sister in a custody battle, court records show.
On Monday, FBI agents were at Broadwell's home in Charlotte, North Carolina, according to spokeswoman Shelley Lynch. She declined to say what the agents were doing there.
A source told CNN national security contributor Fran Townsend that Broadwell was acting as Petraeus' archivist and that the FBI went to the house to look for any documents she might have. It was not clear, the source said, whether any of the material was classified.
As a commissioned officer in the military reserves, Broadwell would have had "secret" or "top secret" security clearance, military officials said.
Access to information would have depended on what she needed to know to carry out an assigned task, said the officials, who would not go on the record about a matter involving an ongoing investigation.
Broadwell has said that she was working on a second book about Petraeus. Her LinkedIn profile lists her as "Archivist/Biographer for General (Retired) David Petreaus." Her first book, "All In," was about Petraeus' leadership.
Broadwell has spoken about how she had to deal with sensitive information in the course of researching the first book.
"I had to follow very clear lines of non-disclosure and sign non-disclosure agreements, like my colleagues. I felt like I was almost held to a higher level of accountability because I could lose my clearance," Broadwell said in a speech last year. "I think it was important to inform my writing, but I knew there was a clear line that I couldn't cross when I was writing it out."
Petraeus has said he never shared classified information with Broadwell, said retired military officer John Nagl, who cited conversations in recent days with Petraeus.
Broadwell, a military intelligence reservist, is assigned to West Point, the Army's military academy, according to her service record, which lists her assignment as "United States Military Academy Staff & Faculty." In August, she was promoted to lieutenant colonel.
The Internet has been abuzz over a speech by Broadwell in which she suggested the Libya attack last September 11 was targeting a secret prison at the Benghazi consulate annex, raising unverified concerns about possible security leaks.
"I don't know if a lot of you have heard this, but the CIA annex had actually taken a couple of Libyan militia members prisoner and they think that the attack on the consulate was an effort to get these prisoners back," she said last month in a speech at the University of Denver.
A senior intelligence official denied the assertion, saying no prisoners had ever been held at the annex. Broadwell did not provide a source for her information, and no evidence has emerged that it came from Petraeus.
Administration officials have said the Benghazi assault was a terrorist attack.
Petraeus has been scheduled to testify this week in private hearings on Capitol Hill about the Benghazi attack. Some Republicans have criticized the administration's response to the attack and have speculated that the timing of Petraeus' departure may have been linked to the congressional inquiry.
Feinstein said Tuesday that she hoped Petraeus would address the panel as early as Friday.