Housing and Urban Development Secretary Ben Carson has reportedly upset several of his colleagues after he expressed concern about "big, hairy men" trying to infiltrate women's homeless shelters.

Sources spoke to the Washington Post about his "dismissive" remarks, which he has made frequently during internal HUD meetings.

"His overall tone is dismissive and joking about these people," the unidentified official said. "It's disrespectful of the people we are trying to serve."

Back in May, the agency introduced a proposal that would allow federally funded shelters to deny people admission on religious grounds or force transgender women to share bathrooms and sleeping quarters with men.

"The sentiment conveyed was these were not women, and they should not be housed in single-sex shelters -- like we shouldn't force people to accept transgender people in this context because it makes other people uncomfortable," a staffer told The Post.

Carson denies that he had made any derogatory remarks towards the transgender community:

"The Secretary does not use derogatory language to refer to transgendered individuals. Any reporting to the contrary is false," his statement responding to the claims reads.

"It's gravely insulting to have the specter of violence from cisgender men used to restrict the rights of transgender people who are ordinarily the victims of that violence," said Gillian Branstetter, spokeswoman for the National Center for Transgender Equality.

"It's a mythical notion that policies that are inclusive of transgender people somehow pose a threat," she said. "It's frankly despicable that such a harmful notion would be used by someone charged with facilitating programs meant to help people in need, many of whom are transgender."