A Conservative minister has become the first government frontbencher to hint that he backs efforts to remove the Speaker of the House of Commons, John Bercow, over remarks he made about the visit of Donald Trump to parliament.

Greg Hands, the minister for international trade, highlighted a tweet criticising Bercow posted by Georgia Toffolo, a member of the Made in Chelsea reality TV show cast.

Toffolo, who frequently tweets about politics, is one of Hands’ constituents in Fulham and Chelsea. She regularly tweets about politics. “John Bercow, ENOUGH IS ENOUGH. The Commons needs a new IMPARTIAL speaker,” she tweeted.

Hands highlighted her tweet with the caption: “Am looking forward to welcoming my constituent & Made in Chelsea’s @ToffTalks to the @HouseofCommons in the coming weeks.” Toffolo replied saying the pair had “lots to discuss.”

Bercow drew the ire of Tory backbenchers last week when he publicly vetoed any invitation to Trump to address both houses of parliament in Westminster Hall, saying the Commons must oppose racism and sexism.

James Duddridge, the MP who has tabled ano-confidence motion in the Speaker, said he was certain Hands was backing the movement to oust Bercow. “Yes is the short answer,” he told the Guardian. “Ministers have been quite guarded in public and that is understandable because they don’t want to be seen to be backing something if they aren’t sure it won’t fail. There is safety in numbers.

“I think we will see more break ranks in the coming days. Greg Hands is an excellent minister who speaks his own mind. The government has issued no official instructions on how to vote and insists it’s a matter for the House. So hats off to Greg Hands.”

This is a modal window.

Commons Speaker ‘strongly opposed’ to Westminster Hall address by President Trump

Duddridge said he had support for his no-confidence motion from across the Commons, including Labour MPs who thought Bercow had overstepped the mark.

His fellow Conservative backbencher Karl McCartney sent out an email to all MPs on Tuesday. “I am contacting you as I am aware a number of colleagues do not normally sign EDMs [early day motions] and so are unaware or unsure of the easy way to support the motion of no confidence in Speaker Bercow as put down by our colleague James Duddridge MP,” he wrote.

“The recent outburst by the Speaker, unilaterally barring the president of the United States – our closest ally – from addressing members of parliament, was a breach of parliamentary protocol that parallels no others that neither I, nor more senior members can recall,” the email went on.

“It was also indicative of a growing trend in politics of people in powerful positions showing contempt for democracy and an electorate who have made their views crystal clear.”

McCartney said MPs should support his motion “whether you agree with Speaker Bercow’s criticisms of President Trump or not” saying it was the principle at stake that Bercow was “not politically impartial”.

Thus far only Dudderidge’s name appears under the official motion to remove Bercow and any MPs who have signed it over the parliamentary recess this week will be revealed on Monday.

Bercow became Speaker in 2009, and said he intended to serve nine years in the post, meaning he would step down in 2018 before the next general election. Several other MPs have been reported to be interested in the role, including the deputy speaker Lindsay Hoyle, the Labour MP Chris Bryant and the Conservative Jacob Rees-Mogg.

Labour MPs have generally come out in support of Bercow including Harriet Harman, the party’s former acting leader, saying it was clear there should be a Speaker who was “strong, relevant, able to challenge the government and high-profile – which government hates”.

Owen Smith, the former Labour leadership candidate, said: “John Bercow showed great leadership … standing up against racism and expressing his objection to President Trump addressing parliament.”