‘I watched Geldof at his photo-opportunity. Another tall man in brown: a slender velvet coat and corduroy cap made him somewhere between folk singer and a City gent, or at least how those types looked 40 years ago.’ Photograph: Alicia Canter for the Guardian

The Richmond Park byelection held its last hustings on Tuesday, and afterwards the Labour candidate, Christian Wolmar, and some of the audience went across the street to the pub. A most interesting-looking figure followed us: a tall man dressed from head to foot in shades of brown – a brown felt hat which he didn’t remove, brown tweeds, brown shoes – as though he’d stepped out of a Patrick Hamilton novel or a prewar play: “Ma’am, there’s a police inspector waiting in the hall.” In his 70s, I guessed, with a soft and amiable face, and on his own. He looked like a man who knew how to do this kind of thing – that is, to go into a pub, buy a beer, ask the barman if he’s having one himself, and then settle into a conversation with an equally respectable stranger. In some ways, Richmond feels more like a county town than a London suburb, and this may be one of those ways.

I asked him how he’d vote. Maybe for the Lib Dem, he said, but not for Labour and not for Zac. “I voted for him last time. What I want to know is, what promise did he make to Sheherazade?” Sheherazade was Zac’s first wife: jeweller, environmental campaigner, author of A Greener Christmas, daughter of the financier John Bentley and the actor Viviane Ventura. She and Zac married in 1999, had three children and divorced in 2010.

I didn’t understand the man in brown’s question, but it turned out that he meant the marriage vows, the implication being that a man who broke his marriage vows couldn’t be trusted with other vows, an astonishingly exacting standard given that we had just attended a meeting that had happened only because Goldsmith had kept his vow – that is, to resign as an MP if the government approved a third runway for Heathrow airport, unusually upright behaviour that has turned out to have bigger and, for him, sadder consequences than he expected. “This election is happening because I kept my word,” Goldsmith told Tuesday’s hustings. “If you think I’ve been a good MP over the last six years, give me your vote.” But this hadn’t washed with the man in brown, for peculiar reasons that had at their roots a general one: that all politicians are suspect and perhaps never more so when they come across as a goody two-shoes.

The obvious explanation of Goldsmith’s defeat lies elsewhere. He was a pro-Brexit candidate in an anti-Brexit constituency with a remain vote of 72%. His rivals shared his antagonism to Heathrow expansion – what could he say that they couldn’t? The Lib Dems poured resources into the fight that he, as an independent with covert support from the Tories, couldn’t match. By deciding not to run, the Greens and the Women’s Equality party increased the Lib Dem share of the vote, which at 6% for the Greens in 2015 wasn’t risible; bigger, after all, than the 4.5% margin of Sarah Olney’s victory. All these things are true, but there was also something else – a sense of disenchantment with the Goldsmith persona.

He was an hour late arriving at the hustings on Tuesday. His trousers had been ripped in an encounter with his car and he’d needed, he said, to go home to change his clothes and “present” himself. People cheered when he turned up, but the cheers were almost as loud when Wolmar went off in a little riff about Goldsmith’s niceness. “Nice guys don’t vote to cut people’s housing benefit – how many spare bedrooms do you have by the way, Zac? Nor do nice guys have a campaign like yours against Sadiq [Khan, the London mayor].”

It felt unusually personal and aggressive in a big church hall that was filled with an audience who looked to have an average age of 60, for whom the phrase “well-modulated tones” might well have been invented, and yet there was little obvious disapproval. During audience questions, a Muslim woman and her daughter invited Goldsmith to apologise for suggesting during the London mayoral contest that Khan had “legitimised” extremism. He wouldn’t – he said he absolutely rejected the charge of Islamophobia – and again the audience seemed more against him than for him (though a man behind me whispered rather savagely, “Integrate, for Christ’s sake!”). He gave off a sense of entitlement and rectitude, which has surely contributed to his political failure.

The next morning I followed his Labour opponent on a canvassing round of Mortlake. Transport journalist, railway enthusiast, cyclist, environmentalist, former London mayoral prospect, shouter: for all these reasons, Wolmar seemed a far stronger candidate than the Lib Dems’ Olney, an accountant by profession whose interest in politics is all of 18 months old and who measures every word.

I’ve known Wolmar for 20-odd years, originally as a newspaper colleague, and what I find hard to understand is why, at 67, he wants a new career in politics. “Because I love it,” he said. “I enjoy it all – knocking on doors, arguing the case, speaking at meetings, maybe changing a few minds.” In the old days, he said, he’d be sick before every television interview, but now the nerves had gone.

This is a modal window.

Byelection was not a political calculation, says Zac Goldsmith

Nerveless, we patrolled the cold but sunny terraced streets, some with railway level-crossing gates at the end. Like the man in brown, they gave the misleading impression that London and modernity were far away. I asked how he would feel as Richmond’s Ralph Nader, supposing he took away enough votes from Olney to let Goldsmith in, and he said that it slightly worried him, but the much-vaunted notion of a progressive alliance was a trickier proposition than its advocates pretended. How happily, for instance, would Labour activists work for a Lib Dem candidate, and vice versa?

As it turned out, his worry was needless. A canvasser spoke of Labour’s vote being “squeezed”. It meant that when he went to an address marked “Labour voter” he sometimes found a Lib Dem poster in the window. Tactical voting: a progressive alliance brought about from beneath. The next day only 1,515 people from an electorate of 77,243 and a turnout of 41,367 voted Labour – fewer voters, in fact, than the constituency Labour party has members (now said to number 1,600). At 3.7% the party returned its lowest share of the vote since it began contesting the seat in 1924. My friend Wolmar lost his deposit.

Nothing could be more electorally ignominious, but it would be foolish to read too much significance into the result. Did the electorate vote on grounds of hard Brexit v soft Brexit? Do they believe, in the words of Bob Geldof, that “the fightback starts here”. I watched Geldof at his photo-opportunity in the forecourt of Richmond station. Another tall man in brown: a slender velvet coat and corduroy cap made him somewhere between folk singer and a City gent, or at least how those types looked 40 years ago. He was fluent, amusing and friendly, and apparently sincere in his commitment to Europe. Passers-by stopped to be photographed with him. A homeless man removed his leg plaster to show him a suppurating wound. His attempt to start a chant, “Zac is crap, Zac is crap”, was a failure – the young Lib Dems who gathered around him with their placards weren’t brazen or vulgar enough. But his progress down Richmond High Street was a triumph that few politicians could have matched.

Of course Geldof didn’t win it for the Lib Dems either. If I had to put money on the biggest single factor, I’d bet on a piece of Lib Dem propaganda that had the same size, shape and typography as the Daily Mail. It called itself the Richmond and Kingston Gazette. “Sarah surges ahead”, was its headline on the day before polling, which was a piece of cunningly written wishful thinking that made itself true.