Richmond residents fear the town's Pizza Express will become its biggest boozer as the council said it would be allowed to become a pub.

The ground floor of Lion House, which has homes above, could be turned into a pub or bar without applying for planning permission again.

Richmond Council's planning committee heard fears from locals on Thursday night that drunk punters will be noisy and disruptive if the restaurant on Red Lion Street becomes a pub. The application from Blue Coast Capital means the use of the venue has changed from a restaurant to a restaurant, pub, wine bar or other such venue. The venue would have to shut at 10pm on Sundays to Wednesdays and 11pm on Thursdays to Saturdays.

Rachel Robinson, agent for the applicant, said the change would reduce the risk of the venue being left vacant if Pizza Express moved out. She said: "Particularly given the current challenges that town centres are facing, the ability to allow a flexible use will improve the chances of reletting the ground floor of Lion House should this be required in the future – to either a future restaurant user or alternatively a drinking establishment type user." 

But Peter Willan, representing the Friends of Richmond Green, said the application "really worries" him. He said locals' main concern is the "family restaurant" being turned into a drinking venue with an "estimated capacity" of 700 or 800 drinkers. He said: "This would be by far the largest drinking establishment in Richmond and with all the risks that follow, such as public nuisance – particularly on dispersal." 

He said Richmond has "probably the most concentrated mix of pubs and bars, residents and open spaces anywhere in London" meaning "noise and disturbance continues to be a major issue". 

Asked when he thinks 800 drinkers would be in the venue at the same time, he said: "There is in my mind a big difference between what the planning permission allows and what the owner or the operator might actually do and they could be two very different things, but this applicant has said… that they want to do this so when they can perhaps move on they will have this in place for their premises." He added: "It may be that they change to a bar or it might be quite peaceful, but I think… the risk that we face is substantial." 

Louise Fluker, representing the Richmond Society, added: "Hardly a week goes by without an office, or permission to convert offices into residential unit[s], being made, and this is in the town centre – so in George Street in particular. So the number of people who may be affected by activities arising from a change of use of restaurant to a pub or drinking establishment may be much greater than I think people realise." 

Ms Robinson said: "A noise assessment has been submitted which concludes that even when based on a worst case scenario, the application proposal is not expected to have any adverse impact." The application was approved by seven councillors to one, with one abstention.