The chancellor says the government has already given money to councils for free school meals in school holidays.
Rishi Sunak adds that the narrative that the government voted against free school meals is not fair, as they provide them in term time.
He also says they are “absolutely committed” to making sure vulnerable children do not go hungry.
But councils say the money they were given over summer had to be spent within 12 weeks.
Footballer Marcus Rashford’s campaign has built pressure on ministers to feed poorer and vulnerable children directly during school holidays in England.
Some Conservative MPs have expressed concerns that extending this to school holidays would mean families become dependent in the long term on government-funded meals.
“We’ve invested billions of pounds in universal credit and housing allowances,” Rishi Sunak tells BBC Radio 1 Newsbeat.
“We’ve got to be able to trust local councils to make decisions for their people.”
He adds: “We have provided recourses for local authorities to help in a targeted way the most vulnerable children that they need to look after.”
By BBC Reality Check
Prime Minister Boris Johnson told the BBC yesterday: “There’s £63m specifically to help deal with holiday hunger and with pressure on families,” referring to payments made to local authorities in June.
The Local Government Association, which represent councils, agrees this type of funding is the best way of reaching children who need food support.
However, the £63m was for a “local welfare assistance fund” to “assist those struggling to afford food and other essentials” and was not just to feed children.
Guidance for the funding stated that the government “anticipates that most of the funding will be spent within 12 weeks”, meaning that it was expected to have been spent before the end of September.
In England, children living in households on income-related benefits (such as universal credit) are eligible for free school meals, as long as their annual household income does not go over £7,400 after tax, not including welfare payments.
This is the same in Wales and Scotland, while in Northern Ireland the cap is set at £14,000 a year.
If a child is eligible, their parent or guardian can claim at any age – from pre-school to further education.
In England and Scotland, all infant state school pupils (those in reception, year 1 and year 2) can get free school meals during term time – regardless of their household income.
Follow Newsbeat on Instagram, Facebook, Twitter and YouTube.
Listen to Newsbeat live at 12:45 and 17:45 weekdays – or listen back here.