Ghost hunters and off-road bikers are among those damaging a centuries-old castle, volunteers said.
Codnor Castle, near Ripley in Derbyshire, is partly on private land but close to a number of footpaths.
The trust which looks after the ruins and nearby farmhouses said lockdown had seen stones knocked from walls and arson attacks.
They said visitors were welcome if they treated the site with respect, but many seemed not to care.
The castle dates from the 13th Century and is Grade II listed, but it is also on English Heritage’s Heritage at Risk Register.
Nearby farm buildings, parts of which date to the 16th Century, are empty but the Castle Codnor Heritage Trust has planning permission to convert them into homes and a visitor centre.
These have been boarded up, but doors and windows have been forced open.
Rokia Brown, chairwoman of the trust, said: “It started up at the beginning of the first lockdown when the parks were closed and we had increased footfall in the Erewash Valley.
“We got more again when the schools closed, but we’ve also had so-called urban explorers and paranormal investigators.
“It’s a mix of people not understanding where they can’t go and those who just want to cause trouble.”
Other problems included spray paint on walls which took four months to remove, off-road bikers tearing up laid grass and fences being kicked down.
Ms Brown said stones from the castle had been spotted for sale online and a fire had been started in the farmhouse which could have destroyed the building.
“The damage that is being done is going to cost thousands to put right,” she said.
A spokesman for the Derbyshire Historic Buildings Trust said: “We are supportive of the [Codnor Castle Heritage] Trust’s heroic efforts and is extremely concerned to learn of the growing threat to this important part of the County’s heritage, not only the castle remains but also the historic farmhouse and farmyard.
“We must not allow this remarkable historic place to descend into greater dereliction.”