Attendees at the packed meeting said they were upset that an August 12 "Unite the Right" rally was allowed to happen.
Demonstrators stood on the dais and unfurled a large banner that read, "BLOOD ON YOUR HANDS." The meeting was briefly suspended.
Speakers at the council meeting said police seemed to stand by as neo-Nazi marchers and other white nationalists demonstrated.
Many at the contentious meeting on Monday also urged the city to pull down the remaining Confederate monuments in Charlottesville, following the lead of Baltimore and other cities.
"Why do these cities have more moral clarity than we do when we had a Nazi rally in our town?" one resident said.
A Virginia law prohibits cities and people from disturbing or interfering with monuments -- a challenge that Charlottesville officials mentioned.
But the city council voted unanimously to take the initial administrative steps to remove the statues of Robert E. Lee and Stonewall Jackson, Charlottesville Vice-Mayor Wes Bellamy told CNN.
