Several residents in Dahlonega reached out to FOX 5 after waking up Thursday morning and noticing an offensive display on a building located on East Main Street.
The sign depicts a Klan member and reads "Historic Ku Klux Klan Meeting Hall."
Those responsible also appeared to have raised a confederate battle flag as well as a KKK flag.
Watch as nearly 40 protesters gathered outside the building Thursday in opposition of the sign and flags. One woman held a poster that said "Not in my town."
Lieutenant Chris Pfrogner with the Lumpkin County Sheriff's Office told FOX 5's Aungelique Proctor they were made aware of the sign and flags about 7:30 this morning.
Lt. Pfrogner said the property owner had no knowledge of what had taken place overnight. City Manager Bill Schmid said the 6 foot by 2 1/2 foot banner violated the city sign ordinance and had to be taken down immediately.
The city removed the Klan sign around 10 a.m., the flags were removed by unidentified men around 11 a.m.
Nearly 40 protesters gathered outside the building Thursday in opposition to the sign and flags. One woman held a poster that said "Not in my town."
"Our country is so divided right now and there is just so much chaos. Maybe everybody is busy shouting back and forth at eac hother so much that it is allowing these seeds of hate to take hold," Valerie Fambrough said.
"It is devastating to see something like that here in Lumpkin County. As long as I have been here back in the days with my dad working for the college nothing ever happen like this," Vanessa Jackson lamented.
According to Lt. Pfrogner, there's pretty easy access to the roof of the building, but this is the first time something like this has happened.
"There has been no previous Klan activity in the area," Lt. Pfrogner said.
Deputies told FOX 5 a criminal investigation will depend on if the property owner wants to file a police report.