It already has land to use, but the area is zoned for agricultural use, not industrial use.
The company first tried to change the zoning in 2013, but the proposal was rejected.
Neighbors want the area to remain as a recreational and tourist spot, not an industrial area.
Deborah Garrison lives less than a mile away from the quarry. She said 30 years ago, the quarry was a very small operation, but has since expanded with more trucks and more dust.
Garrison said, "The noise is so intense that I can be on my deck and not hear the person standing next to me."
She also said she can feel the almost daily underground blasting.
For a second time, the Allen Company is asking Clark County to rezone its land, now 165 acres in size.
It also wants to build an elevated conveyor over Athens Boonesboro Road and the Kentucky River to transport rock.
A public meeting will be held on May 6 for zoning input at the Clark County Courthouse.