Affordable housing advocates facing opposition