@Builder generates its own class classNameBuilder with methods and fields. That's why it's not using your setter from @Setter annotation. Those both annotations does not depend on each other, that's why after you removed setter builder still works.
See the Lombok's doc https://projectlombok.org/features/Builder.html
Could you please share more of your code so it will be easier to understand what you want to achieve?
