I worked in a developing country for three years. Women’s breasts were not considered private. As a result, I saw breasts frequently. By about month two, a uncovered female breast was about as interesting to me as her elbow. There’s no reason to sexualize the breast. They are baby-feeding organs. Not to go all Freud on you, but I wonder if it’s a remnant of the breast-feeding positive feelings.
I mean, there are people who go freaky at the sight of feet even though you can see bare naked ones on every beach in the world.
Could it be that you were never really interested in breasts in the first place?