I remember that awkward period after we outgrew floppies but before USB flash storage became mainstream when CD-RW and CD+RW (Yes there was a difference and it mattered!) being peddled as the next portable storage format. My first USB stick was 128 megabytes and cost around $60. I actually still have it, but it may no longer work. I believe I bought it in around 2003 or 2004. I remember them steadily decreasing in price, or at least price per unit storage, very quickly over the ensuing years, to the point where they were giving away multi-gig sticks by around 2009 or 2010.
I don’t remember the first time I used one, but the first one I bought was a 1GB PNY Attaché for about $29.99 in 2005, at Best Buy. I used that thing for many, many years, even after the glue holding the plastic failed and the connector receded into the housing. I still have it around here somewhere, it’s just a pain having to pull the connector out to use it.