For me, sometimes with huge files even if the copy process ended, and I do a “safely remove”, sometimes I still have to wait a big chunk of time. Safely remove is actually safe, nothing should be left. I also thought this was a feature of the filesystem, where it caches and optimizes the actions before doing it under the hood.

With something like System Monitor you should be able to see if anything is writing or reading on the filesystem constantly, before you do a disk remove.