I have heard reports of this happening to quite a few people, and users of the documents module especially (since it relies on the file manager).
I am not sure of the end game fix on this, but I know that the following series of events helped on person.
NOTE: YOU MUST BACKUP YOUR DATABASE BEFORE DOING THIS, IT IS POSSIBLE THAT THIS WILL PREVENT FILE MANAGER FROM SYNCING AGAIN, BUT IN SOME CASES I HAVE NOTICED IMPROVEMENTS IN FUNCTIONALIYT! I AM NOT RESPONSIBLE FOR ANYTHING THAT GOES WRONG WITH THIS. If you have concerns or questions please contact me first, typically via my website or e-mail is best due to the lack of forum notifications!
First of all turn off the auto sync option for the file system, including the scheduled task.
Second truncate the Files and Folders tables.
Then manually run the file sync via File Manager, be sure to check the Recursive box before running the sync.
NOTE: you will need to modify file links in modules that link to files based on file id.