You could move the file manager page from the Admin menu to another menu structure and make security changes it.   Instead of doing that you might want to look at a file manager module on Snowcovered for $10 that works well (TK File Manager).  It is basicly a modification of the standard file manager, but you can set a top level folder.  It does not have all the features of the other DNN file managers on Snowcovered, but it works.

You can also use the available Repository Module. That doesn't give regular users Admin access to the File Manager, but restrictive Upload/Download and View permsisions on specific files.