I've seen this happen before, in a custom module I built for a 7.1 site. The URL File Picker loads all folders in a portal into viewstate, regardless of what you put in the settings. Not sure if these are cached or not though. At anyrate, if you have a lot of users or a lot of folders and files, you will def notice. The more file upload controls on a page, the worse it gets. I noticed this when I used grunt.js in my skin folder. Grunt will produce a ton of little files/folders when it's doing it's processing, and all those get indexed. Now when I create my skins, I put the css and js in a folder off the installation root and have grunt output the files to the skins directory. 

That was a long way of saying that the File Picker could use some optimization.