FileLoadException, hugh?

We recently built up two new servers in production for a customer facing site.  Several days after going live, we received a note from development stating one of the four servers was throwing errors.  A simple glance at the event logs revealed the following issue and the problematic server:

Exception information:
    Exception type: FileLoadException
    Exception message: Could not load file or assembly '****, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null' or one of its dependencies. Access is denied.

This is odd and definitely permission in nature.  As an initial step, I reset the application pool, then monitored the event log.  Several hours went by and no indication of the same event.  Problem resolved, so I thought.  Next day, another note from development stating the same issue occurring once again.  Started to ask some questions and realized there was a certain component of the site which was using a domain account to do impersonation in the web.config.  As a result, that particular component did not have rights to the .NET temp folder:

C:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files

I added exclusive rights for this folder and all sub-folders, gave the application a quick reset, and the problem went away.  I am still a bit bit perplexed as to why my initial resets were stabilizing the application for hours on end, then all of a sudden the problem would come back.  In any case, it was the impersonation that was causing the issue and no rights to the .NET temp folder.