Check if you call the method FRDocument::Close(). This method releases the system resources held by the FRDocument object. So, each call of the Engine::CreateFRDocument method must be accompanied by a call of the FRDocument::Close() method. If it will not help, the possible reason may be in the paging file size.
The virtual memory is managed by the operation system (e.g. on Windows 2008 Server), most possible it works in the following way. As soon as the system feels that memory is going to be consumed, it begins to increase the size of the paging file. If the requests for allocating come quicker than the system increases the paging file, then a part of the requests fail and their initiator thinks that the memory is run out. So, it is necessary to have enough free space on the disk where the Temporary folder is located.