An attempt was made to load a program with an incorrect format

At the moment FlexiCapture Engine works on 64-bit systems in emulation mode (as a 32-bit application). 
According to System Requirements (see the Help file → Specifications → System Requirements) there are support of the following 64-bit OS:

• Windows Server 2008 R2 
• Windows 7 
• Windows Server 2008 SP1-SP2 
• Windows Vista SP1-SP2 
• Windows Server 2003 SP1-SP2, R2 
• Windows XP SP1-SP3 

However we would like to note that our products are 32-bit, we do not have a native 64-bit support and that’s why the FCE library cannot be used in the 64-bit applications. The applications developed with the FREngine are 32-bit and can be launched in emulation mode.

Some information regarding the message is available at http://blogs.msdn.com/b/karthick/archive/2006/02/28/540780.aspx.