The error message "An attempt was made to load a program with an incorrect format" appears if 32-bits DLL is loaded in 64-bit application.

FlexiCapture Engine 12, FineReader Engine 11 and 12

Please make sure that you have loaded the correct FREngine.dll or FCEngine.dll version. The version for use in 64-bits system is in your ABBYY product installation folder, subfolder "Bin64". The files from "Bin" folder can be used in 32-bits applications only. 

FlexiCapture Engine 11

FlexiCapture Engine 11 works on 64-bit systems in emulation mode (as a 32-bit application). 
According to System Requirements (see the Help file → Specifications → System Requirements) there is support of the following 64-bit OS:

• Windows Server 2016
• Windows 10
• Windows Server 2012 R2 
• Windows Server 2012 
• Windows 8.1
• Windows 8
• Windows Server 2008 R2 
• Windows 7
• Windows Server 2008 SP1-SP2 
• Windows Vista SP1-SP2

However we would like to note that this product is 32-bit, we do not have native 64-bit support and that’s why the FCE library cannot be used in the 64-bit applications directly. The applications developed with the FCEngine are 32-bit and can be launched in emulation mode.

Some information regarding the message is available at http://blogs.msdn.com/b/karthick/archive/2006/02/28/540780.aspx.