This paper presents an optimization technique for clamping forces determination in fixture design. First, the finite element analysis (FEA) is applied to determine the coefficients of compliant matrix of a fixture-workpiece system subjected to machining and clamping forces. Then, a nonlinear optimization model is constructed in terms of the FEA results and mechanical and geometrical constraints. The optimization model is derived to determine the feasible clamps under the corresponding force effects. The optimal magnitude and direction of clamping forces minimize the workpiece deformation at particular key points. Finally, a scaled engine block with the 3-2-1 fixturing principle is given as an example.