This paper describes two optimization algorithms used to calculate the complex permittivity and the thickness of individual dielectric layers of multilayered structure in the W-band. The algorithms are the genetic algorithm (GA) and the sequential quadratic programming (SQP) based on the quasi-Newton method BFGS (BFGS-SQP); the setup procedure of both algorithms is presented. A quasi-optical free space measurement system is used to measure the reflection and the transmission coefficients. The performance of the measurement setup is first tested using different single layer slabs. A detailed comparison between the two algorithms is reported.