This study presents a genetic algorithm optimization of a hybrid analog/digital predistorter, in order to reduce the intermodulation distortion (IMD) caused by the nonlinear properties of the radio frequency (RF) power amplifier (PA). Designed predistorter based on polynomial work function and the coefficient of the polynomial is optimized in order to reduce IMD by spectrum monitoring. The design procedure and validation of predistorter have been carried out by Agilent-ADS2005A. In order to validate the predistorter two different modulation schemes as CDMA and 16-QAM have been used. Also in order to verifying the linearization a test power amplifier circuit has been examined including Motorola MOSFET MRF9742 showing the nonlinear characteristics with memory. Simulations have been shown that adjacent channel power ratio (ACPR) improvements were acceptable for both CDMA and 16-QAM modulation schemes.