This study had 3 objectives: to estimate genetic parameters and predict sires' transmitting abilities for clinical mastitis in a Spanish Holstein population, to propose a methodology for comparing models with different response variables by using a cost-based loss function, and to evaluate alternative genetic evaluation models by using this methodology. On-farm records for clinical mastitis from herds in 3 Spanish regions were analyzed as a binary trait (CM) and as number of episodes (NCM) per lactation. Linear and probit models were fitted for CM, whereas linear and Poisson models were used for NCM. Predictive ability of the models was evaluated by using the average predicted residual sum of squares from cross-validation and an alternative cost-based loss function. The loss function for model comparison was calculated by using average mastitis costs depending on the NCM and average cost per infected lactation. The average cost per infected lactation was $345.58, whereas the cost per lactation ranged from $204.86 to $985.44 for lactations with 1 to 5 cases, respectively. Management and hygiene practices on individual farms had a large impact on clinical mastitis because the herd-year variance was larger than that of other random effects considered. The sire variance was significantly different from zero, confirming that genetic variation exists for clinical mastitis. Estimates of heritability for CM using the linear and probit models were 0.07 and 0.10 on the underlying scale, respectively. For NCM, the estimate of heritability for the linear model was 0.10 and estimates for the Poisson model evaluated at the mean and the median of lambda on the underlying scale were 0.09 and 0.07, respectively. Regarding ranking of sires, the definition of response variable (CM or NCM) was of greater importance than the choice of statistical model. Cross-validation results indicated that models with the best fit for CM and NCM were the probit model and the linear model, respectively. However, a comparison across all models using the alternative cost-based loss function showed that using NCM as a response variable with a Poisson model provided the most accurate predictions of future costs associated with clinical mastitis.