The purposes of the present study were two-fold: first, to evaluate whether reflection journal writing was effective in promoting self-reflection and learning, and whether students become better at self-reflection if they engage continuously in reflection journal writing. To that end, the reflection journals of 690 first-year applied science students at a local polytechnic were studied by means of an automated coding procedures using software. Data was collected twice, once at the beginning and again towards the end of an academic year. Outcomes of the textual content analyses revealed that students reflected on both the process and contents of their learning: critical review of past learning experiences, learning strategies and summaries of what was learned. Correlational analyses showed weak to moderate inter-relationships between the textual categories and their classroom and knowledge acquisition test grades. Taken together, the findings suggest that self-reflection on both how and what students have learned does lead to improvements in academic performance, although to a limited extent.