An explainable multi tiered career recommender system using shap guided natural language interpretations

Abstract

Career guidance plays a vital role in helping students make informed educational and occupational choices. However, the shortage of trained career counsellors in countries such as India limits access to personalised guidance, often resulting in inappropriate course selection, reduced employability, and increased career switching, leading to the underutilisation of human resource potential and reduced returns on public investment in education. To address these challenges, this thesis proposes an explainable multi-tiered career recommender system that combines machine learning, explainable artificial intelligence, and natural language generation to deliver scalable, transparent, and personalised career recommendations. The primary contribution of this research is a novel hierarchical and multi-tiered recommendation architecture designed to generate diverse yet relevant career recommendations aligned with a candidate's aptitude profile. Unlike conventional recommender systems that often produce narrow recommendation lists, the proposed architecture organises careers into aptitude-based clusters and integrates multiple machine learning models within a content-based filtering framework, enabling recommendations from diverse career domains while preserving their suitability to the candidate's abilities. A second major contribution is a hybrid explainability framework that integrates the local and global explainability capabilities of SHAP (SHapley Additive exPlanations) to provide both candidate-level and career-level explanations. These explanations are automatically translated into natural language interpretations using NLG techniques, making recommendation outcomes more transparent and understandable. The recommendation framework is supported by a first-of-its-kind aptitude-based career dataset developed using career ability profiles derived from the Occupational Information Network (O*NET) database and candidate aptitude profiles generated using the Kerala Differential Aptitude Tests (KDAT), thereby addressing the traditional data sparsity and cold-start limitations encountered in career recommender systems. The thesis further introduces a scalable semi-supervised career expansion mechanism that enables the incorporation of emerging careers into the recommendation framework without retraining the underlying recommendation model. Experimental and expert evaluations demonstrated the robustness, effectiveness, and acceptability of both the recommendations and the generated explanations. Overall, this research presents novel contributions in both recommendation and explainability architectures and demonstrates how explainable machine learning can be effectively integrated into career counselling to deliver scalable, diverse, interpretable, and trustworthy recommendations.