Research Papers

The article examines whether it is feasible to apply Vision Transformers (ViTs) and Audio Transformers (ATs) to perform an evaluation of job interview applicants utilizing both visual and auditory data to supplement judgment on the applicants. It compares unimodal models, such as ViT, which is a model that has been trained on visual features, and AT, which is a model that has been trained on auditory features, and the multimodal model that is a combination of the two. An accuracy rate of 78 per cent, which is good in detecting the facial expression, body position and eye contact, yet failed in the verbal analysis, was observed in the ViT model. The AT model was discovered to be accurate in 82 per cent and could identify changes in tones and speech patterns amazingly yet could not understand non-verbal behaviours. The combination model that was referred to as multimodal model involving the incorporation of the best of the two models had an accuracy of 89 percent and was quite impressive compared to the unimodal models. This model also performed the best at the ROC curve with AUC of 0.92 as compared to ViT and AT AUC of 0.82 and 0.85 respectively. Also, the loss and accuracy charts of the training and validation indicated that the multimodal model has superior learning behaviour with training accuracy rising between 74 and 92 and validation loss of 0.2 as opposed to the unimodal models. This means that the use of both the visual and audio information is a significant Stride toward enhancing the accuracy and consistency of using such automated job interviews in determining the passengers more comprehensively and objectively during the process of recruiting them. The paper points out the possibilities of multimodal AI models to conduct scalable, bias-free, and efficient recruitment procedures.