DEVELOPMENT OF A MULTITASK TRANSFORMER-BASED MODEL AND WEB SYSTEM FOR JOINT SENTIMENT ANALYSIS AND AI-GENERATED REVIEW PROVENANCE CLASSIFICATION: A MIXED-DOMAIN CORPUS EVALUATION WITH HOTEL-ONLY PROVENANCE LABELS

Abstract

Online reviews contain both an evaluative component and information regarding the source or method of the text's creation. This article presents the completed experimental phase of a dissertation study, which involved the development of a multi-task model based on the Transformer architecture, as well as a web-based system called ReviewGuard for the comprehensive analysis of reviews. The model addresses two tasks: determining sentiment (positive, negative, or neutral) and identifying the origin of the text (human-written versus AIgenerated, based on the classification used in the MAiDE-up resource). This approach to determining origin is more narrowly defined than the broader task of detecting fake reviews. A corpus of 30,000 entries was compiled from four publicly available datasets. Several approaches were compared: the TF-IDF method combined with logistic regression, two specialized (single-task) XLM-RoBERTa models, and a single multi-task model with a shared encoder. The experiments were conducted with controls in place to prevent data leakage, and five different random seeds were used for training initialization. The multi-task model achieved the following results (Macro-F1 metric): 0.7709 ± 0.0087 for sentiment analysis and 0.9526 ± 0.0094 for origin detection (for comparison, the baseline method yielded 0.7161 and 0.8998, respectively). The differences in performance compared to a single-task Transformer model were marginal: -0.0013 for sentiment analysis and -0.0046 for origin detection. The data obtained do not allow for a definitive conclusion regarding whether a multi-task model is equivalent or superior to single-task models; furthermore, they do not confirm the presence of "positive transfer" of knowledge or the model's ability to generalize beyond the specific sources and objects used. The web-based system provides the user with model predictions, uncalibrated confidence scores, information on text truncation, and warnings–all of which are significant in a research context where a human is involved in the decision-making process.