publications

2024

1. ICIS 2024

   

   Integrating LLMs and Psychometrics: Global Construct Validity

   Kai Larsen, Sen Yan, and Lukyanenko Roman

   In Proceedings of the International Conference on Information Systems 2024, Bangkok, Thailand, Dec 2024

   Abs PDF

   The psychometric approach in IS offers a foundational framework for a broad spectrum of research endeavors, which typically rely on construct validation to confirm that a series of indicators accurately measures the intended construct. However, a longstanding issue with construct validity, unaddressed since its introduction by Cronbach and Meehl in 1955, is that it is evaluated using study-specific response data without comparison to constructs outside the study. This oversight (or, rather, incapability) has significant implications. We introduce a large language model combined with principal components analysis (PCA) and develop the Validity Lodestar application. This approach lays the groundwork for developing more accurate and reliable theoretical models, marking a significant leap forward in the IS discipline’s methodological capabilities, making IS the first psychometric discipline with the capability to properly evaluate construct validity.

2. NBER WP

   

   Mining Chinese Historical Sources At Scale: A Machine Learning-Approach to Qing State Capacity

   Wolfgang Keller, Carol Shiue, and Sen Yan

   Demo Website Link , Dec 2024

   Abs PDF Code

   Primary historical sources are often by-passed for secondary sources due to high human costs of accessing and extracting primary information–especially in lower-resource settings. We propose a supervised machine-learning approach to the natural language processing of Chinese historical data. An application to identifying different forms of social unrest in the Veritable Records of the Qing Dynasty shows that approach cuts dramatically down the cost of using primary source data at the same time when it is free from human bias, reproducible, and flexible enough to address particular questions. External evidence on triggers of unrest also suggests that the computer-based approach is no less successful in identifying social unrest than human researchers are.

2022

1. DESRIST 2022

   

   Toward an Information Systems Ontology

   Roland M. Mueller, Sebastian Huettemann, Kai R. Larsen, and 2 more authors

   In Proceedings of 17th International Conference on Design Science Research in Information Systems and Technology DESRIST-2022, St. Petersburg, FL, USA, Jun 2022

   Abs PDF

   We introduce the Information Systems Ontology (ISO), a new ontology for the Information Systems (IS) discipline designed to enable automated knowledge synthesis and meta-analysis of research findings in IS. We constructed ISO in a methodical manner, following known best practices for ontology construction. We also conducted a series of ontology refinement steps in which we compared and extended ISO by extracting and examining both overlapping and missing key phrases from scientific articles and existing classification schemas. To evaluate ISO, we extracted author-defined keywords from more than 7,000 articles of the senior scholars’ basket of journals and measured terminological coverage. In one experiment, we found that our ontology included 3.6 times more author-defined keywords than an established classification schema for IS. In the future, we plan to use ISO to automatically annotate important IS terms and concepts in IS articles to help synthesize and analyze knowledge in IS.

2021

1. IJCAI 2021

   

   Isotonic Data Augmentation for Knowledge Distillation

   Wanyun Cui, and Sen Yan

   In Proceedings of the 30th International Joint Conference on Artificial Intelligence, IJCAI-2021, Aug 2021

   Main Track

   Abs PDF Code

   Knowledge distillation uses both real hard labels and soft labels predicted by teacher models as supervision. Intuitively, we expect the soft labels and hard labels to be concordant w.r.t. their orders of probabilities. However, we found critical order violations between hard labels and soft labels in augmented samples. For example, for an augmented sample x=0.7∗panda+0.3∗cat, we expect the order of meaningful soft labels to be Psoft(panda|x)>Psoft(cat|x)>Psoft(other|x). But real soft labels usually violate the order, e.g. Psoft(tiger|x)>Psoft(panda|x)>Psoft(cat|x). We attribute this to the unsatisfactory generalization ability of the teacher, which leads to the prediction error of augmented samples. Empirically, we found the violations are common and injure the knowledge transfer. In this paper, we introduce order restrictions to data augmentation for knowledge distillation, which is denoted as isotonic data augmentation (IDA). We use isotonic regression (IR) – a classic technique from statistics – to eliminate the order violations. We show that IDA can be modeled as a tree-structured IR problem. We thereby adapt the classical IRT-BIN algorithm for optimal solutions with O(clogc) time complexity, where c is the number of labels. In order to further reduce the time complexity, we also propose a GPU-friendly approximation with linear time complexity. We have verified on variant datasets and data augmentation techniques that our proposed IDA algorithms effectively increases the accuracy of knowledge distillation by eliminating the rank violations.

2020

1. WITS 2020

   

   UnitedANT: A Multimodal Deep Learning Framework for Predicting Financial Risk from Acoustic, Numeric, and Textual Cues in Earnings Conference Calls

   Sen Yan, Bao Yang, and Fang Hui

   In Proceedings of the 30th WORKSHOP ON INFORMATION TECHNOLOGIES AND SYSTEMS , WITS-2020, Dec 2020

   Abs PDF Code

   Earnings conference calls have been recently recognized as significant information events to the market due to its less constrained fashion and direct interaction between managers and analysts. However, it is a non-trivial task to fully exploit the information contained in these conference calls due to its multimodality. To tackle this problem, we develop a general multimodal deep learning framework called UnitedANT (A, N, T stands for Acoustic, Numeric, and Textual information respectively) which could simultaneously leverage acoustic, numeric, and textual information of conference calls for predicting corporate financial risk. Empirical results on a real-world dataset of S&P 500 companies demonstrate the superiority of our proposed method over competitive baselines from the extant literature. Our ablation study presents evidence that all three modalities are useful for financial risk prediction, and the exclusion of any one or two of them will lead to a drop in model performance.