publications

2024

1. EMNLP Workshop

   Understanding Players as if They Are Talking to the Game in a Customized Language: A Pilot Study

   Tianze Wang, Maryam Honarijahromi, Styliani Katsarou, Olga Mikheeva, Theodoros Panagiotakopoulos, Oleg Smirnov, Lele Cao, and Sahar Asadi

   In EMNLP Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (CustomNLP4U), Nov 2024

   Abs DOI PDF

   This pilot study explores the application of language models (LMs) to model game event sequences, treating them as a customized natural language. We investigate a popular mobile game, transforming raw event data into textual sequences and pretraining a Longformer model on this data. Our approach captures the rich and nuanced interactions within game sessions, effectively identifying meaningful player segments. The results demonstrate the potential of self-supervised LMs in enhancing game design and personalization without relying on ground-truth labels.

2. NAACL Workshop

   Introducing GenCeption for Multimodal LLM Benchmarking: You May Bypass Annotations

   Lele Cao, Valentin Buchner, Zineb Senane, and Fangkai Yang

   In NAACL Workshop on Trustworthy Natural Language Processing (TrustNLP 2024), Jun 2024

   Abs DOI PDF Supp Code

   

   Multimodal Large Language Models (MLLMs) are commonly evaluated using costly annotated multimodal benchmarks. However, these benchmarks often struggle to keep pace with the rapidly advancing requirements of MLLM evaluation. We propose GenCeption, a novel and annotation-free MLLM evaluation framework that merely requires unimodal data to assess inter-modality semantic coherence and inversely reflects the models’ inclination to hallucinate. Analogous to the popular DrawCeption game, GenCeption initiates with a non-textual sample and undergoes a series of iterative description and generation steps. Semantic drift across iterations is quantified using the GC@T metric. Our empirical findings validate GenCeption’s efficacy, showing strong correlations with popular MLLM benchmarking results. GenCeption may be extended to mitigate training data contamination by utilizing ubiquitous, previously unseen unimodal data.

3. ICANN

   Beyond Gut Feel: Using Time Series Transformers to Find Investment Gems

   Lele Cao^*, Gustaf Halvardsson^*, Andrew McCornack, Vilhelm Ehrenheim, and Pawel Herman

   In Artificial Neural Networks and Machine Learning – ICANN 2024, Jun 2024

   Abs DOI arXiv Blog

   This paper addresses the growing application of data-driven approaches within the Private Equity (PE) industry, particularly in sourcing investment targets (i.e., companies) for Venture Capital (VC) and Growth Capital (GC). We present a comprehensive review of the relevant approaches and propose a novel approach leveraging a Transformer-based Multivariate Time Series Classifier (TMTSC) for predicting the success likelihood of any candidate company. The objective of our research is to optimize sourcing performance for VC and GC investments by formally defining the sourcing problem as a multivariate time series classification task. We consecutively introduce the key components of our implementation which collectively contribute to the successful application of TMTSC in VC/GC sourcing: input features, model architecture, optimization target, and investor-centric data processing. Our extensive experiments on two real-world investment tasks, benchmarked towards three popular baselines, demonstrate the effectiveness of our approach in improving decision making within the VC and GC industry.

4. SIGKDD

   Self-Supervised Learning of Time Series Representation via Diffusion Process and Imputation-Interpolation-Forecasting Mask

   Zineb Senane^*, Lele Cao^*, Valentin Leonhard Buchner, Yusuke Tashiro, Lei You, Pawel Andrzej Herman, Mats Nordahl, Ruibo Tu, and Vilhelm Ehrenheim

   In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Barcelona, Spain, Jun 2024

   Abs DOI PDF Video Blog Code

   

   Time Series Representation Learning (TSRL) focuses on generating informative representations for various Time Series (TS) modeling tasks. Traditional Self-Supervised Learning (SSL) methods in TSRL fall into four main categories: reconstructive, adversarial, contrastive, and predictive, each with a common challenge of sensitivity to noise and intricate data nuances. Recently, diffusion-based methods have shown advanced generative capabilities. However, they primarily target specific application scenarios like imputation and forecasting, leaving a gap in leveraging diffusion models for generic TSRL. Our work, Time Series Diffusion Embedding (TSDE), bridges this gap as the first diffusion-based SSL TSRL approach. TSDE segments TS data into observed and masked parts using an Imputation-Interpolation-Forecasting (IIF) mask. It applies a trainable embedding function, featuring dual-orthogonal Transformer encoders with a crossover mechanism, to the observed part. We train a reverse diffusion process conditioned on the embeddings, designed to predict noise added to the masked part. Extensive experiments demonstrate TSDE’s superiority in imputation, interpolation, forecasting, anomaly detection, classification, and clustering. We also conduct an ablation study, present embedding visualizations, and compare inference speed, further substantiating TSDE’s efficiency and validity in learning representations of TS data.

5. SIGKDD

   CompanyKG2: A Large-Scale Heterogeneous Graph for Company Similarity Quantification

   Lele Cao, Vilhelm Ehrenheim, Mark Granroth-Wilding, Richard Anselmo Stahl, Andrew McCornack, Armin Catovic, and Dhiana Deva Cavalcanti Rocha

   In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Barcelona, Spain, Jun 2024

   Abs DOI PDF Video Code Website

   

   This paper presents CompanyKG (version 2), a large-scale heterogeneous graph developed for fine-grained company similarity quantification and relationship prediction, crucial for applications in the investment industry such as market mapping, competitor analysis, and mergers and acquisitions. CompanyKG comprises 1.17 million companies represented as graph nodes, enriched with company description embeddings, and 51.06 million weighted edges denoting 15 distinct inter-company relations. To facilitate a thorough evaluation of methods for company similarity quantification and relationship prediction, we have created four annotated evaluation tasks: similarity prediction, competitor retrieval, similarity ranking, and edge prediction. We offer extensive benchmarking results for 11 reproducible predictive methods, categorized into three groups: node-only, edge-only, and node+edge. To our knowledge, CompanyKG is the first large-scale heterogeneous graph dataset derived from a real-world investment platform, specifically tailored for quantifying inter-company similarity and relationships.

6. NAACL

   Prompt Tuned Embedding Classification for Industry Sector Allocation

   Valentin Buchner^*, Lele Cao^*, Jan-Christoph Kalo, and Vilhelm Von Ehrenheim

   In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), Jun 2024

   Abs DOI PDF Video Code Poster Slides

   

   We introduce Prompt Tuned Embedding Classification (PTEC) for classifying companies within an investment firm’s proprietary industry taxonomy, supporting their thematic investment strategy. PTEC assigns companies to the sectors they primarily operate in, conceptualizing this process as a multi-label text classification task. Prompt Tuning, usually deployed as a text-to-text (T2T) classification approach, ensures low computational cost while maintaining high task performance. However, T2T classification has limitations on multi-label tasks due to the generation of non-existing labels, permutation invariance of the label sequence, and a lack of confidence scores. PTEC addresses these limitations by utilizing a classification head in place of the Large Language Models (LLMs) language head. PTEC surpasses both baselines and human performance while lowering computational demands. This indicates the continuing need to adapt state-of-the-art methods to domain-specific tasks, even in the era of LLMs with strong generalization abilities.

2023

1. IJCAI

   Measuring Acoustics with Collaborative Multiple Agents

   Yinfeng Yu, Changan Chen, Lele Cao, Fangkai Yang, and Fuchun Sun

   In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23, Aug 2023

   Main Track

   Abs DOI arXiv PDF Code Slides Website

   As humans, we hear sound every second of our life. The sound we hear is often affected by the acoustics of the environment surrounding us. For example, a spacious hall leads to more reverberation. Room Impulse Responses (RIR) are commonly used to characterize environment acoustics as a function of the scene geometry, materials, and source/receiver locations. Traditionally, RIRs are measured by setting up a loudspeaker and microphone in the environment for all source/receiver locations, which is time-consuming and inefficient. We propose to let two robots measure the environment’s acoustics by actively moving and emitting/receiving sweep signals. We also devise a collaborative multi-agent policy where these two robots are trained to explore the environment’s acoustics while being rewarded for wide exploration and accurate prediction. We show that the robots learn to collaborate and move to explore environment acoustics while minimizing the prediction error. To the best of our knowledge, we present the very first problem formulation and solution to the task of collaborative environment acoustics measurements with multiple agents.

2. Journal

   Echo-Enhanced Embodied Visual Navigation

   Yinfeng Yu^*, Lele Cao^*, Fuchun Sun, Chao Yang, Huicheng Lai, and Wenbing Huang

   Neural Computation, Apr 2023

   Abs DOI PDF Website

   

   Visual navigation involves a movable robotic agent striving to reach a point goal (target location) using vision sensory input. While navigation with ideal visibility has seen plenty of success, it becomes challenging in suboptimal visual conditions like poor illumination, where traditional approaches suffer from severe performance degradation. We propose E3VN (echo-enhanced embodied visual navigation) to effectively perceive the surroundings even under poor visibility to mitigate this problem. This is made possible by adopting an echoer that actively perceives the environment via auditory signals. E3VN models the robot agent as playing a cooperative Markov game with that echoer. The action policies of robot and echoer are jointly optimized to maximize the reward in a two-stream actor-critic architecture. During optimization, the reward is also adaptively decomposed into the robot and echoer parts. Our experiments and ablation studies show that E3VN is consistently effective and robust in point goal navigation tasks, especially under nonideal visibility.

3. IJCAI Workshop

   Using Deep Learning to Find the Next Unicorn: A Practical Synthesis on Optimization Target, Feature Selection, Data Split and Evaluation Strategy

   Lele Cao, Vilhelm Ehrenheim, Sebastian Krakowski, Xiaoxue Li, and Alexandra Lutz

   In IJCAI Workshop of Multimodal AI For Financial Forecasting (MuFFin), Aug 2023

   Abs arXiv PDF Blog Slides Website

   

   Startups represent newly established business models associated with disruptive innovation and high scalability, hence strongly propel the economic and social development. Meanwhile, startups are heavily constrained by many factors such as limited financial funding and human resources. Therefore, the chance for a startup to succeed is rare like “finding a unicorn in the wild”. Venture Capital strives to identify and invest in unicorn startups as early as possible, hoping to gain a high return. This work is traditionally manual and empirical, making it inherently biased and hard to scale. Recently, the rapid growth of data volume and variety is quickly ushering in deep learning (DL) as a potentially superior approach in this domain. In this work, we carry out a literature review and synthesis on DL-based approaches, emphasizing four key aspects: optimization target, feature selection, data split, and evaluation strategy. For each aspect, we summarize our in-depth understanding and practical learning.

4. IJCAI Workshop

   A Scalable and Adaptive System to Infer the Industry Sectors of Companies: Prompt + Model Tuning of Generative Language Models

   Lele Cao, Vilhelm Ehrenheim, Astrid Berghult, Cecilia Henje, Richard Anselmo Stahl, Joar Wandborg, Sebastian Stan, Armin Catovic, Erik Ferm, and Hannes Ingelhag

   In IJCAI Workshop of Financial Technology and Natural Language Processing (FinNLP), Aug 2023

   Abs arXiv PDF Website

   

   The Private Equity (PE) firms operate investment funds by acquiring and managing companies to achieve a high return upon selling. Many PE funds are thematic, meaning investment professionals aim to identify trends by covering as many industry sectors as possible, and picking promising companies within these sectors. So, inferring sectors for companies is critical to the success of thematic PE funds. In this work, we standardize the sector framework and discuss the typical challenges; we then introduce our sector inference system addressing these challenges. Specifically, our system is built on a medium-sized generative language model, finetuned with a prompt + model tuning procedure. The deployed model demonstrates a superior performance than the common baselines. The system has been serving many PE professionals for over a year, showing great scalability to data volume and adaptability to any change in sector framework and/or annotation.

2022

1. CVPR

   Multimodal Token Fusion for Vision Transformers

   Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, and Yunhe Wang

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2022

   Abs DOI PDF Supp Video Code

   

   Many adaptations of transformers have emerged to address the single-modal vision tasks, where self-attention modules are stacked to handle input sources like images. Intuitively, feeding multiple modalities of data to vision transformers could improve the performance, yet the inner-modal attentive weights may be diluted, which could thus greatly undermine the final performance. In this paper, we propose a multimodal token fusion method (TokenFusion), tailored for transformer-based vision tasks. To effectively fuse multiple modalities, TokenFusion dynamically detects uninformative tokens and substitute these tokens with projected and aggregated inter-modal features. Residual positional alignment is also adopted to enable explicit utilization of the inter-modal alignments after fusion. The design of TokenFusion allows the transformer to learn correlations among multimodal features, while the single-modal transformer architecture remains largely intact. Extensive experiments are conducted on a variety of homogeneous and heterogeneous modalities and demonstrate that TokenFusion surpasses state-of-the-art methods in three typical vision tasks: multimodal image-to-image translation, RGB-depth semantic segmentation, and 3D object detection with point cloud and images.

2. CIKM

   Simulation-Informed Revenue Extrapolation with Confidence Estimate for Scaleup Companies Using Scarce Time-Series Data

   Lele Cao, Sonja Horn, Vilhelm Ehrenheim, Richard Anselmo Stahl, and Henrik Landgren

   In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, Atlanta, GA, USA, Jun 2022

   Abs DOI PDF Video Blog Code Poster Slides

   

   Investment professionals rely on extrapolating company revenue into the future (i.e. revenue forecast) to approximate the valuation of scaleups (private companies in a high-growth stage) and inform their investment decision. This task is manual and empirical, leaving the forecast quality heavily dependent on the investment professionals’ experiences and insights. Furthermore, financial data on scaleups is typically proprietary, costly and scarce, ruling out the wide adoption of data-driven approaches. To this end, we propose a simulation-informed revenue extrapolation (SiRE) algorithm that generates fine-grained long-term revenue predictions on small datasets and short time-series. SiRE models the revenue dynamics as a linear dynamical system (LDS), which is solved using the EM algorithm. The main innovation lies in how the noisy revenue measurements are obtained during training and inferencing. SiRE works for scaleups that operate in various sectors and provides confidence estimates. The quantitative experiments on two practical tasks show that SiRE significantly surpasses the baseline methods by a large margin. We also observe high performance when SiRE extrapolates long-term predictions from short time-series. The performance-efficiency balance and result explainability of SiRE are also validated empirically. Evaluated from the perspective of investment professionals, SiRE can precisely locate the scaleups that have a great potential return in 2 to 5 years. Furthermore, our qualitative inspection illustrates some advantageous attributes of the SiRE revenue forecasts.

3. CVPR

   Bridged Transformer for Vision and Point Cloud 3D Object Detection

   Yikai Wang, TengQi Ye, Lele Cao, Wenbing Huang, Fuchun Sun, Fengxiang He, and Dacheng Tao

   In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2022

   Abs DOI PDF Supp

   

   3D object detection is a crucial research topic in computer vision, which usually uses 3D point clouds as input in conventional setups. Recently, there is a trend of leveraging multiple sources of input data, such as complementing the 3D point cloud with 2D images that often have richer color and fewer noises. However, due to the heterogeneous geometrics of the 2D and 3D representations, it prevents us from applying off-the-shelf neural networks to achieve multimodal fusion. To that end, we propose Bridged Transformer (BrT), an end-to-end architecture for 3D object detection. BrT is simple and effective, which learns to identify 3D and 2D object bounding boxes from both points and image patches. A key element of BrT lies in the utilization of object queries for bridging 3D and 2D spaces, which unifies different sources of data representations in Transformer. We adopt a form of feature aggregation realized by point-to-patch projections which further strengthen the interaction between images and points. Moreover, BrT works seamlessly for fusing the point cloud with multi-view images. We experimentally show that BrT surpasses state-of-the-art methods on SUN RGB-D and ScanNetV2 datasets.

4. BMVC

   Pay Self-Attention to Audio-Visual Navigation

   Yinfeng Yu^*, Lele Cao^*, Fuchun Sun, Xiaohong Liu, and Liejun Wang

   In Proceedings of British Machine Vision Conference, Jun 2022

   Abs PDF Code Poster Website

   

   Audio-visual embodied navigation, as a hot research topic, aims training a robot to reach an audio target using egocentric visual (from the sensors mounted on the robot) and audio (emitted from the target) input. The audio-visual information fusion strategy is naturally important to the navigation performance, but the state-of-the-art methods still simply concatenate the visual and audio features, potentially ignoring the direct impact of context. Moreover, the existing approaches requires either phase-wise training or additional aid (e.g. topology graph and sound semantics). Up till this date, the work that deals with the more challenging setup with moving target(s) is still rare. As a result, we propose an end-to-end framework FSAAVN (feature self-attention audio-visual navigation) to learn chasing after a moving audio target using a context-aware audio-visual fusion strategy implemented as a self-attention module. Our thorough experiments validate the superior performance (both quantitatively and qualitatively) of FSAAVN in comparison with the state-of-the-arts, and also provide unique insights about the choice of visual modalities, visual/audio encoder backbones and fusion patterns.

2021

1. EMNLP

   PAUSE: Positive and Annealed Unlabeled Sentence Embedding

   Lele Cao, Emil Larsson, Vilhelm Ehrenheim, Dhiana Deva Cavalcanti Rocha, Anna Martin, and Sonja Horn

   In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Nov 2021

   Abs DOI PDF Video Code

   Sentence embedding refers to a set of effective and versatile techniques for converting raw text into numerical vector representations that can be used in a wide range of natural language processing (NLP) applications. The majority of these techniques are either supervised or unsupervised. Compared to the unsupervised methods, the supervised ones make less assumptions about optimization objectives and usually achieve better results. However, the training requires a large amount of labeled sentence pairs, which is not available in many industrial scenarios. To that end, we propose a generic and end-to-end approach – PAUSE (Positive and Annealed Unlabeled Sentence Embedding), capable of learning high-quality sentence embeddings from a partially labeled dataset. We experimentally show that PAUSE achieves, and sometimes surpasses, state-of-the-art results using only a small fraction of labeled sentence pairs on various benchmark tasks. When applied to a real industrial use case where labeled samples are scarce, PAUSE encourages us to extend our dataset without the burden of extensive manual annotation work.

2. ECML-PKDD

   Simple, Scalable, and Stable Variational Deep Clustering

   Lele Cao, Sahar Asadi, Wenfei Zhu, Christian Schmidli, and Michael Sjöberg

   In Machine Learning and Knowledge Discovery in Databases, Nov 2021

   Abs DOI PDF Video Code

   

   Deep clustering (DC) has become the state-of-the-art for unsupervised clustering. In principle, DC represents a variety of unsupervised methods that jointly learn the underlying clusters and the latent representation directly from unstructured datasets. However, DC methods are generally poorly applied due to high operational costs, low scalability, and unstable results. In this paper, we first evaluate several popular DC variants in the context of industrial applicability using eight empirical criteria. We then choose to focus on variational deep clustering (VDC) methods, since they mostly meet those criteria except for simplicity, scalability, and stability. To address these three unmet criteria, we introduce four generic algorithmic improvements: initial }}\backslashgamma }}γ-training, periodic }}\backslashbeta }}β-annealing, mini-batch GMM (Gaussian mixture model) initialization, and inverse min-max transform. We also propose a novel clustering algorithm S3VDC (simple, scalable, and stable VDC) that incorporates all those improvements. Our experiments show that S3VDC outperforms the state-of-the-art on both benchmark tasks and a large unstructured industrial dataset without any ground truth label. In addition, we analytically evaluate the usability and interpretability of S3VDC.

2020

1. Journal

   Real-Time Recurrent Tactile Recognition: Momentum Batch-Sequential Echo State Networks

   Lele Cao, Fuchun Sun, Ramamohanarao Kotagiri, Wenbing Huang, Weihao Cheng, and Xiaolong Liu

   IEEE Transactions on Systems, Man, and Cybernetics: Systems, Nov 2020

   Abs DOI PDF

   

   Tactile recognition aims at identifying target objects according to tactile sensory readings. Tactile data have two salient properties: 1) sequentially real-time and 2) temporally correlated, which essentially calls for a real-time (i.e., online fixed-budget) and recurrent recognition procedure. Based on an efficient and robust spatio-temporal feature representation for tactile sequences, we handle the problem of real-time recurrent tactile recognition by proposing a bounded online-sequential learning framework, and incorporates the strength of batch-regularization bootstrapping, bounded recursive reservoir, and momentum-based estimation. Experimental evaluations show that it outperforms the state-of-the-art methods by a large margin on test accuracy; and its training performance is superior to most compare

2. RecSys Workshop

   Debiasing Few-Shot Recommendation in Mobile Games

   Lele Cao, Sahar Asadi, Matteo Biasielli, and Michael Sjöberg

   In RecSys Workshop on Online Recommender Systems and User Modeling (ORSUM), Nov 2020

   Abs PDF Website

   

   Mobile gaming has become increasingly popular due to the growing usage of smartphones in day to day life. In recent years, this advancement has led to an interest in the application of in-game recommendation systems. However, the in-game recommendation is more challenging than common recommendation scenarios, such as e-commerce, for a number of reasons:(1) the player behavior and context change at a fast pace,(2) only a few items (few-shot) can be exposed, and (3) with an existing hand-crafted heuristic recommendation, performing randomized explorations to collect data is not a business choice that is preferred by game stakeholders. To that end, we propose an end-to-end model called DFSNet (Debiasing Few-Shot Network) that enables training an in-game recommender on an imbalanced dataset that is biased by the existing heuristic policy. We experimentally evaluate the performance of DFSNet both in an offline setup on a validation dataset and online in a real-time serving environment, illustrating the correctness and effectiveness of the trained model.

3. CoG

   Use All Your Skills, Not Only The Most Popular Ones

   Francesco Lorenzo, Sahar Asadi, Alice Karnsund, Lele Cao, Tianze Wang, and Amir H. Payberah

   In 2020 IEEE Conference on Games (CoG), Nov 2020

   Abs DOI PDF

   

   Reinforcement Learning (RL) has shown promising results across various domains. However, applying it to develop gameplaying agents is challenging due to sparsity of extrinsic rewards, where agents get rewards from the environments only at the end of game levels. Previous works have shown that using intrinsic rewards is an effective way to deal with such cases. Intrinsic rewards allow to incorporate basic skills in agent policies to better generalize over various game levels. In a gameplay, it is common that certain actions (skills) are observed more often than others, which leads to a biased selection of actions. This problem boils down to a normalization issue in formulating the skill-based reward function. In this paper, we propose a novel solution to this problem by taking into account the frequency of all skills in the reward function. We show that our method improves the performance of agents by enabling them to select effective skills up to 2.5 times more frequently than that of the state-of-the-art in the context of the match-3 game Candy Crush Friends Saga.

2019

1. IJCNN

   Fine-Grained Road Mining from Satellite Images with Bilateral Xception and DeepLab

   Lele Cao

   In 2019 International Joint Conference on Neural Networks, Nov 2019

   Abs DOI PDF Code Website

   

   With the recent development of remote sensing and deep learning techniques, automatic and robust road extraction from satellite imaging data has become one of the most popular topics in both fields of Geographic Information System (GIS) and Computer Vision. Despite of the superior performance of Convolutional Neural Networks (DCNNs), a common problem of choosing between the classification and segmentation DCNNs still remains. By comparing two state-of-the-art baseline classification/segmentation DCNNs in several industrial application scenarios, we illustrate that their relative performance may vary, leading to different choices. Based on that observation, we propose a general fusion strategy that conveniently combines the strength of both classification and segmentation DCNNs using an end-to-end network architecture; this paradigm only requires pre-train segmentation/classification DCNNs once, which then can be reused in different road feature mining tasks. The task-specific experiments show that our fusion strategy guarantees superior results in all tested industrial scenarios.

2018

1. CoG

   Human-Like Playtesting with Deep Learning

   Stefan Freyr Gudmundsson, Philipp Eisen, Erik Poromaa, Alex Nodet, Sami Purmonen, Bartlomiej Kozakowski, Richard Meurling, and Lele Cao

   In 2018 IEEE Conference on Computational Intelligence and Games (CIG, n.k.a. CoG), Nov 2018

   Abs DOI PDF

   

   We present an approach to learn and deploy human-like playtesting in computer games based on deep learning from player data. We are able to learn and predict the most "human" action in a given position through supervised learning on a convolutional neural network. Furthermore, we show how we can use the learned network to predict key metrics of new content - most notably the difficulty of levels. Our player data and empirical data come from Candy Crush Saga (CCS) and Candy Crush Soda Saga (CCSS). However, the method is general and well suited for many games, in particular where content creation is sequential. CCS and CCSS are non-deterministic match-3 puzzle games with multiple game modes spread over a few thousand levels, providing a diverse testbed for this technique. Compared to Monte Carlo Tree Search (MCTS) we show that this approach increases correlation with average level difficulty, giving more accurate predictions as well as requiring only a fraction of the computation time.

2. Journal

   End-to-End ConvNet for Tactile Recognition Using Residual Orthogonal Tiling and Pyramid Convolution Ensemble

   Lele Cao, Fuchun Sun, Xiaolong Liu, Wenbing Huang, Ramamohanarao Kotagiri, and Hongbo Li

   Cognitive Computation, Nov 2018

   Abs DOI PDF

   

   Tactile recognition enables robots identify target objects or environments from tactile sensory readings. The recent advancement of deep learning and biological tactile sensing inspire us proposing an end-to-end architecture ROTConvPCE-mv that performs tactile recognition using residual orthogonal tiling and pyramid convolution ensemble. Our approach uses stacks of raw frames and tactile flow as dual input, and incorporates the strength of multi-layer OTConvs (orthogonal tiling convolutions) organized in a residual learning paradigm. We empirically demonstrate that OTConvs have adjustable invariance capability to different input transformations such as translation, rotation, and scaling. To effectively capture multi-scale global context, a pyramid convolution structure is attached to the concatenated output of two residual OTConv pathways. The extensive experimental evaluations show that ROTConvPCE-mv outperforms several state-of-the-art methods with a large margin regarding recognition accuracy, robustness, and fault-tolerance. Practical suggestions and hints are summarized throughout this paper to facilitate the effective recognition using tactile sensory data.

2017

1. ICONIP

   Fix-Budget and Recurrent Data Mining for Online Haptic Perception

   Lele Cao, Fuchun Sun, Xiaolong Liu, Wenbing Huang, Weihao Cheng, and Ramamohanarao Kotagiri

   In Neural Information Processing, Nov 2017

   Abs DOI

   Haptic perception is to identify different targets from haptic input. Haptic data have two prominent features: sequentially real-time and temporally correlated, which calls for a fixed-budget and recurrent perception procedure. Based on an efficient-robust spatio-temporal feature representation, we handle the problem with a bounded online-sequential learning framework (MBS-ESN), and incorporates the strength of batch-regularization bootstrapping, bounded recursive reservoir, and momentum-based estimation. Experimental evaluations show that it outperforms the state-of-the-art methods by a large margin on test accuracy; and its training performance is superior to most compared models from aspects of computational complexity and storage efficiency.

2. ITSC

   Benchmark for road marking detection: Dataset specification and performance baseline

   Xiaolong Liu, Zhidong Deng, Hongchao Lu, and Lele Cao

   In 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), Nov 2017

   Abs DOI PDF Code

   

   Detection of drivable road area and other critical objects like obstacles and landmarks in traffic scenes is fundamental to advanced driver assistance systems (ADAS) and self-driving car. Although scene parsing is able to make segmentation of road area from other objects and background, it basically does not get involved in recognizing other on-road markings. In fact, detection and classification of road area are only one small step towards true autonomous driving, because there are many categories of informative markings embodied within road area, such as lane markings, arrows, guiding lines, pedestrian crosswalks, and no-vehicle signs. If system identifies those markings, more information for both ADAS and self-driving car system can be provided. For this purpose, we release a benchmark dataset named TRoM (Tsinghua Road Marking), which is served for detection of 19 road-marking categories in urban scenarios. TRoM was built by means of over one-month data covering a full spectrum of time, weather, and traffic-load. An annotation toolkit was also presented to facilitate enriching such dataset. By directly applying our state-of-the-art method called RPP (ResNet with Pyramid Pooling), a reasonably accurate baseline on TRoM benchmark is made public for further performance comparison and evaluation.

3. Journal

   Advancing the incremental fusion of robotic sensory features using online multi-kernel extreme learning machine

   Lele Cao, Fuchun Sun, Hongbo Li, and Wenbing Huang

   Frontiers of Computer Science, Nov 2017

   Abs DOI PDF

   

   Robot recognition tasks usually require multiple homogeneous or heterogeneous sensors which intrinsically generate sequential, redundant, and storage demanding data with various noise pollution. Thus, online machine learning algorithms performing efficient sensory feature fusion have become a hot topic in robot recognition domain. This paper proposes an online multi-kernel extreme learning machine (OM-ELM) which assembles multiple ELM classifiers and optimizes the kernel weights with a p-norm formulation of multi-kernel learning (MKL) problem. It can be applied in feature fusion applications that require incremental learning over multiple sequential sensory readings. The performance of OM-ELM is tested towards four different robot recognition tasks. By comparing to several state-of-the-art online models for multi-kernel learning, we claim that our method achieves a superior or equivalent training accuracy and generalization ability with less training time. Practical suggestions are also given to aid effective online fusion of robot sensory features.

2016

1. AAAI

   Efficient spatio-temporal tactile object recognition with randomized tiling convolutional networks in a hierarchical fusion strategy

   Lele Cao, Ramamohanarao Kotagiri, Fuchun Sun, Hongbo Li, Wenbing Huang, and Zay Maung Maung Aye

   In Proceedings of the AAAI conference on artificial intelligence, Nov 2016

   Abs DOI PDF

   

   Robotic tactile recognition aims at identifying target objects or environments from tactile sensory readings. The advancement of unsupervised feature learning and biological tactile sensing inspire us proposing the model of 3T-RTCN that performs spatio-temporal feature representation and fusion for tactile recognition. It decomposes tactile data into spatial and temporal threads, and incorporates the strength of randomized tiling convolutional networks. Experimental evaluations show that it outperforms some state-of-the-art methods with a large margin regarding recognition accuracy, robustness, and fault-tolerance; we also achieve an order-of-magnitude speedup over equivalent networks with pretraining and finetuning. Practical suggestions and hints are summarized in the end for effectively handling the tactile data.

2. Journal

   Building feature space of extreme learning machine with sparse denoising stacked-autoencoder

   Le-le Cao, Wen-bing Huang, and Fu-chun Sun

   Neurocomputing, Nov 2016

   Abs DOI PDF

   

   The random-hidden-node extreme learning machine (ELM) is a much more generalized cluster of single-hidden-layer feed-forward neural networks (SLFNs) which has three parts: random projection, non-linear transformation, and ridge regression (RR) model. Networks with deep architectures have demonstrated state-of-the-art performance in a variety of settings, especially with computer vision tasks. Deep learning algorithms such as stacked autoencoder (SAE) and deep belief network (DBN) are built on learning several levels of representation of the input. Beyond simply learning features by stacking autoencoders (AE), there is a need for increasing its robustness to noise and reinforcing the sparsity of weights to make it easier to discover interesting and prominent features. The sparse AE and denoising AE was hence developed for this purpose. This paper proposes an approach: SSDAE-RR (stacked sparse denoising autoencoder – ridge regression) that effectively integrates the advantages in SAE, sparse AE, denoising AE, and the RR implementation in ELM algorithm. We conducted experimental study on real-world classification (binary and multiclass) and regression problems with different scales among several relevant approaches: SSDAE-RR, ELM, DBN, neural network (NN), and SAE. The performance analysis shows that the SSDAE-RR tends to achieve a better generalization ability on relatively large datasets (large sample size and high dimension) that were not pre-processed for feature abstraction. For 16 out of 18 tested datasets, the performance of SSDAE-RR is more stable than other tested approaches. We also note that the sparsity regularization and denoising mechanism seem to be mandatory for constructing interpretable feature representations. The fact that a SSDAE-RR approach often has a comparable training time to ELM makes it useful in some real applications.

3. IJCAI

   Learning stable linear dynamical systems with the weighted least square method

   Wenbing Huang, Lele Cao, Fuchun Sun, Deli Zhao, Huaping Liu, and Shanshan Yu

   In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence, New York, USA, Nov 2016

   Abs PDF Code Website

   

   Standard subspace algorithms learn Linear Dynamical Systems (LDSs) from time series with the least-square method, where the stability of the system is not naturally guaranteed. In this paper, we propose a novel approach for learning stable systems by enforcing stability directly on the least-square solutions. To this end, we first explore the spectral-radius property of the least-square transition matrix and then determine the key component that incurs the instability of the transition matrix. By multiplying the unstable component with a weight matrix on the right side, we obtain a weighted-least-square transition matrix that is further optimized to minimize the reconstruction error of the state sequence while still maintaining the stable constraint. Comparative experimental evaluations demonstrate that our proposed methods outperform the state-of-the-art methods regarding the reconstruction accuracy and the learning efficiency.

4. CVPR

   Sparse Coding and Dictionary Learning with Linear Dynamical Systems

   Wenbing Huang, Fuchun Sun, Lele Cao, Deli Zhao, Huaping Liu, and Mehrtash Harandi

   In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Nov 2016

   Abs DOI PDF Supp Video Website

   

   Linear Dynamical Systems (LDSs) are the fundamental tools for encoding spatio-temporal data in various disciplines. To enhance the performance of LDSs, in this paper, we address the challenging issue of performing sparse coding on the space of LDSs, where both data and dictionary atoms are LDSs. Rather than approximate the extended observability with a finite-order matrix, we represent the space of LDSs by an infinite Grassmannian consisting of the orthonormalized extended observability subspaces. Via a homeomorphic mapping, such Grassmannian is embedded into the space of symmetric matrices, where a tractable objective function can be derived for sparse coding. Then, we propose an efficient method to learn the system parameters of the dictionary atoms explicitly, by imposing the symmetric constraint to the transition matrices of the data and dictionary systems. Moreover, we combine the state covariance into the algorithm formulation, thus further promoting the performance of the models with symmetric transition matrices. Comparative experimental evaluations reveal the superior performance of proposed methods on various tasks including video classification and tactile recognition.

5. PAKDD

   A Precise and Robust Clustering Approach Using Homophilic Degrees of Graph Kernel

   Haolin Yang, Deli Zhao, Lele Cao, and Fuchun Sun

   In Advances in Knowledge Discovery and Data Mining, Nov 2016

   Abs DOI PDF

   

   To address the difficulties of "data noise sensitivity" and "cluster center variance" in mainstream clustering algorithms, we propose a novel robust approach for identifying cluster centers unambiguously from data contaminated with noise; it incorporates the strength of homophilic degrees and graph kernel. Exploiting that in-degrees can breed the homophilic distribution if ordered by their associated sorted out-degrees, it is easy to separate clusters from noise. Then we apply the diffusion kernel to the graph formed by clusters so as to obtain graph kernel matrix, which is treated as the measurement of global similarities. Based on local data densities and global similarities, the proposed approach manages to identify cluster centers precisely. Experiments on various synthetic and real-world databases verify the superiority of our algorithm in comparison with state-of-the-art algorithms.

6. IJCNN

   Tactile sequence based object categorization: A Bag of features modeled by Linear Dynamic System with Symmetric Transition Matrix

   Haolin Yang, Fuchun Sun, Wenbing Huang, Lele Cao, and Bin Fang

   In 2016 International Joint Conference on Neural Networks (IJCNN), Nov 2016

   Abs DOI PDF

   

   In this paper, we propose a novel categorization framework to recognize tactile sequences based on two particular properties of the tactile data. For the first one, tactile sequences are spatio-temporal data which is sequential and dynamic, depicting the process of grasping an object in different grasping stages; therefore, it is reasonable to discover the dynamical pattern by modeling tactile data as integral sequences rather than individual frames. For the second one, a tactile sequence contains various dynamical patterns in different stages of the grasping process; therefore, we decompose the whole sequence into multiple mini-sequences so as to enhance feature resolution. To address both properties in our framework, we take advantage of a Bag-of-System model using parameters of the Linear Dynamic System (LDS) as feature descriptors. Moreover, we employ the LDS with Symmetric Transition matrix (LDSST) rather than the original LDS as the building-block in order to obtain accurate codewords of the codebook of the Bag-of-System. The performance of our framework is evaluated on six real-world databases of three groups. Our experiments show that classification using LDSST is better than the original LDS, and the decomposition of tactile sequences does improve the accuracy of classification. The experiment results also show the superiority of our framework in comparison with other state-of-the-art sequence classifiers.

2015

1. Journal

   The quantitative and qualitative evaluation of a multi-agent microsimulation model for subway carriage design

   Le-le Cao, Xiao-xue Li, Fen-ni Kang, Chang Liu, Fu-chun Sun, and Ramamohanarao Kotagiri

   International Journal of Microsimulation (IJM), Dec 2015

   Abs DOI PDF

   

   Multi-agent microsimulation, as a third way of doing science other than induction and deduction methods, is explored to aid subway carriage design in this paper. Realizing that passenger behavior shapes the environment and in turn is shaped by the environment itself, we intend to model this interaction and examine the effectiveness and usability of the proposed model. We address our micro-model from essential aspects of environment space, agent attributes, agent behaviors, simulation process, and global objective/convergence function. Based on the real and simulated data, we evaluate our model with a combination of quantitative and qualitative procedures. For quantitative approach, we proposed two evaluation paradigms (i.e. “unified multinomial classifier” and “one-vs.-all binary classifiers”) using the state-of-the-art machine learning techniques and frameworks; and we manage to show from various perspectives that our model matches the reality in the majority of cases. For qualitative verification, we present a small-scale case study to evaluate different seat layouts in a subway carriage, and identify their advantages and disadvantages with little effort. By enriching microsimulation theory with innovative techniques, our research aims at promoting its acceptance level in design communities by means of avoiding costly creation of real-world experiments.

2. Conference

   A new slip-detection method based on pairwise high frequency components of capacitive sensor signals

   Haolin Yang, Xiaohui Hu, Lele Cao, and Fuchun Sun

   In 2015 International Conference on Information Science and Technology (ICIST), Dec 2015

   Abs DOI PDF

   

   In this paper, a novel method for slip detection using a capacitive sensor is proposed. We perform the Discrete Wavelet Transform (DWT) on the original signals of sensor. By comparing different wavelets, we find that the Haar wavelet is the most suitable to separate different frequency components. After performing the DWT by using the Haar wavelet, the separated high frequency components are pairwise due to properties of the Haar wavelet. Different from setting thresholds to detect object slip, our method detects slip by observing the variation trend of pairwise high frequency components. Meanwhile, we can distinguish signals of object loading and slip respectively. We carry out experiments on several objects with different surface properties and the results are consistent with our observations.

2014

1. ICPR

   Optimization-Based Extreme Learning Machine with Multi-kernel Learning Approach for Classification

   Le-le Cao, Wen-bing Huang, and Fu-chun Sun

   In 2014 22nd International Conference on Pattern Recognition, Dec 2014

   Abs DOI PDF

   

   The optimization method based extreme learning machine (optimization-based ELM) is generalized from single-hidden-layer feed-forward neural networks (SLFNs) by making use of kernels instead of neuron-alike hidden nodes. This approach is known for its high scalability, low computational complexity, and mild optimization constrains. The multi-kernel learning (MKL) framework Simple MKL iteratively determines the combination of kernels by gradient descent wrapping a standard support vector machine (SVM) solver. Simple MKL can be applied to many kinds of supervised learning problems to receive a more stable performance with rapid convergence speed. This paper proposes a new approach: MK-ELM (multi-kernel extreme learning machine) that applies Simple MKL framework to the optimization-based ELM algorithm. The performance analysis on binary classification problems with various scales shows that MK-ELM tends to achieve the best generalization performance as well as being the most insensitive to parameters comparing to optimization-based ELM and Simple MKL. As a result, MK-ELM can be implemented in real applications easily.

2. Conference

   Optimization-based multikernel extreme learning for multimodal object image classification

   Le-le Cao, Wen-bing Huang, and Fu-chun Sun

   In 2014 International Conference on Multisensor Fusion and Information Integration for Intelligent Systems (MFI), Dec 2014

   Abs DOI PDF

   

   This paper is concerned with multi-kernel extreme learning machine (MK-ELM) which adapts the multi-kernel learning (MKL) framework to extreme learning machine (ELM). MK-ELM approach iteratively determines the combination of kernels by gradient descent wrapping a standard optimization method based ELM. Such MKL methods are very useful in information fusion research and applications. MK-ELM’s performance on object image classification via multimodal feature (visual and textual) fusion is experimented and studied. By comparing to other widely used fusion methods (i.e. SVM-based SimpleMKL, feature concatenation, and decision fusion), several advantages and characteristics of MK-ELM fusion are revealed and discussed showing MK-ELM is an easy and effective approach to implement in object image classification applications.