This book provides a series of gesture and behavior recognition methods based on multimodal data representation. The data modalities include image data and skeleton data, andthe modeling methods include traditional codebook, topological graph, and LSTM architectures. The tasks include single gesture recognition classification, single action recognition classification, continuous gesture classification, complex behavior classification of human interaction and other tasks of different complexity. This book focuses on the data processing methods of each modality, and the modeling methods for different tasks. We hope thereadercan learn basic gesture and action recognition methods from this book, and develop a model system that suits their needs on this basis. This book can be used as a textbook for graduate, postgraduate and PhD students majoring in computer science, automation, etc. It can also be used as a reference for the reader who is interested in gesture recognition, human action interaction, sequence data processing, and deep neural network design, and who hopes to contribute to the fields.