Recently, deep learning (DL) algorithms have become one of the most promising methods in biomedical research. Although DL has the strength of outstanding predictive performance by capturing nonlinear and hierarchical representations through multi-layered neural network architectures, the most common structure of fully-connected neural networks lacks in interpreting what the hierarchical representation features describe in biology, since the high-level representations are products involving all features. I and my research team have focused on developing novel deep learning methodologies which are biologically interpretable as new domain knowledge and help ones to discover new patterns that may explain complex systems. In this talk, I will present three research works of interpretable and integrative deep learning. Firstly, I will present Pathway-based Sparse Neural Network (PASNet) that predicts clinical outcomes of cancer patients as well as describing biological mechanism in cancer. Secondly, I will present Pathological image and Genomic data integrative deep learning (PAGE-Net) that predict survivals by integrating pathological images and genomic data together. Finally, I will demonstrate a pathway-based multi-omics data analysis model using convolutional neural networks (PathCNN).