Book Description

In the era of AI, statistical or machine learning methods towards distribution-free assumptions are becoming increasingly important due to the growing amount of data that is being collected and analyzed. Traditional parametric methods may not always be appropriate or may lead to model mis-specification and inaccurate results when dealing with large or complex data sets. Besides, as specific distributional assumptions or parametric modeling are removed, the challenge of model interpretation and prediction inference arises and has been currently at the forefront of research efforts. One problem of our interests in this regard is non-parametric or semi-parametric network estimation for data that are not independent. Specifically, influence network estimation from a multi-variate point process or time series data is a problem of fundamental importance. Prior work has focused on parametric approaches that require a known parametric model, which makes estimation procedures less robust to model mis-specification, non-linearities and heterogeneities. In Chapter 2, we develop a semi-parametric approach based on the monotone single-index multi-variate autoregressive model (SIMAM) which addresses these challenges. In particular, rather than using standard parametric approaches, we use the monotone single index model (SIM) for network estimation. We provide theoretical guarantees for dependent data, and an alternating projected gradient descent algorithm. Significantly we achieve rates of the form O(T^{-1/3} \sqrt{s\log(TM)}) (optimal in the independent design case) where s is {he number of edges in the influence network that indicates the sparsity level, M is the number of actors and T is the number of time points. In addition, we demonstrate the performance of SIMAM both on simulated data and two real data examples, and show it outperforms state-of-the-art parametric methods both in terms of prediction and network estimation. Another aspect important for distribution-free or model-free learning is the interpretation, i.e. to make the complicated non-parametric predictive models explainable. A number of model-agnostic methods for measuring variable importance (VI) have emerged in recent times, which assess the difference in predictive power between a full model trained on all variables and a reduced model that omits the variable(s) of interest. However, these methods typically encounter a bottleneck when estimating the reduced model for each variable or subset of variables, which is both costly and lacks theoretical guarantees. To address this problem, Chapter 3 proposes an efficient and adaptable approach for approximating the reduced model while ensuring important inferential guarantees. Specifically, we replace the need for fully retraining a wide neural network with a linearization that is initiated using the full model parameters. By including a ridge-like penalty to make the problem convex, we establish that our method can estimate the variable importance measure with an error rate of O({1}/{\sqrt{n}), where n represents the number of training samples, provided that the ridge penalty parameter is adequately large. Furthermore, we demonstrate that our estimator is asymptotically normal, enabling us to provide confidence bounds for the VI estimates. Finally, we demonstrate the method's speed and accuracy under different data-generating regimes and showcase its applicability in a real-world seasonal climate forecasting example. In addition to semi-parametric network estimation and fast estimation of variable importance for interpretation, an efficient method for prediction inference without specific distributional assumptions on the data is of our interest as well. In Chapter 4, we present a novel, computationally-efficient algorithm for predictive inference (PI) that requires no distributional assumptions in the data and can be computed faster than existing bootstrap-type methods for neural networks. Specifically, if there are $n$ training samples, bootstrap methods require training a model on each of the n subsamples of size n-1; for large models like neural networks, this process can be computationally prohibitive. In contrast, the proposed method trains one neural network on the full dataset with ([epsilon], [delta]) -differential privacy (DP) and then approximates each leave-one-out model efficiently using a linear approximation around the neural network estimate. With exchangeable data, we prove that our approach has a rigorous coverage guarantee that depends on the preset privacy parameters and the stability of the neural network, regardless of the data distribution. Simulations and experiments on real data demonstrate that our method satisfies the coverage guarantees with substantially reduced computation compared to bootstrap methods.