In an era of complex deep learning architectures like transformers CNNs and LSTM cells the
challenge persists: the hunger for labeled data and high energy. This dissertation explores
Echo State Network (ESN) an RNN variant. ESN's efficiency in linear regression training and
simplicity suggest pathways to resource-efficient adaptable deep learning. Systematically
deconstructing ESN architecture into flexible modules it introduces basic ESN models with
random weights and efficient deterministic ESN models as baselines. Diverse unsupervised
pre-training methods for ESN components are evaluated against these baselines. Rigorous
benchmarking across datasets - time-series classification audio recognition - shows
competitive performance of ESN models with state-of-the-art approaches. Identified nuanced use
cases guiding model preferences and limitations in training methods highlight the importance of
proposed ESN models in bridging reservoir computing and deep learning.
