This repository hosts the code, experiments, and analyses for the independent study "Tracing the Evolution of Neural Machine Learning: From Artificial Neurons to Transformers and Beyond." Conducted as part of CS 496 in Spring 2025 under the guidance of Professor Jaime Dávila by Aryan Sajith, this project explores the evolution of neural network architectures—from the perceptron to modern models like Transformers and emerging state-space alternatives such as Mamba.
This research project explores the evolution of neural network architectures in response to the challenges faced in machine learning. By tracing the progression from early models like perceptrons to modern architectures such as CNNs, RNNs, Transformers, and emerging alternatives like Mamba, the study aims to uncover the motivations behind each design, analyze their underlying mechanics, and understand their respective trade-offs and strengths in various ML contexts.
- Core Foundations:
Implementation and comparative analysis of early neural architectures including the Perceptron and Multilayer Perceptron (MLP). - Deep Learning Transition:
Exploration of Convolutional Neural Networks (CNNs) and their revolutionary impact on computer vision tasks. - Sequence Modeling:
Study and implementation of Recurrent Neural Networks (RNNs), LSTMs, and analysis of their limitations with sequential data. - Transformers and Alternatives:
Implementation of a minimal Transformer model alongside an investigation into alternative architectures like Mamba. - Final Synthesis:
Comprehensive cross-comparative testing, culminating in a structured research paper that connects historical insights with practical performance metrics.
- code/ - Source code for all neural network implementations.
- output/ - Contains outputs from testing models during the course of this study.
- paper/ - Drafts and the final research paper summarizing findings.
- docs/ - Contains important documents like the project proposal, notes on research papers, and more.
- README.md - This primary file which outlines the details of the independent study.
TBD - Will be updated as the project gets going