Obtaining the ontological account of an image numerically can be earned via a Siamese network. The anatomy of this network has a twin architecture, consisting of convolutional and fully connected layers with shared weights. Each architecture digests an image and yields the vector embedding (the ontological or latent representation) of that image. These two vectors are then subjected to the Euclidean distance calculation. Next, the result is funneled to the last fully connected layer to get the logit describing their similarity. To learn the representation, here, we can leverage contrastive loss as our objective function to be optimized. The network is trained on paired images, i.e., positive and negative. In this project, the positive pairs are two images that belong to the same dataset, and the negative pairs are two images from distinct datasets. Here, subsets of the MedMNIST dataset are utilized: DermaMNIST, PneumoniaMNIST, RetinaMNIST, and BreastMNIST. Then, accuracy is used to evaluate the trained network. Afterward, we encode all images of the train and validation sets into embedding vectors and store them in the PostgreSQL database. So, sometimes later, we can use the embedding vectors to retrieve similar images based on a query image (we can obtain it from the test set). To find similar images, FAISS (Facebook AI Similarity Search) is employed. FAISS helps us seek the closest vectors to the query vector.
This notebook shows the overall code of this project.
The loss and accuracy of the test set are exhibited quantitatively below:
| Test Metric | Score |
|---|---|
| Loss | 0.012 |
| Accuracy | 98.58% |
The loss curve on the train and validation sets of the Siamese network.
The accuracy curve on the train and validation sets of the Siamese network.
The following picture presents four similar images contingent upon a query image of four different datasets, i.e., DermaMNIST, PneumoniaMNIST, RetinaMNIST, and BreastMNIST.
The image similarity search results for DermaMNIST (first row), PneumoniaMNIST (second row), RetinaMNIST (third row), and BreastMNIST (fourth row).
- Image Similarity Estimation Using a Siamese Network With a Contrastive Loss
- Near-Duplicate Image Search
- Signature Verification using a "Siamese" Time Delay Neural Network
- Neural Networks for Fingerprint Recognition
- Contrastive Representation Learning
- Learning a Similarity Metric Discriminatively, with Application to Face Verification
- Python-Colab-Postgres
- Leveraging Google Colab to run Postgres: A Comprehensive Guide
- Using PostgreSQL in Python
- Introduction to Facebook AI Similarity Search (Faiss)
- MedMNIST v2 - A large-scale lightweight benchmark for 2D and 3D biomedical image classification
- PyTorch Lightning