Skip to content
This repository was archived by the owner on Oct 31, 2021. It is now read-only.

Latest commit

 

History

History
18 lines (14 loc) · 806 Bytes

README.md

File metadata and controls

18 lines (14 loc) · 806 Bytes

PyTorch Highway Networks

Highway networks implemented in PyTorch.

Highway Equation

Just the MNIST example from PyTorch hacked to work with Highway layers.

Todo

  • Make the Highway nn.Module reuseable and configurable.
  • Why does softmax work better than sigmoid? This shouldn't be the case...
  • Make training graphs on the MNIST dataset.
  • Add convolutional highway networks.
  • Add recurrent highway networks.
  • Experiment with convolutional highway networks for character embeddings.

Notes

  • ELU doesn't work better than RELU for the layer activation.
  • Softmax seems to work better than sigmoid for the gate function?!