visualize.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Visual Q&A Demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Setup\n",
    "1-Installs required packages   \n",
    "2-Downloads sample data and a pretrained model.     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "using Pkg; Pkg.activate(\".\"); Pkg.instantiate(); #install required packages\n",
    "include(\"demosetup.jl\") \n",
    "include(\"src/main.jl\")\n",
    "include(\"src/rnnfix.jl\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialization\n",
    "1-Loads the sample demo data (image features,questions,vocabulary).   \n",
    "    * w2i : words to one-hot index   : w2i[\"cube\"] = 30  \n",
    "    * a2i : answer to one-hot index  : a2i[\"gray\"] = 8\n",
    "    * i2w : one-hot index to words   : i2w[2] = \"Are\"   \n",
    "    * i2a : one-hot index to answers : i2a[5] = \"large\"   \n",
    "2-Loads the pretrained model, and its hyper-parameters `o`.     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "feats,qstsns,(w2i,a2i,i2w,i2a) = loadDemoData(\"data/demo/\");\n",
    "_,Mrun,o = loadmodel(\"models/macnet2.jld2\";onlywrun=true);\n",
    "value(params(Mrun)[1]) isa Array ? KnetLayers.settype!(Array) : KnetLayers.settype!(KnetArray)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sample Data\n",
    "1-Randomly selects (question,image) pair from the sample data   \n",
    "2-Make predictions for the question and checks whether the prediction is correct   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rnd        = 41 # rand(1:length(qstsns))\n",
    "inst       = qstsns[rnd]\n",
    "feat       = arrtype(feats[:,:,:,rnd:rnd])\n",
    "question   = Array{Int}(inst[2])\n",
    "answer     = inst[3];\n",
    "family     = inst[4];\n",
    "results,prediction,interoutputs = singlerun(Mrun,feat,question;p=o[:p],selfattn=o[:selfattn],gating=o[:gating]);\n",
    "answer==prediction[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "interoutputs,prediction[1],answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "img = load(\"data/demo/CLEVR_v1.0/images/val/$(inst[1])\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "textq  = i2w[question];\n",
    "println(\"Question: \",join(textq,\" \"))\n",
    "texta  = i2a[answer];\n",
    "println(\"Answer: $(texta)\\nPrediction: $(i2a[prediction]) \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## User Data\n",
    "You can enter your own question about the image and test whether the prediction is correct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "userinput = readline(stdin)\n",
    "words = split(userinput) # tokenize(userinput)\n",
    "question = [get!(w2i,wr,1) for wr in words]\n",
    "results,prediction = singlerun(Mrun,feat,question;p=o[:p],selfattn=o[:selfattn],gating=o[:gating]);\n",
    "println(\"Question: $(join(i2w[question],\" \"))\")\n",
    "println(\"Prediction: $(i2a[prediction])\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualize\n",
    "`visualize` function visualizes attention maps for each time step of the mac network"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "visualize(img,results;p=o[:p])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "@webio": {
   "lastCommId": null,
   "lastKernelId": null
  },
  "kernelspec": {
   "display_name": "Julia 1.3.1",
   "language": "julia",
   "name": "julia-1.3"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "1.3.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}