tensorrt_mistral.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a1e8128a",
   "metadata": {},
   "source": [
    "# Run inference on Mistral 7B using NVIDIA TensorRT-LLM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "05d9aa81",
   "metadata": {},
   "source": [
    "Welcome!\n",
    "\n",
    "In this notebook, we will walk through on converting mistral into the TensorRT format. TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM was recently featured in the Phind-70B release as their preferred framework for performing inference! \n",
    "\n",
    "See the [Github repo](https://github.com/NVIDIA/TensorRT-LLM) for more examples and documentation!\n",
    "\n",
    "A note about running Jupyter Notebooks: Press Shift + Enter to run a cell. A * in the left-hand cell box means the cell is running. A number means it has completed. If your Notebook is acting weird, you can interrupt a too-long process by interrupting the kernel (Kernel tab -> Interrupt Kernel) or even restarting the kernel (Kernel tab -> Restart Kernel). Note restarting the kernel will require you to run everything from the beginning.\n",
    "\n",
    "Deployment powered by [Brev.dev](https://x.com/brevdev) 🤙\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1a32d22-5e81-4124-8e29-96ce7b0c1f50",
   "metadata": {},
   "source": [
    "#### Step 1 - Install TensorRT-LLM"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e3ce905",
   "metadata": {},
   "source": [
    "We first install TensorRT-LLM and some additional packages that are using during the conversion process"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "866149c7-882e-4728-86cf-29b46b6325ab",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pypi.nvidia.com\n",
      "Collecting tensorrt_llm\n",
      "  Downloading https://pypi.nvidia.com/tensorrt-llm/tensorrt_llm-0.9.0.dev2024022000-cp310-cp310-linux_x86_64.whl (1229.9 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 GB\u001b[0m \u001b[31m933.9 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting accelerate==0.25.0 (from tensorrt_llm)\n",
      "  Downloading accelerate-0.25.0-py3-none-any.whl.metadata (18 kB)\n",
      "Collecting build (from tensorrt_llm)\n",
      "  Downloading build-1.0.3-py3-none-any.whl.metadata (4.2 kB)\n",
      "Collecting colored (from tensorrt_llm)\n",
      "  Downloading colored-2.2.4-py3-none-any.whl.metadata (3.6 kB)\n",
      "Collecting cuda-python (from tensorrt_llm)\n",
      "  Downloading cuda_python-12.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
      "Collecting diffusers==0.15.0 (from tensorrt_llm)\n",
      "  Downloading diffusers-0.15.0-py3-none-any.whl.metadata (19 kB)\n",
      "Collecting lark (from tensorrt_llm)\n",
      "  Downloading lark-1.1.9-py3-none-any.whl.metadata (1.9 kB)\n",
      "Collecting mpi4py (from tensorrt_llm)\n",
      "  Downloading mpi4py-3.1.5.tar.gz (2.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.5/2.5 MB\u001b[0m \u001b[31m11.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25h  Installing build dependencies ... \u001b[?25ldone\n",
      "\u001b[?25h  Getting requirements to build wheel ... \u001b[?25ldone\n",
      "\u001b[?25h  Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from tensorrt_llm) (1.23.5)\n",
      "Collecting onnx>=1.12.0 (from tensorrt_llm)\n",
      "  Downloading onnx-1.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (15 kB)\n",
      "Requirement already satisfied: polygraphy in /usr/local/lib/python3.10/dist-packages (from tensorrt_llm) (0.49.4)\n",
      "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from tensorrt_llm) (5.9.8)\n",
      "Collecting pynvml>=11.5.0 (from tensorrt_llm)\n",
      "  Downloading pynvml-11.5.0-py3-none-any.whl.metadata (7.8 kB)\n",
      "Collecting sentencepiece>=0.1.99 (from tensorrt_llm)\n",
      "  Downloading sentencepiece-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.7 kB)\n",
      "Collecting tensorrt==9.2.0.post12.dev5 (from tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/tensorrt/tensorrt-9.2.0.post12.dev5.tar.gz (18 kB)\n",
      "  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hCollecting torch<=2.2.0a (from tensorrt_llm)\n",
      "  Downloading torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl.metadata (25 kB)\n",
      "Collecting transformers==4.36.1 (from tensorrt_llm)\n",
      "  Downloading transformers-4.36.1-py3-none-any.whl.metadata (126 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m126.8/126.8 kB\u001b[0m \u001b[31m20.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: wheel in /usr/lib/python3/dist-packages (from tensorrt_llm) (0.37.1)\n",
      "Collecting optimum (from tensorrt_llm)\n",
      "  Downloading optimum-1.17.1-py3-none-any.whl.metadata (18 kB)\n",
      "Collecting evaluate (from tensorrt_llm)\n",
      "  Downloading evaluate-0.4.1-py3-none-any.whl.metadata (9.4 kB)\n",
      "Collecting janus (from tensorrt_llm)\n",
      "  Downloading janus-1.0.0-py3-none-any.whl.metadata (4.5 kB)\n",
      "Collecting nvidia-ammo~=0.7.0 (from tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-ammo/nvidia_ammo-0.7.3-cp310-cp310-linux_x86_64.whl (976 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m976.1/976.1 kB\u001b[0m \u001b[31m73.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.0->tensorrt_llm) (23.2)\n",
      "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate==0.25.0->tensorrt_llm) (6.0.1)\n",
      "Collecting huggingface-hub (from accelerate==0.25.0->tensorrt_llm)\n",
      "  Downloading huggingface_hub-0.20.3-py3-none-any.whl.metadata (12 kB)\n",
      "Collecting safetensors>=0.3.1 (from accelerate==0.25.0->tensorrt_llm)\n",
      "  Downloading safetensors-0.4.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (3.8 kB)\n",
      "Requirement already satisfied: Pillow in /usr/local/lib/python3.10/dist-packages (from diffusers==0.15.0->tensorrt_llm) (10.2.0)\n",
      "Collecting filelock (from diffusers==0.15.0->tensorrt_llm)\n",
      "  Downloading filelock-3.13.1-py3-none-any.whl.metadata (2.8 kB)\n",
      "Collecting importlib-metadata (from diffusers==0.15.0->tensorrt_llm)\n",
      "  Downloading importlib_metadata-7.0.1-py3-none-any.whl.metadata (4.9 kB)\n",
      "Collecting regex!=2019.12.17 (from diffusers==0.15.0->tensorrt_llm)\n",
      "  Downloading regex-2023.12.25-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (40 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.9/40.9 kB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hRequirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from diffusers==0.15.0->tensorrt_llm) (2.31.0)\n",
      "Collecting tensorrt_bindings==9.2.0.post12.dev5 (from tensorrt==9.2.0.post12.dev5->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/tensorrt-bindings/tensorrt_bindings-9.2.0.post12.dev5-cp310-none-manylinux_2_17_x86_64.whl (1.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m69.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting tensorrt_libs==9.2.0.post12.dev5 (from tensorrt==9.2.0.post12.dev5->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/tensorrt-libs/tensorrt_libs-9.2.0.post12.dev5-py2.py3-none-manylinux_2_17_x86_64.whl (1076.3 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.1/1.1 GB\u001b[0m \u001b[31m1.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m0:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting tokenizers<0.19,>=0.14 (from transformers==4.36.1->tensorrt_llm)\n",
      "  Downloading tokenizers-0.15.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (6.7 kB)\n",
      "Collecting tqdm>=4.27 (from transformers==4.36.1->tensorrt_llm)\n",
      "  Downloading tqdm-4.66.2-py3-none-any.whl.metadata (57 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m57.6/57.6 kB\u001b[0m \u001b[31m396.5 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cuda-runtime-cu12 (from tensorrt_libs==9.2.0.post12.dev5->tensorrt==9.2.0.post12.dev5->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-cuda-runtime-cu12/nvidia_cuda_runtime_cu12-12.3.101-py3-none-manylinux1_x86_64.whl (867 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m867.7/867.7 kB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cudnn-cu12 (from tensorrt_libs==9.2.0.post12.dev5->tensorrt==9.2.0.post12.dev5->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-cudnn-cu12/nvidia_cudnn_cu12-8.9.7.29-py3-none-manylinux1_x86_64.whl (704.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m704.7/704.7 MB\u001b[0m \u001b[31m1.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cublas-cu12 (from tensorrt_libs==9.2.0.post12.dev5->tensorrt==9.2.0.post12.dev5->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-cublas-cu12/nvidia_cublas_cu12-12.3.4.1-py3-none-manylinux1_x86_64.whl (412.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m412.6/412.6 MB\u001b[0m \u001b[31m4.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting ninja (from nvidia-ammo~=0.7.0->tensorrt_llm)\n",
      "  Downloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl.metadata (5.3 kB)\n",
      "Collecting networkx (from nvidia-ammo~=0.7.0->tensorrt_llm)\n",
      "  Downloading networkx-3.2.1-py3-none-any.whl.metadata (5.2 kB)\n",
      "Collecting onnxruntime~=1.16.1 (from nvidia-ammo~=0.7.0->tensorrt_llm)\n",
      "  Downloading onnxruntime-1.16.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.3 kB)\n",
      "Collecting onnx-graphsurgeon (from nvidia-ammo~=0.7.0->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/onnx-graphsurgeon/onnx_graphsurgeon-0.3.25-py2.py3-none-any.whl (40 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m40.9/40.9 kB\u001b[0m \u001b[31m5.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting scipy (from nvidia-ammo~=0.7.0->tensorrt_llm)\n",
      "  Downloading scipy-1.12.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (60 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m60.4/60.4 kB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting torchprofile>=0.0.4 (from nvidia-ammo~=0.7.0->tensorrt_llm)\n",
      "  Downloading torchprofile-0.0.4-py3-none-any.whl.metadata (303 bytes)\n",
      "Requirement already satisfied: protobuf>=3.20.2 in /usr/local/lib/python3.10/dist-packages (from onnx>=1.12.0->tensorrt_llm) (4.25.1)\n",
      "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch<=2.2.0a->tensorrt_llm) (4.9.0)\n",
      "Collecting sympy (from torch<=2.2.0a->tensorrt_llm)\n",
      "  Downloading sympy-1.12-py3-none-any.whl.metadata (12 kB)\n",
      "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch<=2.2.0a->tensorrt_llm) (3.1.3)\n",
      "Collecting fsspec (from torch<=2.2.0a->tensorrt_llm)\n",
      "  Downloading fsspec-2024.2.0-py3-none-any.whl.metadata (6.8 kB)\n",
      "Collecting nvidia-cuda-nvrtc-cu12==12.1.105 (from torch<=2.2.0a->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-cuda-nvrtc-cu12/nvidia_cuda_nvrtc_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (23.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m23.7/23.7 MB\u001b[0m \u001b[31m74.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cuda-runtime-cu12 (from tensorrt_libs==9.2.0.post12.dev5->tensorrt==9.2.0.post12.dev5->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-cuda-runtime-cu12/nvidia_cuda_runtime_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (823 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m823.6/823.6 kB\u001b[0m \u001b[31m72.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cuda-cupti-cu12==12.1.105 (from torch<=2.2.0a->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-cuda-cupti-cu12/nvidia_cuda_cupti_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (14.1 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m14.1/14.1 MB\u001b[0m \u001b[31m104.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cudnn-cu12 (from tensorrt_libs==9.2.0.post12.dev5->tensorrt==9.2.0.post12.dev5->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-cudnn-cu12/nvidia_cudnn_cu12-8.9.2.26-py3-none-manylinux1_x86_64.whl (731.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m731.7/731.7 MB\u001b[0m \u001b[31m1.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cublas-cu12 (from tensorrt_libs==9.2.0.post12.dev5->tensorrt==9.2.0.post12.dev5->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-cublas-cu12/nvidia_cublas_cu12-12.1.3.1-py3-none-manylinux1_x86_64.whl (410.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m410.6/410.6 MB\u001b[0m \u001b[31m2.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cufft-cu12==11.0.2.54 (from torch<=2.2.0a->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-cufft-cu12/nvidia_cufft_cu12-11.0.2.54-py3-none-manylinux1_x86_64.whl (121.6 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m121.6/121.6 MB\u001b[0m \u001b[31m5.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-curand-cu12==10.3.2.106 (from torch<=2.2.0a->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-curand-cu12/nvidia_curand_cu12-10.3.2.106-py3-none-manylinux1_x86_64.whl (56.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m56.5/56.5 MB\u001b[0m \u001b[31m11.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cusolver-cu12==11.4.5.107 (from torch<=2.2.0a->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-cusolver-cu12/nvidia_cusolver_cu12-11.4.5.107-py3-none-manylinux1_x86_64.whl (124.2 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m124.2/124.2 MB\u001b[0m \u001b[31m6.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-cusparse-cu12==12.1.0.106 (from torch<=2.2.0a->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-cusparse-cu12/nvidia_cusparse_cu12-12.1.0.106-py3-none-manylinux1_x86_64.whl (196.0 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m196.0/196.0 MB\u001b[0m \u001b[31m4.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-nccl-cu12==2.18.1 (from torch<=2.2.0a->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-nccl-cu12/nvidia_nccl_cu12-2.18.1-py3-none-manylinux1_x86_64.whl (209.7 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m209.7/209.7 MB\u001b[0m \u001b[31m4.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting nvidia-nvtx-cu12==12.1.105 (from torch<=2.2.0a->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-nvtx-cu12/nvidia_nvtx_cu12-12.1.105-py3-none-manylinux1_x86_64.whl (99 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m99.1/99.1 kB\u001b[0m \u001b[31m510.1 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hCollecting triton==2.1.0 (from torch<=2.2.0a->tensorrt_llm)\n",
      "  Downloading triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl.metadata (1.3 kB)\n",
      "Collecting nvidia-nvjitlink-cu12 (from nvidia-cusolver-cu12==11.4.5.107->torch<=2.2.0a->tensorrt_llm)\n",
      "  Downloading https://pypi.nvidia.com/nvidia-nvjitlink-cu12/nvidia_nvjitlink_cu12-12.3.101-py3-none-manylinux1_x86_64.whl (20.5 MB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m20.5/20.5 MB\u001b[0m \u001b[31m33.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hCollecting pyproject_hooks (from build->tensorrt_llm)\n",
      "  Downloading pyproject_hooks-1.0.0-py3-none-any.whl.metadata (1.3 kB)\n",
      "Requirement already satisfied: tomli>=1.1.0 in /usr/local/lib/python3.10/dist-packages (from build->tensorrt_llm) (2.0.1)\n",
      "Collecting datasets>=2.0.0 (from evaluate->tensorrt_llm)\n",
      "  Downloading datasets-2.17.1-py3-none-any.whl.metadata (20 kB)\n",
      "Collecting dill (from evaluate->tensorrt_llm)\n",
      "  Downloading dill-0.3.8-py3-none-any.whl.metadata (10 kB)\n",
      "Collecting pandas (from evaluate->tensorrt_llm)\n",
      "  Downloading pandas-2.2.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (19 kB)\n",
      "Collecting xxhash (from evaluate->tensorrt_llm)\n",
      "  Downloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
      "Collecting multiprocess (from evaluate->tensorrt_llm)\n",
      "  Downloading multiprocess-0.70.16-py310-none-any.whl.metadata (7.2 kB)\n",
      "Collecting responses<0.19 (from evaluate->tensorrt_llm)\n",
      "  Downloading responses-0.18.0-py3-none-any.whl.metadata (29 kB)\n",
      "Collecting coloredlogs (from optimum->tensorrt_llm)\n",
      "  Downloading coloredlogs-15.0.1-py2.py3-none-any.whl.metadata (12 kB)\n",
      "Collecting pyarrow>=12.0.0 (from datasets>=2.0.0->evaluate->tensorrt_llm)\n",
      "  Downloading pyarrow-15.0.0-cp310-cp310-manylinux_2_28_x86_64.whl.metadata (3.0 kB)\n",
      "Collecting pyarrow-hotfix (from datasets>=2.0.0->evaluate->tensorrt_llm)\n",
      "  Downloading pyarrow_hotfix-0.6-py3-none-any.whl.metadata (3.6 kB)\n",
      "Collecting fsspec (from torch<=2.2.0a->tensorrt_llm)\n",
      "  Downloading fsspec-2023.10.0-py3-none-any.whl.metadata (6.8 kB)\n",
      "Collecting aiohttp (from datasets>=2.0.0->evaluate->tensorrt_llm)\n",
      "  Downloading aiohttp-3.9.4rc0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (7.4 kB)\n",
      "Collecting flatbuffers (from onnxruntime~=1.16.1->nvidia-ammo~=0.7.0->tensorrt_llm)\n",
      "  Downloading flatbuffers-23.5.26-py2.py3-none-any.whl.metadata (850 bytes)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.10/dist-packages (from requests->diffusers==0.15.0->tensorrt_llm) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->diffusers==0.15.0->tensorrt_llm) (3.6)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->diffusers==0.15.0->tensorrt_llm) (2.2.1)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->diffusers==0.15.0->tensorrt_llm) (2024.2.2)\n",
      "Collecting torchvision>=0.4 (from torchprofile>=0.0.4->nvidia-ammo~=0.7.0->tensorrt_llm)\n",
      "  Downloading torchvision-0.17.1-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)\n",
      "Collecting humanfriendly>=9.1 (from coloredlogs->optimum->tensorrt_llm)\n",
      "  Downloading humanfriendly-10.0-py2.py3-none-any.whl.metadata (9.2 kB)\n",
      "Collecting zipp>=0.5 (from importlib-metadata->diffusers==0.15.0->tensorrt_llm)\n",
      "  Downloading zipp-3.17.0-py3-none-any.whl.metadata (3.7 kB)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch<=2.2.0a->tensorrt_llm) (2.1.3)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /usr/local/lib/python3.10/dist-packages (from pandas->evaluate->tensorrt_llm) (2.8.2)\n",
      "Collecting pytz>=2020.1 (from pandas->evaluate->tensorrt_llm)\n",
      "  Downloading pytz-2024.1-py2.py3-none-any.whl.metadata (22 kB)\n",
      "Collecting tzdata>=2022.7 (from pandas->evaluate->tensorrt_llm)\n",
      "  Downloading tzdata-2024.1-py2.py3-none-any.whl.metadata (1.4 kB)\n",
      "Collecting mpmath>=0.19 (from sympy->torch<=2.2.0a->tensorrt_llm)\n",
      "  Downloading mpmath-1.4.0a0-py3-none-any.whl.metadata (8.7 kB)\n",
      "Collecting aiosignal>=1.1.2 (from aiohttp->datasets>=2.0.0->evaluate->tensorrt_llm)\n",
      "  Downloading aiosignal-1.3.1-py3-none-any.whl.metadata (4.0 kB)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets>=2.0.0->evaluate->tensorrt_llm) (23.2.0)\n",
      "Collecting frozenlist>=1.1.1 (from aiohttp->datasets>=2.0.0->evaluate->tensorrt_llm)\n",
      "  Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (12 kB)\n",
      "Collecting multidict<7.0,>=4.5 (from aiohttp->datasets>=2.0.0->evaluate->tensorrt_llm)\n",
      "  Downloading multidict-6.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.2 kB)\n",
      "Collecting yarl<2.0,>=1.0 (from aiohttp->datasets>=2.0.0->evaluate->tensorrt_llm)\n",
      "  Downloading yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (31 kB)\n",
      "Collecting async-timeout<5.0,>=4.0 (from aiohttp->datasets>=2.0.0->evaluate->tensorrt_llm)\n",
      "  Downloading async_timeout-4.0.3-py3-none-any.whl.metadata (4.2 kB)\n",
      "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.8.2->pandas->evaluate->tensorrt_llm) (1.16.0)\n",
      "INFO: pip is looking at multiple versions of torchvision to determine which version is compatible with other requirements. This could take a while.\n",
      "Collecting torchvision>=0.4 (from torchprofile>=0.0.4->nvidia-ammo~=0.7.0->tensorrt_llm)\n",
      "  Downloading torchvision-0.17.0-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)\n",
      "  Downloading torchvision-0.16.2-cp310-cp310-manylinux1_x86_64.whl.metadata (6.6 kB)\n",
      "Downloading accelerate-0.25.0-py3-none-any.whl (265 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m265.7/265.7 kB\u001b[0m \u001b[31m33.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading diffusers-0.15.0-py3-none-any.whl (851 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m851.8/851.8 kB\u001b[0m \u001b[31m64.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading transformers-4.36.1-py3-none-any.whl (8.3 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m8.3/8.3 MB\u001b[0m \u001b[31m111.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading onnx-1.15.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.7 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m15.7/15.7 MB\u001b[0m \u001b[31m91.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading pynvml-11.5.0-py3-none-any.whl (53 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m53.1/53.1 kB\u001b[0m \u001b[31m7.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading sentencepiece-0.2.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m82.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading torch-2.1.2-cp310-cp310-manylinux1_x86_64.whl (670.2 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m670.2/670.2 MB\u001b[0m \u001b[31m2.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading triton-2.1.0-0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (89.2 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m89.2/89.2 MB\u001b[0m \u001b[31m31.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading build-1.0.3-py3-none-any.whl (18 kB)\n",
      "Downloading colored-2.2.4-py3-none-any.whl (16 kB)\n",
      "Downloading cuda_python-12.3.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.6 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m23.6/23.6 MB\u001b[0m \u001b[31m83.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading evaluate-0.4.1-py3-none-any.whl (84 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m84.1/84.1 kB\u001b[0m \u001b[31m2.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading janus-1.0.0-py3-none-any.whl (6.9 kB)\n",
      "Downloading lark-1.1.9-py3-none-any.whl (111 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m111.7/111.7 kB\u001b[0m \u001b[31m720.2 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading optimum-1.17.1-py3-none-any.whl (407 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m407.1/407.1 kB\u001b[0m \u001b[31m3.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading datasets-2.17.1-py3-none-any.whl (536 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m536.7/536.7 kB\u001b[0m \u001b[31m3.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading dill-0.3.8-py3-none-any.whl (116 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m116.3/116.3 kB\u001b[0m \u001b[31m963.0 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading fsspec-2023.10.0-py3-none-any.whl (166 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m166.4/166.4 kB\u001b[0m \u001b[31m1.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading huggingface_hub-0.20.3-py3-none-any.whl (330 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m330.1/330.1 kB\u001b[0m \u001b[31m2.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading onnxruntime-1.16.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.4 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.4/6.4 MB\u001b[0m \u001b[31m28.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading regex-2023.12.25-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (773 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m774.0/774.0 kB\u001b[0m \u001b[31m6.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading responses-0.18.0-py3-none-any.whl (38 kB)\n",
      "Downloading safetensors-0.4.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m10.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading tokenizers-0.15.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.6 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.6/3.6 MB\u001b[0m \u001b[31m15.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading torchprofile-0.0.4-py3-none-any.whl (7.7 kB)\n",
      "Downloading tqdm-4.66.2-py3-none-any.whl (78 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.3/78.3 kB\u001b[0m \u001b[31m530.9 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m302.2 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading filelock-3.13.1-py3-none-any.whl (11 kB)\n",
      "Downloading importlib_metadata-7.0.1-py3-none-any.whl (23 kB)\n",
      "Downloading multiprocess-0.70.16-py310-none-any.whl (134 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m134.8/134.8 kB\u001b[0m \u001b[31m1.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading networkx-3.2.1-py3-none-any.whl (1.6 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.6/1.6 MB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading ninja-1.11.1.1-py2.py3-none-manylinux1_x86_64.manylinux_2_5_x86_64.whl (307 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m307.2/307.2 kB\u001b[0m \u001b[31m2.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading pandas-2.2.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (13.0 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m13.0/13.0 MB\u001b[0m \u001b[31m34.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m0:01\u001b[0m\n",
      "\u001b[?25hDownloading pyproject_hooks-1.0.0-py3-none-any.whl (9.3 kB)\n",
      "Downloading scipy-1.12.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (38.4 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m38.4/38.4 MB\u001b[0m \u001b[31m13.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading sympy-1.12-py3-none-any.whl (5.7 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.7/5.7 MB\u001b[0m \u001b[31m20.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading xxhash-3.4.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (194 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m194.1/194.1 kB\u001b[0m \u001b[31m1.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0mta \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading aiohttp-3.9.4rc0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m1.2/1.2 MB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m86.8/86.8 kB\u001b[0m \u001b[31m649.1 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hDownloading mpmath-1.4.0a0-py3-none-any.whl (537 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m537.3/537.3 kB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading pyarrow-15.0.0-cp310-cp310-manylinux_2_28_x86_64.whl (38.3 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m38.3/38.3 MB\u001b[0m \u001b[31m15.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading pytz-2024.1-py2.py3-none-any.whl (505 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m505.5/505.5 kB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading torchvision-0.16.2-cp310-cp310-manylinux1_x86_64.whl (6.8 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m6.8/6.8 MB\u001b[0m \u001b[31m26.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading tzdata-2024.1-py2.py3-none-any.whl (345 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m345.4/345.4 kB\u001b[0m \u001b[31m2.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading zipp-3.17.0-py3-none-any.whl (7.4 kB)\n",
      "Downloading flatbuffers-23.5.26-py2.py3-none-any.whl (26 kB)\n",
      "Downloading pyarrow_hotfix-0.6-py3-none-any.whl (7.9 kB)\n",
      "Downloading aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n",
      "Downloading async_timeout-4.0.3-py3-none-any.whl (5.7 kB)\n",
      "Downloading frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (239 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m239.5/239.5 kB\u001b[0m \u001b[31m1.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hDownloading multidict-6.0.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (124 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m124.3/124.3 kB\u001b[0m \u001b[31m968.5 kB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hDownloading yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (301 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m301.6/301.6 kB\u001b[0m \u001b[31m2.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m:00:01\u001b[0m\n",
      "\u001b[?25hBuilding wheels for collected packages: tensorrt, mpi4py\n",
      "  Building wheel for tensorrt (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for tensorrt: filename=tensorrt-9.2.0.post12.dev5-py2.py3-none-any.whl size=17643 sha256=0b87772920492ffa0c1c9ac88ebdd2960bfc02d8f09b1f7a86e16d8cb4c4b906\n",
      "  Stored in directory: /root/.cache/pip/wheels/aa/96/bf/028c219d3560856a5fdb8b3aec8bf01e9d485521c092a64d02\n",
      "  Building wheel for mpi4py (pyproject.toml) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for mpi4py: filename=mpi4py-3.1.5-cp310-cp310-linux_x86_64.whl size=2746507 sha256=964395322677f3cbcb049620daf113b1028fb8e05b23dd5142d82faec1d8f4fc\n",
      "  Stored in directory: /root/.cache/pip/wheels/18/2b/7f/c852523089e9182b45fca50ff56f49a51eeb6284fd25a66713\n",
      "Successfully built tensorrt mpi4py\n",
      "Installing collected packages: tensorrt_bindings, sentencepiece, pytz, ninja, flatbuffers, cuda-python, zipp, xxhash, tzdata, tqdm, scipy, safetensors, regex, pyproject_hooks, pynvml, pyarrow-hotfix, pyarrow, onnx, nvidia-nvtx-cu12, nvidia-nvjitlink-cu12, nvidia-nccl-cu12, nvidia-curand-cu12, nvidia-cufft-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, networkx, multidict, mpmath, mpi4py, lark, janus, humanfriendly, fsspec, frozenlist, filelock, dill, colored, async-timeout, yarl, triton, sympy, responses, pandas, onnx-graphsurgeon, nvidia-cusparse-cu12, nvidia-cudnn-cu12, multiprocess, importlib-metadata, huggingface-hub, coloredlogs, build, aiosignal, tokenizers, tensorrt_libs, onnxruntime, nvidia-cusolver-cu12, diffusers, aiohttp, transformers, torch, tensorrt, torchvision, datasets, accelerate, torchprofile, optimum, evaluate, nvidia-ammo, tensorrt_llm\n",
      "  Attempting uninstall: tensorrt\n",
      "    Found existing installation: tensorrt 8.6.1\n",
      "    Uninstalling tensorrt-8.6.1:\n",
      "      Successfully uninstalled tensorrt-8.6.1\n",
      "Successfully installed accelerate-0.25.0 aiohttp-3.9.4rc0 aiosignal-1.3.1 async-timeout-4.0.3 build-1.0.3 colored-2.2.4 coloredlogs-15.0.1 cuda-python-12.3.0 datasets-2.17.1 diffusers-0.15.0 dill-0.3.8 evaluate-0.4.1 filelock-3.13.1 flatbuffers-23.5.26 frozenlist-1.4.1 fsspec-2023.10.0 huggingface-hub-0.20.3 humanfriendly-10.0 importlib-metadata-7.0.1 janus-1.0.0 lark-1.1.9 mpi4py-3.1.5 mpmath-1.4.0a0 multidict-6.0.5 multiprocess-0.70.16 networkx-3.2.1 ninja-1.11.1.1 nvidia-ammo-0.7.3 nvidia-cublas-cu12-12.1.3.1 nvidia-cuda-cupti-cu12-12.1.105 nvidia-cuda-nvrtc-cu12-12.1.105 nvidia-cuda-runtime-cu12-12.1.105 nvidia-cudnn-cu12-8.9.2.26 nvidia-cufft-cu12-11.0.2.54 nvidia-curand-cu12-10.3.2.106 nvidia-cusolver-cu12-11.4.5.107 nvidia-cusparse-cu12-12.1.0.106 nvidia-nccl-cu12-2.18.1 nvidia-nvjitlink-cu12-12.3.101 nvidia-nvtx-cu12-12.1.105 onnx-1.15.0 onnx-graphsurgeon-0.3.25 onnxruntime-1.16.3 optimum-1.17.1 pandas-2.2.1 pyarrow-15.0.0 pyarrow-hotfix-0.6 pynvml-11.5.0 pyproject_hooks-1.0.0 pytz-2024.1 regex-2023.12.25 responses-0.18.0 safetensors-0.4.2 scipy-1.12.0 sentencepiece-0.2.0 sympy-1.12 tensorrt-9.2.0.post12.dev5 tensorrt_bindings-9.2.0.post12.dev5 tensorrt_libs-9.2.0.post12.dev5 tensorrt_llm-0.9.0.dev2024022000 tokenizers-0.15.2 torch-2.1.2 torchprofile-0.0.4 torchvision-0.16.2 tqdm-4.66.2 transformers-4.36.1 triton-2.1.0 tzdata-2024.1 xxhash-3.4.1 yarl-1.9.4 zipp-3.17.0\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython -m pip install --upgrade pip\u001b[0m\n",
      "Found existing installation: mpmath 1.4.0a0\n",
      "Uninstalling mpmath-1.4.0a0:\n",
      "  Successfully uninstalled mpmath-1.4.0a0\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0mCollecting mpmath==1.3.0\n",
      "  Downloading mpmath-1.3.0-py3-none-any.whl.metadata (8.6 kB)\n",
      "Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m536.2/536.2 kB\u001b[0m \u001b[31m9.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: mpmath\n",
      "Successfully installed mpmath-1.3.0\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython -m pip install --upgrade pip\u001b[0m\n",
      "Collecting ipywidgets\n",
      "  Downloading ipywidgets-8.1.2-py3-none-any.whl.metadata (2.4 kB)\n",
      "Requirement already satisfied: comm>=0.1.3 in /usr/local/lib/python3.10/dist-packages (from ipywidgets) (0.2.1)\n",
      "Requirement already satisfied: ipython>=6.1.0 in /usr/local/lib/python3.10/dist-packages (from ipywidgets) (8.22.1)\n",
      "Requirement already satisfied: traitlets>=4.3.1 in /usr/local/lib/python3.10/dist-packages (from ipywidgets) (5.14.1)\n",
      "Collecting widgetsnbextension~=4.0.10 (from ipywidgets)\n",
      "  Downloading widgetsnbextension-4.0.10-py3-none-any.whl.metadata (1.6 kB)\n",
      "Collecting jupyterlab-widgets~=3.0.10 (from ipywidgets)\n",
      "  Downloading jupyterlab_widgets-3.0.10-py3-none-any.whl.metadata (4.1 kB)\n",
      "Requirement already satisfied: decorator in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets) (5.1.1)\n",
      "Requirement already satisfied: jedi>=0.16 in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets) (0.19.1)\n",
      "Requirement already satisfied: matplotlib-inline in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets) (0.1.6)\n",
      "Requirement already satisfied: prompt-toolkit<3.1.0,>=3.0.41 in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets) (3.0.43)\n",
      "Requirement already satisfied: pygments>=2.4.0 in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets) (2.17.2)\n",
      "Requirement already satisfied: stack-data in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets) (0.6.3)\n",
      "Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets) (1.2.0)\n",
      "Requirement already satisfied: pexpect>4.3 in /usr/local/lib/python3.10/dist-packages (from ipython>=6.1.0->ipywidgets) (4.9.0)\n",
      "Requirement already satisfied: parso<0.9.0,>=0.8.3 in /usr/local/lib/python3.10/dist-packages (from jedi>=0.16->ipython>=6.1.0->ipywidgets) (0.8.3)\n",
      "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.10/dist-packages (from pexpect>4.3->ipython>=6.1.0->ipywidgets) (0.7.0)\n",
      "Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from prompt-toolkit<3.1.0,>=3.0.41->ipython>=6.1.0->ipywidgets) (0.2.13)\n",
      "Requirement already satisfied: executing>=1.2.0 in /usr/local/lib/python3.10/dist-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.0.1)\n",
      "Requirement already satisfied: asttokens>=2.1.0 in /usr/local/lib/python3.10/dist-packages (from stack-data->ipython>=6.1.0->ipywidgets) (2.4.1)\n",
      "Requirement already satisfied: pure-eval in /usr/local/lib/python3.10/dist-packages (from stack-data->ipython>=6.1.0->ipywidgets) (0.2.2)\n",
      "Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.10/dist-packages (from asttokens>=2.1.0->stack-data->ipython>=6.1.0->ipywidgets) (1.16.0)\n",
      "Downloading ipywidgets-8.1.2-py3-none-any.whl (139 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m139.4/139.4 kB\u001b[0m \u001b[31m4.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading jupyterlab_widgets-3.0.10-py3-none-any.whl (215 kB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m215.0/215.0 kB\u001b[0m \u001b[31m8.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
      "\u001b[?25hDownloading widgetsnbextension-4.0.10-py3-none-any.whl (2.3 MB)\n",
      "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m2.3/2.3 MB\u001b[0m \u001b[31m15.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m00:01\u001b[0m00:01\u001b[0m\n",
      "\u001b[?25hInstalling collected packages: widgetsnbextension, jupyterlab-widgets, ipywidgets\n",
      "Successfully installed ipywidgets-8.1.2 jupyterlab-widgets-3.0.10 widgetsnbextension-4.0.10\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip is available: \u001b[0m\u001b[31;49m23.3.2\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpython -m pip install --upgrade pip\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "!pip install tensorrt_llm -U --pre --extra-index-url https://pypi.nvidia.com\n",
    "!pip uninstall -y mpmath\n",
    "!pip install mpmath==1.3.0\n",
    "!pip install ipywidgets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3a3fdbb1-d918-447b-8276-d3c23f5c2633",
   "metadata": {},
   "source": [
    "#### Step 2 - Convert Mistral to the TensorRT format"
   ]
  },
  {
   "cell_type": "raw",
   "id": "a29973b3-b26a-40b5-af39-3a8f7a46faf9",
   "metadata": {},
   "source": [
    "Next we use TensorRT's conversion and build scripts to first convert the model and then build the engine. TensorRT-LLM offers a plethora of features that you can enable during the conversion including. See more examples in the documentation [here](\"https://github.com/NVIDIA/TensorRT-LLM/tree/main/examples/llama#int8-kv-cache)\n",
    "\n",
    "1. FP8 KV Cache\n",
    "2. SmoothQuant\n",
    "3. INT8 KV Cache"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "97b429d9-4ac0-401e-b4ba-1625389bc298",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-02-27 07:33:47--  https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/llama/convert_checkpoint.py\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 63559 (62K) [text/plain]\n",
      "Saving to: ‘./convert_checkpoint.py’\n",
      "\n",
      "convert_checkpoint. 100%[===================>]  62.07K  --.-KB/s    in 0.009s  \n",
      "\n",
      "2024-02-27 07:33:47 (6.98 MB/s) - ‘./convert_checkpoint.py’ saved [63559/63559]\n",
      "\n",
      "--2024-02-27 07:33:47--  https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/run.py\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.108.133, 185.199.111.133, 185.199.110.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.108.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 21871 (21K) [text/plain]\n",
      "Saving to: ‘./run.py’\n",
      "\n",
      "run.py              100%[===================>]  21.36K  --.-KB/s    in 0.002s  \n",
      "\n",
      "2024-02-27 07:33:47 (12.2 MB/s) - ‘./run.py’ saved [21871/21871]\n",
      "\n",
      "--2024-02-27 07:33:47--  https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/utils.py\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.111.133, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 4883 (4.8K) [text/plain]\n",
      "Saving to: ‘./utils.py’\n",
      "\n",
      "utils.py            100%[===================>]   4.77K  --.-KB/s    in 0s      \n",
      "\n",
      "2024-02-27 07:33:48 (98.2 MB/s) - ‘./utils.py’ saved [4883/4883]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!wget https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/llama/convert_checkpoint.py -P .\n",
    "!wget https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/run.py -P .\n",
    "!wget https://raw.githubusercontent.com/NVIDIA/TensorRT-LLM/main/examples/utils.py -P ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "6eed0769-53e1-47f6-b0fb-8a10d97f3556",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[TensorRT-LLM] TensorRT-LLM version: 0.9.0.dev2024022000\n",
      "0.9.0.dev2024022000\n",
      "config.json: 100%|█████████████████████████████| 571/571 [00:00<00:00, 4.44MB/s]\n",
      "model.safetensors.index.json: 100%|████████| 25.1k/25.1k [00:00<00:00, 81.3MB/s]\n",
      "Downloading shards:   0%|                                 | 0/2 [00:00<?, ?it/s]\n",
      "model-00001-of-00002.safetensors:   0%|             | 0.00/9.94G [00:00<?, ?B/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   0%|     | 21.0M/9.94G [00:00<01:09, 142MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   0%|     | 41.9M/9.94G [00:00<00:59, 168MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   1%|     | 62.9M/9.94G [00:00<00:53, 184MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   1%|     | 83.9M/9.94G [00:00<00:51, 192MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   1%|      | 115M/9.94G [00:00<00:48, 201MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   1%|      | 147M/9.94G [00:00<00:47, 205MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   2%|      | 178M/9.94G [00:00<00:46, 208MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   2%|▏     | 210M/9.94G [00:01<00:46, 209MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   2%|▏     | 241M/9.94G [00:01<00:46, 210MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   3%|▏     | 273M/9.94G [00:01<00:45, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   3%|▏     | 304M/9.94G [00:01<00:45, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   3%|▏     | 336M/9.94G [00:01<00:45, 212MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   4%|▏     | 367M/9.94G [00:01<00:44, 213MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   4%|▏     | 398M/9.94G [00:01<00:44, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   4%|▎     | 430M/9.94G [00:02<00:44, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   5%|▎     | 461M/9.94G [00:02<00:43, 216MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   5%|▎     | 493M/9.94G [00:02<00:43, 216MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   5%|▎     | 524M/9.94G [00:02<00:44, 213MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   6%|▎     | 556M/9.94G [00:02<00:44, 210MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   6%|▎     | 587M/9.94G [00:02<00:44, 212MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   6%|▎     | 619M/9.94G [00:02<00:43, 214MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   7%|▍     | 650M/9.94G [00:03<00:43, 216MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   7%|▍     | 682M/9.94G [00:03<00:42, 219MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   7%|▍     | 713M/9.94G [00:03<00:41, 222MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   7%|▍     | 744M/9.94G [00:03<00:41, 224MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   8%|▍     | 776M/9.94G [00:03<00:40, 227MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   8%|▍     | 807M/9.94G [00:03<00:39, 229MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   8%|▌     | 839M/9.94G [00:03<00:39, 232MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   9%|▌     | 870M/9.94G [00:04<00:38, 236MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   9%|▌     | 902M/9.94G [00:04<00:39, 227MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:   9%|▌     | 933M/9.94G [00:04<00:41, 218MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  10%|▌     | 965M/9.94G [00:04<00:42, 209MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  10%|▌     | 996M/9.94G [00:04<00:43, 207MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  10%|▌    | 1.02G/9.94G [00:04<00:43, 206MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  10%|▌    | 1.04G/9.94G [00:04<00:43, 207MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  11%|▌    | 1.07G/9.94G [00:05<00:42, 209MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  11%|▌    | 1.10G/9.94G [00:05<00:41, 212MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  11%|▌    | 1.13G/9.94G [00:05<00:41, 214MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  12%|▌    | 1.16G/9.94G [00:05<00:40, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  12%|▌    | 1.20G/9.94G [00:05<00:40, 217MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  12%|▌    | 1.23G/9.94G [00:05<00:40, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  13%|▋    | 1.26G/9.94G [00:05<00:39, 219MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  13%|▋    | 1.29G/9.94G [00:06<00:38, 223MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  13%|▋    | 1.32G/9.94G [00:06<00:38, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  14%|▋    | 1.35G/9.94G [00:06<00:37, 228MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  14%|▋    | 1.38G/9.94G [00:06<00:36, 232MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  14%|▋    | 1.42G/9.94G [00:06<00:36, 235MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  15%|▋    | 1.45G/9.94G [00:06<00:35, 240MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  15%|▋    | 1.48G/9.94G [00:06<00:34, 243MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  15%|▊    | 1.51G/9.94G [00:06<00:34, 246MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  16%|▊    | 1.54G/9.94G [00:07<00:33, 250MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  16%|▊    | 1.57G/9.94G [00:07<00:35, 237MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  16%|▊    | 1.60G/9.94G [00:07<00:36, 229MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  16%|▊    | 1.64G/9.94G [00:07<00:36, 225MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  17%|▊    | 1.67G/9.94G [00:07<00:36, 224MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  17%|▊    | 1.70G/9.94G [00:07<00:36, 224MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  17%|▊    | 1.73G/9.94G [00:07<00:36, 224MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  18%|▉    | 1.76G/9.94G [00:08<00:37, 218MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  18%|▉    | 1.79G/9.94G [00:08<00:37, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  18%|▉    | 1.82G/9.94G [00:08<00:38, 213MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  19%|▉    | 1.86G/9.94G [00:08<00:37, 213MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  19%|▉    | 1.89G/9.94G [00:08<00:37, 213MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  19%|▉    | 1.92G/9.94G [00:08<00:37, 213MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  20%|▉    | 1.95G/9.94G [00:08<00:37, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  20%|▉    | 1.98G/9.94G [00:09<00:37, 214MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  20%|█    | 2.01G/9.94G [00:09<00:37, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  21%|█    | 2.04G/9.94G [00:09<00:37, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  21%|█    | 2.08G/9.94G [00:09<00:41, 190MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  21%|█    | 2.11G/9.94G [00:09<00:39, 197MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  22%|█    | 2.14G/9.94G [00:09<00:38, 203MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  22%|█    | 2.17G/9.94G [00:10<00:37, 209MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  22%|█    | 2.20G/9.94G [00:10<00:39, 195MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  22%|█    | 2.23G/9.94G [00:10<00:37, 204MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  23%|█▏   | 2.26G/9.94G [00:10<00:36, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  23%|█▏   | 2.30G/9.94G [00:10<00:35, 218MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  23%|█▏   | 2.33G/9.94G [00:10<00:33, 225MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  24%|█▏   | 2.36G/9.94G [00:10<00:32, 231MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  24%|█▏   | 2.39G/9.94G [00:11<00:31, 238MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  24%|█▏   | 2.42G/9.94G [00:11<00:31, 242MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  25%|█▏   | 2.45G/9.94G [00:11<00:30, 246MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  25%|█▏   | 2.49G/9.94G [00:11<00:29, 249MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  25%|█▎   | 2.52G/9.94G [00:11<00:30, 245MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  26%|█▎   | 2.55G/9.94G [00:11<00:31, 234MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  26%|█▎   | 2.58G/9.94G [00:11<00:32, 227MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  26%|█▎   | 2.61G/9.94G [00:11<00:33, 222MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  27%|█▎   | 2.64G/9.94G [00:12<00:33, 217MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  27%|█▎   | 2.67G/9.94G [00:12<00:35, 208MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  27%|█▎   | 2.71G/9.94G [00:12<00:34, 208MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  27%|█▎   | 2.73G/9.94G [00:12<00:34, 208MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  28%|█▍   | 2.75G/9.94G [00:12<00:34, 208MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  28%|█▍   | 2.78G/9.94G [00:12<00:34, 210MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  28%|█▍   | 2.81G/9.94G [00:12<00:33, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  29%|█▍   | 2.84G/9.94G [00:13<00:33, 212MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  29%|█▍   | 2.87G/9.94G [00:13<00:32, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  29%|█▍   | 2.90G/9.94G [00:13<00:32, 217MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  30%|█▍   | 2.94G/9.94G [00:13<00:31, 220MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  30%|█▍   | 2.97G/9.94G [00:13<00:31, 223MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  30%|█▌   | 3.00G/9.94G [00:13<00:30, 228MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  30%|█▌   | 3.03G/9.94G [00:13<00:29, 233MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  31%|█▌   | 3.06G/9.94G [00:14<00:28, 239MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  31%|█▌   | 3.09G/9.94G [00:14<00:28, 243MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  31%|█▌   | 3.12G/9.94G [00:14<00:29, 231MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  32%|█▌   | 3.16G/9.94G [00:14<00:30, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  32%|█▌   | 3.19G/9.94G [00:14<00:32, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  32%|█▌   | 3.22G/9.94G [00:14<00:32, 210MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  33%|█▋   | 3.25G/9.94G [00:14<00:31, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  33%|█▋   | 3.28G/9.94G [00:15<00:31, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  33%|█▋   | 3.31G/9.94G [00:15<00:31, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  34%|█▋   | 3.34G/9.94G [00:15<00:31, 212MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  34%|█▋   | 3.38G/9.94G [00:15<00:30, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  34%|█▋   | 3.41G/9.94G [00:15<00:30, 218MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  35%|█▋   | 3.44G/9.94G [00:15<00:29, 220MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  35%|█▋   | 3.47G/9.94G [00:15<00:29, 223MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  35%|█▊   | 3.50G/9.94G [00:16<00:28, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  36%|█▊   | 3.53G/9.94G [00:16<00:28, 229MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  36%|█▊   | 3.57G/9.94G [00:16<00:27, 231MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  36%|█▊   | 3.60G/9.94G [00:16<00:27, 233MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  36%|█▊   | 3.63G/9.94G [00:16<00:26, 237MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  37%|█▊   | 3.66G/9.94G [00:16<00:26, 240MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  37%|█▊   | 3.69G/9.94G [00:16<00:25, 244MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  37%|█▊   | 3.72G/9.94G [00:16<00:25, 246MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  38%|█▉   | 3.75G/9.94G [00:17<00:24, 248MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  38%|█▉   | 3.79G/9.94G [00:17<00:24, 251MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  38%|█▉   | 3.82G/9.94G [00:17<00:26, 235MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  39%|█▉   | 3.85G/9.94G [00:17<00:26, 227MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  39%|█▉   | 3.88G/9.94G [00:17<00:27, 222MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  39%|█▉   | 3.91G/9.94G [00:17<00:27, 220MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  40%|█▉   | 3.94G/9.94G [00:17<00:27, 220MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  40%|█▉   | 3.97G/9.94G [00:18<00:27, 220MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  40%|██   | 4.01G/9.94G [00:18<00:26, 221MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  41%|██   | 4.04G/9.94G [00:18<00:26, 223MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  41%|██   | 4.07G/9.94G [00:18<00:26, 224MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  41%|██   | 4.10G/9.94G [00:18<00:25, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  42%|██   | 4.13G/9.94G [00:18<00:25, 229MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  42%|██   | 4.16G/9.94G [00:18<00:25, 231MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  42%|██   | 4.19G/9.94G [00:19<00:24, 234MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  42%|██   | 4.23G/9.94G [00:19<00:24, 238MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  43%|██▏  | 4.26G/9.94G [00:19<00:23, 242MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  43%|██▏  | 4.29G/9.94G [00:19<00:24, 229MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  43%|██▏  | 4.32G/9.94G [00:19<00:28, 194MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  44%|██▏  | 4.34G/9.94G [00:19<00:28, 196MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  44%|██▏  | 4.37G/9.94G [00:19<00:27, 201MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  44%|██▏  | 4.40G/9.94G [00:20<00:27, 205MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  45%|██▏  | 4.44G/9.94G [00:20<00:26, 209MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  45%|██▏  | 4.47G/9.94G [00:20<00:36, 149MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  45%|██▎  | 4.50G/9.94G [00:20<00:32, 165MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  46%|██▎  | 4.53G/9.94G [00:20<00:30, 180MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  46%|██▎  | 4.56G/9.94G [00:20<00:27, 192MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  46%|██▎  | 4.59G/9.94G [00:21<00:26, 199MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  47%|██▎  | 4.62G/9.94G [00:21<00:26, 202MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  47%|██▎  | 4.66G/9.94G [00:21<00:25, 205MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  47%|██▎  | 4.69G/9.94G [00:21<00:25, 206MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  47%|██▎  | 4.72G/9.94G [00:21<00:24, 210MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  48%|██▍  | 4.75G/9.94G [00:21<00:24, 212MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  48%|██▍  | 4.78G/9.94G [00:22<00:24, 214MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  48%|██▍  | 4.81G/9.94G [00:22<00:23, 218MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  49%|██▍  | 4.84G/9.94G [00:22<00:22, 222MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  49%|██▍  | 4.88G/9.94G [00:22<00:22, 228MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  49%|██▍  | 4.91G/9.94G [00:22<00:27, 184MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  50%|██▍  | 4.94G/9.94G [00:22<00:25, 198MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  50%|██▍  | 4.97G/9.94G [00:22<00:23, 214MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  50%|██▌  | 5.00G/9.94G [00:23<00:23, 213MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  51%|██▌  | 5.03G/9.94G [00:23<00:23, 209MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  51%|██▌  | 5.06G/9.94G [00:23<00:23, 208MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  51%|██▌  | 5.10G/9.94G [00:23<00:23, 208MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  52%|██▌  | 5.13G/9.94G [00:23<00:23, 208MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  52%|██▌  | 5.16G/9.94G [00:23<00:22, 208MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  52%|██▌  | 5.19G/9.94G [00:23<00:22, 210MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  53%|██▋  | 5.22G/9.94G [00:24<00:22, 212MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  53%|██▋  | 5.25G/9.94G [00:24<00:21, 214MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  53%|██▋  | 5.28G/9.94G [00:24<00:21, 216MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  53%|██▋  | 5.32G/9.94G [00:24<00:21, 217MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  54%|██▋  | 5.35G/9.94G [00:24<00:20, 220MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  54%|██▋  | 5.38G/9.94G [00:24<00:20, 222MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  54%|██▋  | 5.41G/9.94G [00:24<00:20, 224MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  55%|██▋  | 5.44G/9.94G [00:25<00:19, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  55%|██▊  | 5.47G/9.94G [00:25<00:19, 228MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  55%|██▊  | 5.51G/9.94G [00:25<00:19, 231MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  56%|██▊  | 5.54G/9.94G [00:25<00:18, 234MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  56%|██▊  | 5.57G/9.94G [00:25<00:18, 237MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  56%|██▊  | 5.60G/9.94G [00:25<00:18, 241MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  57%|██▊  | 5.63G/9.94G [00:25<00:17, 245MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  57%|██▊  | 5.66G/9.94G [00:25<00:17, 248MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  57%|██▊  | 5.69G/9.94G [00:26<00:16, 252MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  58%|██▉  | 5.73G/9.94G [00:26<00:17, 238MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  58%|██▉  | 5.76G/9.94G [00:26<00:18, 230MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  58%|██▉  | 5.79G/9.94G [00:26<00:18, 227MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  59%|██▉  | 5.82G/9.94G [00:26<00:18, 225MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  59%|██▉  | 5.85G/9.94G [00:26<00:18, 224MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  59%|██▉  | 5.88G/9.94G [00:26<00:18, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  59%|██▉  | 5.91G/9.94G [00:27<00:17, 227MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  60%|██▉  | 5.95G/9.94G [00:27<00:17, 229MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  60%|███  | 5.98G/9.94G [00:27<00:17, 231MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  60%|███  | 6.01G/9.94G [00:27<00:16, 235MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  61%|███  | 6.04G/9.94G [00:27<00:16, 239MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  61%|███  | 6.07G/9.94G [00:27<00:15, 243MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  61%|███  | 6.10G/9.94G [00:27<00:16, 233MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  62%|███  | 6.13G/9.94G [00:28<00:16, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  62%|███  | 6.17G/9.94G [00:28<00:16, 223MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  62%|███  | 6.20G/9.94G [00:28<00:16, 222MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  63%|███▏ | 6.23G/9.94G [00:28<00:16, 221MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  63%|███▏ | 6.26G/9.94G [00:28<00:16, 222MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  63%|███▏ | 6.29G/9.94G [00:28<00:16, 223MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  64%|███▏ | 6.32G/9.94G [00:28<00:16, 224MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  64%|███▏ | 6.35G/9.94G [00:29<00:15, 225MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  64%|███▏ | 6.39G/9.94G [00:29<00:15, 227MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  65%|███▏ | 6.42G/9.94G [00:29<00:17, 202MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  65%|███▏ | 6.45G/9.94G [00:29<00:16, 206MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  65%|███▎ | 6.48G/9.94G [00:29<00:16, 208MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  65%|███▎ | 6.51G/9.94G [00:29<00:16, 209MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  66%|███▎ | 6.54G/9.94G [00:29<00:16, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  66%|███▎ | 6.57G/9.94G [00:30<00:15, 213MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  66%|███▎ | 6.61G/9.94G [00:30<00:15, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  67%|███▎ | 6.64G/9.94G [00:30<00:15, 217MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  67%|███▎ | 6.67G/9.94G [00:30<00:14, 220MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  67%|███▎ | 6.70G/9.94G [00:30<00:14, 223MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  68%|███▍ | 6.73G/9.94G [00:30<00:14, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  68%|███▍ | 6.76G/9.94G [00:30<00:14, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  68%|███▍ | 6.79G/9.94G [00:31<00:14, 216MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  69%|███▍ | 6.83G/9.94G [00:31<00:14, 213MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  69%|███▍ | 6.86G/9.94G [00:31<00:14, 212MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  69%|███▍ | 6.89G/9.94G [00:31<00:14, 212MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  70%|███▍ | 6.92G/9.94G [00:31<00:14, 213MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  70%|███▍ | 6.95G/9.94G [00:31<00:13, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  70%|███▌ | 6.98G/9.94G [00:31<00:13, 212MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  71%|███▌ | 7.01G/9.94G [00:32<00:13, 212MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  71%|███▌ | 7.05G/9.94G [00:32<00:13, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  71%|███▌ | 7.08G/9.94G [00:32<00:13, 212MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  72%|███▌ | 7.11G/9.94G [00:32<00:13, 214MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  72%|███▌ | 7.14G/9.94G [00:32<00:12, 216MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  72%|███▌ | 7.17G/9.94G [00:32<00:12, 217MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  72%|███▌ | 7.20G/9.94G [00:33<00:12, 220MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  73%|███▋ | 7.24G/9.94G [00:33<00:12, 223MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  73%|███▋ | 7.27G/9.94G [00:33<00:11, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  73%|███▋ | 7.30G/9.94G [00:33<00:11, 231MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  74%|███▋ | 7.33G/9.94G [00:33<00:11, 236MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  74%|███▋ | 7.36G/9.94G [00:33<00:11, 227MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  74%|███▋ | 7.39G/9.94G [00:33<00:11, 223MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  75%|███▋ | 7.42G/9.94G [00:33<00:11, 221MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  75%|███▋ | 7.46G/9.94G [00:34<00:11, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  75%|███▊ | 7.49G/9.94G [00:34<00:11, 217MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  76%|███▊ | 7.52G/9.94G [00:34<00:11, 219MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  76%|███▊ | 7.55G/9.94G [00:34<00:10, 220MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  76%|███▊ | 7.58G/9.94G [00:34<00:10, 222MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  77%|███▊ | 7.61G/9.94G [00:34<00:10, 220MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  77%|███▊ | 7.64G/9.94G [00:34<00:10, 224MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  77%|███▊ | 7.68G/9.94G [00:35<00:10, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  78%|███▉ | 7.71G/9.94G [00:35<00:09, 230MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  78%|███▉ | 7.74G/9.94G [00:35<00:09, 235MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  78%|███▉ | 7.77G/9.94G [00:35<00:09, 233MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  78%|███▉ | 7.80G/9.94G [00:35<00:09, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  79%|███▉ | 7.83G/9.94G [00:35<00:09, 224MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  79%|███▉ | 7.86G/9.94G [00:35<00:09, 224MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  79%|███▉ | 7.90G/9.94G [00:36<00:09, 224MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  80%|███▉ | 7.93G/9.94G [00:36<00:08, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  80%|████ | 7.96G/9.94G [00:36<00:09, 220MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  80%|████ | 7.99G/9.94G [00:36<00:09, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  81%|████ | 8.02G/9.94G [00:36<00:09, 210MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  81%|████ | 8.05G/9.94G [00:36<00:08, 210MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  81%|████ | 8.08G/9.94G [00:36<00:08, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  82%|████ | 8.12G/9.94G [00:37<00:08, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  82%|████ | 8.15G/9.94G [00:37<00:08, 213MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  82%|████ | 8.18G/9.94G [00:37<00:08, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  83%|████▏| 8.21G/9.94G [00:37<00:08, 217MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  83%|████▏| 8.24G/9.94G [00:37<00:07, 218MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  83%|████▏| 8.27G/9.94G [00:38<00:10, 153MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  84%|████▏| 8.30G/9.94G [00:38<00:09, 170MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  84%|████▏| 8.34G/9.94G [00:38<00:08, 184MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  84%|████▏| 8.37G/9.94G [00:38<00:07, 197MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  84%|████▏| 8.40G/9.94G [00:38<00:07, 208MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  85%|████▏| 8.43G/9.94G [00:38<00:06, 217MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  85%|████▎| 8.46G/9.94G [00:38<00:06, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  85%|████▎| 8.49G/9.94G [00:39<00:06, 217MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  86%|████▎| 8.52G/9.94G [00:39<00:06, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  86%|████▎| 8.56G/9.94G [00:39<00:06, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  86%|████▎| 8.59G/9.94G [00:39<00:06, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  87%|████▎| 8.62G/9.94G [00:39<00:06, 204MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  87%|████▎| 8.65G/9.94G [00:39<00:06, 209MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  87%|████▎| 8.68G/9.94G [00:39<00:05, 214MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  88%|████▍| 8.71G/9.94G [00:40<00:05, 218MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  88%|████▍| 8.75G/9.94G [00:40<00:05, 221MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  88%|████▍| 8.78G/9.94G [00:40<00:05, 223MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  89%|████▍| 8.81G/9.94G [00:40<00:05, 225MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  89%|████▍| 8.84G/9.94G [00:40<00:04, 228MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  89%|████▍| 8.87G/9.94G [00:40<00:04, 229MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  90%|████▍| 8.90G/9.94G [00:40<00:04, 228MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  90%|████▍| 8.93G/9.94G [00:41<00:04, 217MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  90%|████▌| 8.97G/9.94G [00:41<00:04, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  90%|████▌| 9.00G/9.94G [00:41<00:04, 214MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  91%|████▌| 9.03G/9.94G [00:41<00:04, 214MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  91%|████▌| 9.06G/9.94G [00:41<00:04, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  91%|████▌| 9.09G/9.94G [00:41<00:03, 216MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  92%|████▌| 9.12G/9.94G [00:41<00:03, 218MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  92%|████▌| 9.15G/9.94G [00:42<00:03, 220MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  92%|████▌| 9.19G/9.94G [00:42<00:03, 209MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  93%|████▋| 9.22G/9.94G [00:42<00:03, 208MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  93%|████▋| 9.24G/9.94G [00:42<00:03, 209MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  93%|████▋| 9.27G/9.94G [00:42<00:03, 211MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  94%|████▋| 9.30G/9.94G [00:42<00:03, 212MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  94%|████▋| 9.33G/9.94G [00:42<00:02, 214MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  94%|████▋| 9.36G/9.94G [00:43<00:02, 217MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  94%|████▋| 9.40G/9.94G [00:43<00:02, 208MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  95%|████▋| 9.43G/9.94G [00:43<00:02, 212MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  95%|████▊| 9.46G/9.94G [00:43<00:02, 215MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  95%|████▊| 9.49G/9.94G [00:43<00:02, 218MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  96%|████▊| 9.52G/9.94G [00:43<00:01, 220MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  96%|████▊| 9.55G/9.94G [00:43<00:01, 223MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  96%|████▊| 9.58G/9.94G [00:44<00:01, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  97%|████▊| 9.62G/9.94G [00:44<00:01, 228MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  97%|████▊| 9.65G/9.94G [00:44<00:01, 226MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  97%|████▊| 9.68G/9.94G [00:44<00:01, 209MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  98%|████▉| 9.71G/9.94G [00:44<00:01, 220MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  98%|████▉| 9.74G/9.94G [00:44<00:00, 214MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  98%|████▉| 9.77G/9.94G [00:44<00:00, 213MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  99%|████▉| 9.80G/9.94G [00:45<00:00, 213MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  99%|████▉| 9.84G/9.94G [00:45<00:00, 213MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors:  99%|████▉| 9.87G/9.94G [00:45<00:00, 214MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors: 100%|████▉| 9.90G/9.94G [00:45<00:00, 216MB/s]\u001b[A\n",
      "model-00001-of-00002.safetensors: 100%|█████| 9.94G/9.94G [00:45<00:00, 218MB/s]\u001b[A\n",
      "Downloading shards:  50%|████████████▌            | 1/2 [00:45<00:45, 45.89s/it]\n",
      "model-00002-of-00002.safetensors:   0%|             | 0.00/4.54G [00:00<?, ?B/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   0%|     | 21.0M/4.54G [00:00<00:21, 209MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   1%|     | 41.9M/4.54G [00:00<00:22, 200MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   2%|     | 73.4M/4.54G [00:00<00:21, 207MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   2%|▏     | 105M/4.54G [00:00<00:21, 208MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   3%|▏     | 126M/4.54G [00:00<00:21, 203MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   3%|▏     | 157M/4.54G [00:00<00:21, 208MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   4%|▏     | 189M/4.54G [00:00<00:21, 205MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   5%|▎     | 210M/4.54G [00:01<00:21, 205MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   5%|▎     | 241M/4.54G [00:01<00:20, 207MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   6%|▎     | 262M/4.54G [00:01<00:20, 205MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   6%|▎     | 283M/4.54G [00:01<00:20, 206MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   7%|▍     | 304M/4.54G [00:01<00:20, 205MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   7%|▍     | 336M/4.54G [00:01<00:20, 208MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   8%|▍     | 367M/4.54G [00:01<00:19, 211MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   9%|▌     | 398M/4.54G [00:01<00:19, 211MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:   9%|▌     | 430M/4.54G [00:02<00:19, 210MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  10%|▌     | 461M/4.54G [00:02<00:19, 211MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  11%|▋     | 493M/4.54G [00:02<00:19, 210MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  12%|▋     | 524M/4.54G [00:02<00:18, 212MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  12%|▋     | 556M/4.54G [00:02<00:18, 214MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  13%|▊     | 587M/4.54G [00:02<00:18, 215MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  14%|▊     | 619M/4.54G [00:02<00:18, 210MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  14%|▊     | 650M/4.54G [00:03<00:26, 148MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  15%|▉     | 682M/4.54G [00:03<00:23, 164MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  16%|▉     | 713M/4.54G [00:03<00:21, 175MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  16%|▉     | 734M/4.54G [00:03<00:20, 181MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  17%|▉     | 755M/4.54G [00:03<00:20, 185MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  17%|█     | 776M/4.54G [00:03<00:20, 184MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  18%|█     | 797M/4.54G [00:04<00:19, 190MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  18%|█     | 818M/4.54G [00:04<00:19, 194MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  18%|█     | 839M/4.54G [00:04<00:20, 184MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  19%|█▏    | 860M/4.54G [00:04<00:19, 189MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  20%|█▏    | 891M/4.54G [00:04<00:18, 198MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  20%|█▏    | 923M/4.54G [00:04<00:17, 204MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  21%|█▎    | 954M/4.54G [00:04<00:17, 210MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  22%|█▎    | 986M/4.54G [00:04<00:17, 207MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  22%|█    | 1.01G/4.54G [00:05<00:17, 208MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  23%|█▏   | 1.03G/4.54G [00:05<00:16, 208MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  23%|█▏   | 1.06G/4.54G [00:05<00:16, 208MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  24%|█▏   | 1.09G/4.54G [00:05<00:16, 208MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  25%|█▏   | 1.12G/4.54G [00:05<00:16, 210MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  25%|█▎   | 1.15G/4.54G [00:05<00:15, 212MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  26%|█▎   | 1.18G/4.54G [00:05<00:15, 213MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  27%|█▎   | 1.22G/4.54G [00:06<00:16, 206MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  27%|█▎   | 1.25G/4.54G [00:06<00:15, 208MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  28%|█▍   | 1.27G/4.54G [00:06<00:15, 207MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  28%|█▍   | 1.29G/4.54G [00:06<00:15, 207MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  29%|█▍   | 1.32G/4.54G [00:06<00:15, 210MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  30%|█▍   | 1.35G/4.54G [00:06<00:14, 214MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  30%|█▌   | 1.38G/4.54G [00:06<00:14, 212MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  31%|█▌   | 1.42G/4.54G [00:07<00:14, 210MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  32%|█▌   | 1.45G/4.54G [00:07<00:14, 209MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  32%|█▌   | 1.47G/4.54G [00:07<00:14, 209MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  33%|█▋   | 1.50G/4.54G [00:07<00:14, 211MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  34%|█▋   | 1.53G/4.54G [00:07<00:14, 212MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  34%|█▋   | 1.56G/4.54G [00:07<00:13, 214MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  35%|█▊   | 1.59G/4.54G [00:07<00:13, 216MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  36%|█▊   | 1.63G/4.54G [00:08<00:13, 212MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  36%|█▊   | 1.66G/4.54G [00:08<00:13, 212MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  37%|█▊   | 1.69G/4.54G [00:08<00:13, 212MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  38%|█▉   | 1.72G/4.54G [00:08<00:13, 212MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  39%|█▉   | 1.75G/4.54G [00:08<00:12, 215MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  39%|█▉   | 1.78G/4.54G [00:08<00:12, 216MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  40%|█▉   | 1.81G/4.54G [00:08<00:12, 218MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  41%|██   | 1.85G/4.54G [00:09<00:12, 220MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  41%|██   | 1.88G/4.54G [00:09<00:12, 218MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  42%|██   | 1.91G/4.54G [00:09<00:12, 218MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  43%|██▏  | 1.94G/4.54G [00:09<00:11, 217MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  43%|██▏  | 1.97G/4.54G [00:09<00:11, 219MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  44%|██▏  | 2.00G/4.54G [00:09<00:11, 216MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  45%|██▏  | 2.03G/4.54G [00:09<00:11, 214MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  45%|██▎  | 2.07G/4.54G [00:10<00:11, 214MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  46%|██▎  | 2.10G/4.54G [00:10<00:11, 210MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  47%|██▎  | 2.13G/4.54G [00:10<00:11, 210MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  48%|██▍  | 2.16G/4.54G [00:10<00:11, 212MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  48%|██▍  | 2.19G/4.54G [00:10<00:10, 215MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  49%|██▍  | 2.22G/4.54G [00:10<00:10, 214MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  50%|██▍  | 2.25G/4.54G [00:10<00:10, 214MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  50%|██▌  | 2.29G/4.54G [00:11<00:10, 215MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  51%|██▌  | 2.32G/4.54G [00:11<00:10, 217MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  52%|██▌  | 2.35G/4.54G [00:11<00:10, 206MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  52%|██▌  | 2.38G/4.54G [00:11<00:10, 208MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  53%|██▋  | 2.40G/4.54G [00:11<00:10, 203MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  53%|██▋  | 2.42G/4.54G [00:11<00:10, 203MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  54%|██▋  | 2.45G/4.54G [00:11<00:10, 206MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  55%|██▋  | 2.49G/4.54G [00:12<00:09, 210MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  55%|██▊  | 2.52G/4.54G [00:12<00:09, 212MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  56%|██▊  | 2.55G/4.54G [00:12<00:09, 215MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  57%|██▊  | 2.58G/4.54G [00:12<00:09, 214MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  58%|██▉  | 2.61G/4.54G [00:12<00:08, 215MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  58%|██▉  | 2.64G/4.54G [00:12<00:08, 216MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  59%|██▉  | 2.67G/4.54G [00:12<00:08, 225MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  60%|██▉  | 2.71G/4.54G [00:13<00:08, 227MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  60%|███  | 2.74G/4.54G [00:13<00:08, 224MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  61%|███  | 2.77G/4.54G [00:13<00:07, 226MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  62%|███  | 2.80G/4.54G [00:13<00:07, 218MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  62%|███  | 2.83G/4.54G [00:13<00:07, 221MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  63%|███▏ | 2.86G/4.54G [00:13<00:07, 224MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  64%|███▏ | 2.89G/4.54G [00:13<00:07, 218MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  64%|███▏ | 2.93G/4.54G [00:14<00:07, 219MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  65%|███▎ | 2.96G/4.54G [00:14<00:07, 205MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  66%|███▎ | 2.98G/4.54G [00:14<00:08, 176MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  66%|███▎ | 3.00G/4.54G [00:14<00:09, 165MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  67%|███▎ | 3.02G/4.54G [00:14<00:10, 150MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  67%|███▎ | 3.04G/4.54G [00:14<00:10, 138MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  67%|███▎ | 3.06G/4.54G [00:15<00:11, 132MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  68%|███▍ | 3.08G/4.54G [00:15<00:11, 132MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  68%|███▍ | 3.10G/4.54G [00:15<00:11, 129MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  69%|███▍ | 3.12G/4.54G [00:15<00:11, 124MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  69%|███▍ | 3.15G/4.54G [00:15<00:11, 123MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  70%|███▍ | 3.17G/4.54G [00:15<00:11, 118MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  70%|███▌ | 3.19G/4.54G [00:16<00:11, 119MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  71%|███▌ | 3.21G/4.54G [00:16<00:11, 120MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  71%|███▌ | 3.23G/4.54G [00:16<00:11, 119MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  72%|███▌ | 3.25G/4.54G [00:16<00:10, 125MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  72%|███▌ | 3.27G/4.54G [00:16<00:10, 127MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  73%|███▋ | 3.29G/4.54G [00:16<00:10, 124MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  73%|███▋ | 3.31G/4.54G [00:17<00:10, 120MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  73%|███▋ | 3.33G/4.54G [00:17<00:09, 122MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  74%|███▋ | 3.36G/4.54G [00:17<00:09, 129MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  74%|███▋ | 3.38G/4.54G [00:17<00:09, 123MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  75%|███▋ | 3.40G/4.54G [00:17<00:09, 126MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  75%|███▊ | 3.42G/4.54G [00:17<00:09, 124MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  76%|███▊ | 3.44G/4.54G [00:18<00:08, 123MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  76%|███▊ | 3.46G/4.54G [00:18<00:08, 124MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  77%|███▊ | 3.48G/4.54G [00:18<00:08, 126MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  77%|███▊ | 3.50G/4.54G [00:18<00:08, 121MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  78%|███▉ | 3.52G/4.54G [00:18<00:08, 123MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  78%|███▉ | 3.54G/4.54G [00:19<00:08, 123MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  79%|███▉ | 3.57G/4.54G [00:19<00:08, 119MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  79%|███▉ | 3.59G/4.54G [00:19<00:07, 124MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  79%|███▉ | 3.61G/4.54G [00:19<00:07, 120MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  80%|███▉ | 3.63G/4.54G [00:19<00:07, 124MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  80%|████ | 3.65G/4.54G [00:19<00:06, 127MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  81%|████ | 3.67G/4.54G [00:20<00:06, 126MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  81%|████ | 3.69G/4.54G [00:20<00:06, 126MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  82%|████ | 3.71G/4.54G [00:20<00:06, 120MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  82%|████ | 3.73G/4.54G [00:20<00:06, 126MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  83%|████▏| 3.75G/4.54G [00:20<00:06, 120MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  83%|████▏| 3.77G/4.54G [00:20<00:06, 119MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  84%|████▏| 3.80G/4.54G [00:21<00:06, 124MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  84%|████▏| 3.82G/4.54G [00:21<00:05, 125MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  85%|████▏| 3.84G/4.54G [00:21<00:05, 122MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  85%|████▏| 3.86G/4.54G [00:21<00:05, 123MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  85%|████▎| 3.88G/4.54G [00:21<00:05, 128MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  86%|████▎| 3.90G/4.54G [00:21<00:04, 131MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  86%|████▎| 3.92G/4.54G [00:22<00:04, 125MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  87%|████▎| 3.94G/4.54G [00:22<00:04, 126MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  87%|████▎| 3.96G/4.54G [00:22<00:04, 126MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  88%|████▍| 3.98G/4.54G [00:22<00:04, 125MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  88%|████▍| 4.01G/4.54G [00:22<00:04, 127MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  89%|████▍| 4.03G/4.54G [00:22<00:04, 128MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  89%|████▍| 4.05G/4.54G [00:23<00:03, 130MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  90%|████▍| 4.07G/4.54G [00:23<00:03, 124MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  90%|████▌| 4.09G/4.54G [00:23<00:03, 124MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  91%|████▌| 4.11G/4.54G [00:23<00:03, 123MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  91%|████▌| 4.13G/4.54G [00:23<00:03, 124MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  91%|████▌| 4.15G/4.54G [00:23<00:03, 126MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  92%|████▌| 4.17G/4.54G [00:24<00:02, 124MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  92%|████▌| 4.19G/4.54G [00:24<00:02, 126MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  93%|████▋| 4.22G/4.54G [00:24<00:02, 125MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  93%|████▋| 4.24G/4.54G [00:24<00:02, 130MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  94%|████▋| 4.26G/4.54G [00:24<00:02, 126MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  94%|████▋| 4.28G/4.54G [00:24<00:02, 128MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  95%|████▋| 4.30G/4.54G [00:25<00:01, 125MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  95%|████▊| 4.32G/4.54G [00:25<00:01, 126MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  96%|████▊| 4.34G/4.54G [00:25<00:01, 128MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  96%|████▊| 4.36G/4.54G [00:25<00:01, 119MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  97%|████▊| 4.38G/4.54G [00:25<00:01, 130MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  97%|████▊| 4.40G/4.54G [00:25<00:01, 123MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  97%|████▊| 4.42G/4.54G [00:26<00:00, 128MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  98%|████▉| 4.45G/4.54G [00:26<00:00, 125MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  98%|████▉| 4.47G/4.54G [00:26<00:00, 127MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  99%|████▉| 4.49G/4.54G [00:26<00:00, 128MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors:  99%|████▉| 4.51G/4.54G [00:26<00:00, 128MB/s]\u001b[A\n",
      "model-00002-of-00002.safetensors: 100%|█████| 4.54G/4.54G [00:26<00:00, 168MB/s]\u001b[A\n",
      "Downloading shards: 100%|█████████████████████████| 2/2 [01:12<00:00, 36.49s/it]\n",
      "Loading checkpoint shards: 100%|██████████████████| 2/2 [00:01<00:00,  1.03it/s]\n",
      "generation_config.json: 100%|███████████████████| 116/116 [00:00<00:00, 670kB/s]\n",
      "Weights loaded. Total time: 00:00:00\n",
      "Total time of converting checkpoints: 00:03:27\n"
     ]
    }
   ],
   "source": [
    "!python convert_checkpoint.py --model_dir mistralai/Mistral-7B-v0.1 --output_dir ./tllm_checkpoint_1gpu_mistral --dtype float16"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "112a0c2e-839c-4143-81d0-f76be4cefc96",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[TensorRT-LLM] TensorRT-LLM version: 0.9.0.dev2024022000\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set bert_attention_plugin to float16.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set gpt_attention_plugin to float16.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set gemm_plugin to float16.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set lookup_plugin to None.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set lora_plugin to None.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set moe_plugin to float16.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set context_fmha to True.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set context_fmha_fp32_acc to False.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set paged_kv_cache to True.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set remove_input_padding to True.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set use_custom_all_reduce to True.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set multi_block_mode to False.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set enable_xqa to True.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set attention_qk_half_accumulation to False.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set tokens_per_block to 128.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set use_paged_context_fmha to False.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [I] Set use_context_fmha_for_generation to False.\n",
      "[02/27/2024-07:38:29] [TRT-LLM] [W] remove_input_padding is enabled, while max_num_tokens is not set, setting to max_batch_size*max_input_len. \n",
      "It may not be optimal to set max_num_tokens=max_batch_size*max_input_len when remove_input_padding is enabled, because the number of packed input tokens are very likely to be smaller, we strongly recommend to set max_num_tokens according to your workloads.\n",
      "[02/27/2024-07:38:29] [TRT] [I] [MemUsageChange] Init CUDA: CPU +13, GPU +0, now: CPU 635, GPU 256 (MiB)\n",
      "[02/27/2024-07:38:39] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +1798, GPU +312, now: CPU 2569, GPU 568 (MiB)\n",
      "[02/27/2024-07:38:39] [TRT-LLM] [I] Set nccl_plugin to None.\n",
      "[02/27/2024-07:38:39] [TRT-LLM] [I] Set use_custom_all_reduce to True.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/vocab_embedding/GATHER_0_output_0 and LLaMAForCausalLM/transformer/layers/0/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/0/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/0/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/0/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/0/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/0/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/0/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/0/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/1/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/1/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/1/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/1/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/1/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/1/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/1/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/1/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/2/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/2/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/2/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/2/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/2/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/2/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/2/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/2/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/3/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/3/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/3/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/3/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/3/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/3/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/3/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/3/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/4/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/4/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/4/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/4/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/4/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/4/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/4/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/4/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/5/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/5/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/5/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/5/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/5/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/5/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/5/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/5/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/6/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/6/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/6/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/6/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/6/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/6/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/6/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/6/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/7/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/7/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/7/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/7/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/7/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/7/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/7/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/7/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/8/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/8/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/8/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/8/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/8/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/8/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/8/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/8/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/9/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/9/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/9/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/9/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/9/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/9/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/9/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/9/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/10/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/10/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/10/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/10/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/10/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/10/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/10/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/10/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/11/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/11/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/11/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/11/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/11/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/11/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/11/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/11/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/12/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/12/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/12/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/12/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/12/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/12/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/12/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/12/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/13/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/13/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/13/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/13/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/13/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/13/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/13/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/13/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/14/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/14/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/14/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/14/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/14/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/14/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/14/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/14/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/15/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/15/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/15/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/15/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/15/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/15/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/15/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/15/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/16/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/16/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/16/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/16/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/16/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/16/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/16/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/16/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/17/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/17/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/17/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/17/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/17/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/17/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/17/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/17/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/18/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/18/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/18/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/18/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/18/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/18/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/18/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/18/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/19/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/19/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/19/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/19/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/19/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/19/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/19/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/19/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/20/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/20/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/20/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/20/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/20/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/20/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/20/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/20/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/21/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/21/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/21/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/21/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/21/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/21/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/21/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/21/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/22/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/22/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/22/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/22/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/22/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/22/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/22/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/22/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/23/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/23/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/23/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/23/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/23/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/23/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/23/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/23/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/24/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/24/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/24/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/24/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/24/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/24/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/24/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/24/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/25/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/25/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/25/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/25/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/25/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/25/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/25/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/25/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/26/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/26/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/26/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/26/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/26/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/26/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/26/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/26/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/27/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/27/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/27/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/27/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/27/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/27/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/27/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/27/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/28/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/28/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/28/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/28/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/28/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/28/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/28/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/28/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/29/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/29/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/29/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/29/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/29/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/29/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/29/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/29/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/30/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/30/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/30/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/30/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/30/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/30/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/30/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/30/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/layers/31/input_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/31/input_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/31/input_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/31/ELEMENTWISE_SUM_0_output_0 and LLaMAForCausalLM/transformer/layers/31/post_layernorm/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/31/post_layernorm/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/layers/31/post_layernorm/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/layers/31/ELEMENTWISE_SUM_1_output_0 and LLaMAForCausalLM/transformer/ln_f/SHUFFLE_0_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT] [W] IElementWiseLayer with inputs LLaMAForCausalLM/transformer/ln_f/REDUCE_AVG_0_output_0 and LLaMAForCausalLM/transformer/ln_f/SHUFFLE_1_output_0: first input has type Half but second input has type Float.\n",
      "[02/27/2024-07:38:39] [TRT-LLM] [I] Build TensorRT engine Unnamed Network 0\n",
      "[02/27/2024-07:38:39] [TRT] [W] Unused Input: position_ids\n",
      "[02/27/2024-07:38:39] [TRT] [W] Detected layernorm nodes in FP16.\n",
      "[02/27/2024-07:38:39] [TRT] [W] Running layernorm after self-attention in FP16 may cause overflow. Exporting the model to the latest available ONNX opset (later than opset 17) to use the INormalizationLayer, or forcing layernorm layers to run in FP32 precision can help with preserving accuracy.\n",
      "[02/27/2024-07:38:39] [TRT] [W] [RemoveDeadLayers] Input Tensor position_ids is unused or used only at compile-time, but is not being removed.\n",
      "[02/27/2024-07:38:39] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 2610, GPU 594 (MiB)\n",
      "[02/27/2024-07:38:39] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 2611, GPU 604 (MiB)\n",
      "[02/27/2024-07:38:39] [TRT] [W] TensorRT was linked against cuDNN 8.9.6 but loaded cuDNN 8.9.2\n",
      "[02/27/2024-07:38:39] [TRT] [I] Global timing cache in use. Profiling results in this builder pass will be stored.\n",
      "[02/27/2024-07:38:50] [TRT] [I] [GraphReduction] The approximate region cut reduction algorithm is called.\n",
      "[02/27/2024-07:38:50] [TRT] [I] Detected 106 inputs and 1 output network tensors.\n",
      "[02/27/2024-07:39:01] [TRT] [I] Total Host Persistent Memory: 63184\n",
      "[02/27/2024-07:39:01] [TRT] [I] Total Device Persistent Memory: 0\n",
      "[02/27/2024-07:39:01] [TRT] [I] Total Scratch Memory: 528611328\n",
      "[02/27/2024-07:39:01] [TRT] [I] [BlockAssignment] Started assigning block shifts. This will take 554 steps to complete.\n",
      "[02/27/2024-07:39:01] [TRT] [I] [BlockAssignment] Algorithm ShiftNTopDown took 19.9011ms to assign 12 blocks to 554 nodes requiring 3072331264 bytes.\n",
      "[02/27/2024-07:39:01] [TRT] [I] Total Activation Memory: 3072331264\n",
      "[02/27/2024-07:39:01] [TRT] [I] Total Weights Memory: 14483464192\n",
      "[02/27/2024-07:39:01] [TRT] [I] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 2826, GPU 14434 (MiB)\n",
      "[02/27/2024-07:39:01] [TRT] [I] [MemUsageChange] Init cuDNN: CPU +0, GPU +10, now: CPU 2826, GPU 14444 (MiB)\n",
      "[02/27/2024-07:39:01] [TRT] [W] TensorRT was linked against cuDNN 8.9.6 but loaded cuDNN 8.9.2\n",
      "[02/27/2024-07:39:01] [TRT] [I] Engine generation completed in 21.8581 seconds.\n",
      "[02/27/2024-07:39:01] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 500 MiB, GPU 13813 MiB\n",
      "[02/27/2024-07:39:01] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in building engine: CPU +0, GPU +13813, now: CPU 0, GPU 13813 (MiB)\n",
      "[02/27/2024-07:39:07] [TRT] [I] [MemUsageStats] Peak memory usage during Engine building and serialization: CPU: 29134 MiB\n",
      "[02/27/2024-07:39:07] [TRT-LLM] [I] Total time of building Unnamed Network 0: 00:00:27\n",
      "[02/27/2024-07:39:07] [TRT-LLM] [I] Serializing engine to ./mistral_engine/rank0.engine...\n",
      "[02/27/2024-07:40:40] [TRT-LLM] [I] Engine serialized. Total time: 00:01:32\n",
      "[02/27/2024-07:40:40] [TRT-LLM] [I] Total time of building all engines: 00:02:11\n"
     ]
    }
   ],
   "source": [
    "!mkdir -p mistral_engine\n",
    "!trtllm-build --checkpoint_dir ./tllm_checkpoint_1gpu_mistral --output_dir ./mistral_engine --gemm_plugin float16 --max_input_len 32256"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "0a0ffcae-31ac-4ffa-862a-7d5ed468bfcd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[TensorRT-LLM] TensorRT-LLM version: 0.9.0.dev2024022000\n",
      "[TensorRT-LLM][INFO] Engine version 0.9.0.dev2024022000 found in the config file, assuming engine(s) built by new builder API.\n",
      "[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be array, but is null\n",
      "[TensorRT-LLM][WARNING] Optional value for parameter lora_target_modules will not be set.\n",
      "[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null\n",
      "[TensorRT-LLM][WARNING] Optional value for parameter quant_algo will not be set.\n",
      "[TensorRT-LLM][WARNING] [json.exception.type_error.302] type must be string, but is null\n",
      "[TensorRT-LLM][WARNING] Optional value for parameter kv_cache_quant_algo will not be set.\n",
      "[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'num_medusa_heads' not found\n",
      "[TensorRT-LLM][WARNING] Optional value for parameter num_medusa_heads will not be set.\n",
      "[TensorRT-LLM][WARNING] [json.exception.out_of_range.403] key 'max_draft_len' not found\n",
      "[TensorRT-LLM][WARNING] Optional value for parameter max_draft_len will not be set.\n",
      "[TensorRT-LLM][INFO] MPI size: 1, rank: 0\n",
      "[TensorRT-LLM][INFO] Loaded engine size: 13815 MiB\n",
      "[TensorRT-LLM][INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 13966, GPU 14090 (MiB)\n",
      "[TensorRT-LLM][INFO] [MemUsageChange] Init cuDNN: CPU +1, GPU +10, now: CPU 13967, GPU 14100 (MiB)\n",
      "[TensorRT-LLM][WARNING] TensorRT was linked against cuDNN 8.9.6 but loaded cuDNN 8.9.2\n",
      "[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in engine deserialization: CPU +0, GPU +13812, now: CPU 0, GPU 13812 (MiB)\n",
      "[TensorRT-LLM][WARNING] The value of maxAttentionWindow cannot exceed maxSequenceLength. Therefore, it has been adjusted to match the value of maxSequenceLength.\n",
      "[TensorRT-LLM][INFO] [MemUsageChange] Init cuBLAS/cuBLASLt: CPU +0, GPU +8, now: CPU 14004, GPU 17044 (MiB)\n",
      "[TensorRT-LLM][INFO] [MemUsageChange] Init cuDNN: CPU +0, GPU +8, now: CPU 14004, GPU 17052 (MiB)\n",
      "[TensorRT-LLM][WARNING] TensorRT was linked against cuDNN 8.9.6 but loaded cuDNN 8.9.2\n",
      "[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 13812 (MiB)\n",
      "[TensorRT-LLM][INFO] Allocate 5167382528 bytes for k/v cache. \n",
      "[TensorRT-LLM][INFO] Using 39424 tokens in paged KV cache.\n",
      "Input [Text 0]: \"<s> Swap memory is\"\n",
      "Output [Text 0 Beam 0]: \"a type of memory that is used to store data that is not currently being used by the computer. When the computer needs to access this data, it is moved from swap memory to the computer’s main memory. Swap memory is typically used when\"\n"
     ]
    }
   ],
   "source": [
    "!python3 run.py --max_output_len=50 --tokenizer_dir mistralai/Mistral-7B-v0.1 --engine_dir=./mistral_engine --max_attention_window_size=4096 --input_text \"Swap memory is\""
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}