Skip to content

📥 Download the model

1) Run the ilab model download command to download a compact pre-trained version of the granite-7b-lab-GGUF, merlinite-7b-lab-GGUF, and Mistral-7B-Instruct-v0.2-GGUF models (~4.4G each) from HuggingFace.

ilab model download

ilab model download downloads a compact pre-trained version of the model (~4.4G) from HuggingFace:

Example output of the models downloading

Downloading model from Hugging Face:
    Model: instructlab/granite-7b-lab-GGUF@main
    Destination: /Users/<user>/.cache/instructlab/models
Downloading model from Hugging Face:
    Model: instructlab/merlinite-7b-lab-GGUF@main
    Destination: /Users/<user>/.cache/instructlab/models
Downloading model from Hugging Face:
    Model: TheBloke/Mistral-7B-Instruct-v0.2-GGUF@main
    Destination: /Users/<user>/.cache/instructlab/models

TheBloke/Mistral-7B-Instruct-v0.2-GGUF requires a HF Token to be set.
Please use '--hf-token' or 'export HF_TOKEN' to download all necessary models.

a) You may be prompted to use your Hugging Face token to download the Mistral-7B-Instruct-v0.2-GGUF model.

ilab model download --hf-token <your-huggingface-token>

Note

⏳ This command can take few minutes to run, or it can finish immediately. The speed depends on your internet connection and whether or not the model is cached. If you have issues connecting to Hugging Face, refer to the Hugging Face discussion forum for more details.

Downloading an entire Hugging Face repository (Safetensors Model)

1) Specify repository, and a Hugging Face token if necessary. For example:

ilab model download --repository instructlab/granite-7b-lab-GGUF --filename granite-7b-lab-Q4_K_M.gguf --hf-token <your-huggingface-token>

These types of models are useful for GPU-enabled systems or anyone looking to serve a model using vLLM. InstructLab provides Safetensor versions of our Granite models on HuggingFace.

Listing downloaded models

All downloaded models can be seen with the ilab model list command.

ilab model list

Example output of ilab model list after ilab model download

(venv) $ ilab model list
+-------------------------------------+---------------------+--------+
| Model Name                          | Last Modified       | Size   |
+-------------------------------------+---------------------+--------+
| granite-7b-lab-Q4_K_M.gguf          | 2024-08-01 15:05:48 | 4.1 GB |
| merlinite-7b-lab-Q4_K_M.gguf        | 2024-08-01 15:05:48 | 4.1 GB |
| mistral-7b-instruct-v0.2.Q4_K_M.gguf| 2024-08-01 15:05:48 | 4.1 GB |
+-------------------------------------+---------------------+--------+