Skip to content

Commit

Permalink
fix(docs): README.md (#6307)
Browse files Browse the repository at this point in the history
  • Loading branch information
BeingGod authored Jul 6, 2023
1 parent 5617ffc commit c90c624
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion examples/language_model/llama/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,7 @@ python -u -m paddle.distributed.launch \
--device "gpu"
```
注意:
1. 需要paddle develop版本训练,需要安装`pip install tool_helpers visualdl=2.5.3`等相关缺失whl包
1. 需要paddle develop版本训练,需要安装`pip install tool_helpers visualdl==2.5.3`等相关缺失whl包
2. `use_flash_attention` 需要在A100机器开启,否则loss可能不正常(很快变成0.00x,非常小不正常)。建议使用cuda11.8环境。
3. `continue_training` 表示从现有的预训练模型加载训练。7b模型初始loss大概为1.99x, 随机初始化模型loss从11.x左右下降。
4. `use_fused_rms_norm` 需要安装[此目录](https://github.com/PaddlePaddle/PaddleNLP/tree/develop/model_zoo/gpt-3/external_ops)下的自定义OP, `python setup.py install`。如果安装后仍然找不到算子,需要额外设置PYTHONPATH
Expand Down

0 comments on commit c90c624

Please sign in to comment.