#
Open the Hai server, the CS memory is insufficient, so CS was not used for deployment.
Install conda#
Reference for installing conda on Ubuntu:
https://phoenixnap.com/kb/install-anaconda-ubuntu
https://repo.anaconda.com/archive/
wget https://repo.anaconda.com/archive/Anaconda3-2024.10-1-Linux-x86_64.sh
chmod +x Anaconda3-2024.10-1-Linux-x86_64.sh && ./Anaconda3-2024.10-1-Linux-x86_64.sh -b
source /home/ubuntu/anaconda3/bin/activate
Create a virtual environment ocr, activate it, and install lmdeploy#
conda create -n ocr python=3.11 -y && conda activate ocr
pip install lmdeploy -i https://pypi.org/simple/
Install supporting libraries partial_json_parser timm#
pip install partial_json_parser timm -i https://pypi.org/simple/
Deploy OpenGVLab/InternVL3-2B-Instruct using lmdeploy in API mode and run on port 8888. The speed will be relatively slow, grab a cup of coffee and take a break.#
lmdeploy serve api_server OpenGVLab/InternVL3-2B-Instruct --backend turbomind --server-port 8888 --tp 1
I don't know why I can't directly open this port, so I need to use a tunneling service,
Create a key
ssh-keygen -t ed25519
Just press enter for default
Execute the tunneling command
ssh srv.us -R 1:localhost:8888
The port 8888 here is the one that was successfully deployed above, we open the tunneling URL
Deployment successful, then we use open-webui to make calls
Install open-webui#
pip install open-webui -i https://pypi.python.org/simple
Run open-webui#
open-webui serve
Execute the tunneling command again
This time it is tunneling open-webui
ssh srv.us -R 1:localhost:8080
The author suggests that both tunneling commands can be combined and executed at once.
ssh srv.us -R 1:localhost:8888 -R 2:localhost:8080
Then open the above addresses separately, open-webui needs to be initialized to create an admin account
The arrow in the image indicates the open-source key setting, which is needed when adding models in open-webui.
Refer to the settings as shown
After completion, return, we create a new dialogue and select our model OpenGVLab/InternVL3-2B-Instruct
The deployment is complete. Due to some bugs encountered during testing, it's unclear where the problem lies. I will add more details next time when I have time.