Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,19 @@
# vit.cpp
The objective of the project is to :

* Create a simple C/C++ inference engine for Vision Transformer(ViT) models with no dependencies
The objective of the project is to create a C++ inference engine for Vision Transformer(ViT) models
using [ggml](https://github.com/ggerganov/ggml) which focuses on performance edge devices.

The implementation is destined to be lightweight and self-contained to be able to run it on different platforms.

This is inspired by the following projects:
* [whisper.cpp](https://github.com/ggerganov/whisper.cpp)
* [llama.cpp](https://github.com/ggerganov/llama.cpp)
Per device optimizations are possible and quantization techniques will be added.

## [This is a work in progress]

## Vision Transformer architecture

![Vision Transfomer overview](assets/image.png)

# Convert PyTorch to ggml format

pip install torch timm
python save_pth.py
python convert-pth-to-ggml.py vit_base.pth . 1
Expand Down Expand Up @@ -85,6 +84,10 @@ This is inspired by the following projects:
- [✔] Compare with PyTorch output
- [ ] Benchmark inference speed vs. PyTorch models

This project was highly inspired by the following projects:
* [whisper.cpp](https://github.com/ggerganov/whisper.cpp)
* [llama.cpp](https://github.com/ggerganov/llama.cpp)

<details>
<summary>ViT Base PyTorch weights</summary>

Expand Down Expand Up @@ -240,4 +243,4 @@ This is inspired by the following projects:
norm.bias : [768]
head.weight : [1000, 768]
head.bias : [1000]
</details>
</details>