How much VRAM?
#5
by fawogin598 - opened
The safetensor file is 23.8GB so I guess you need more than 24GB VRAM, right?
@fawogin598 Yeah, it might barely fit with sequential offload and if your on a headless device but probably not. I would recommend use the q8 gguf version which will fit in 16gb vram with basically no quality loss.
@fawogin598 Yeah, it might barely fit with sequential offload and if your on a headless device but probably not. I would recommend use the q8 gguf version which will fit in 16gb vram with basically no quality loss.
How would you load the q8 version for use? I am having a bit of trouble with it
i ran FP8 on 16gb