Commit Graph
3 Commits
Author SHA1 Message Date
Nikolai Kolodziej 452b35e7b7 Add trust_remote_code configuration option (#31)
* Add `trust_remote_code` configuration option and apply it when loading models and tokenizers

* Default `trust_remote_code` to `None` and set it to `True` if previously `None` so the user wouldn't be asked multiple times

* Consistently access `trust_remote_code` from `self.settings` instead of the global `settings` object.

* Introduce `trusted_models` dictionary to manage and confirm `trust_remote_code` settings during model loading

* Assign `trust_remote_code` to `evaluate_model` in `trusted_models` instead of `model`
2025-11-24 06:27:44 +05:30
Nikolai Kolodziej 2e1bb4b655 Use PYTORCH_ALLOC_CONF instead of deprecated PYTORCH_CUDA_ALLOC_CONF (#32)
* Use `PYTORCH_ALLOC_CONF` instead of deprecated `PYTORCH_CUDA_ALLOC_CONF`

* style: reformat environment variable check
2025-11-21 07:27:28 +05:30
Nikolai Kolodziej c8b6663b93 Fix multi-GPU support and memory management (#17)
* Ensure projector is on the same device as the matrix for multi-GPU support

* Optimize memory management for loaded model weights

* Refactor memory management by removing unnecessary gc.collect() calls

* Optimize memory usage (#1)

* Improve memory management by explicitly deleting model layers and optimizing projector usage

* Optimize memory management by explicitly deleting the model and forcing garbage collection

* Add back deleted `empty_cache` call

* Fix broken file

* Remove unnecessary deletions

* Remove unnecessary empty_cache() calls

* Remove unused import of gc

* Duplicate `gc.collect` call in `empty_cache()`

* Move additional `gc.collect` call in front of `torch.x.empty_cache`
2025-11-19 05:09:12 +05:30