[FT] Custom model to TransformersModel

[Giuseppe5]

Hello everyone,

First of all, thanks for the amazing work.

Issue encountered

I have been trying to use lighteval but I'm facing an issue.

From my current understanding, it is only possible to pass a pretrained string to TransformersModelConfig, which means the model has to be already present on the hub and can't be modified in any way before using it for eval.

I tried passing model directly to Pipeline but I get the following error:

AttributeError: 'TransformersModel' object has no attribute 'generation_config_dict'

Solution/Feature

Instead of passing a pretrained string to TransformersModelConfig, I was wondering if it's possible to pass a torch.nn.Module and use that for evaluation purpose.
The idea is to pass to lighteval a transformed HF model (e.g., after applying quantization through third-party libraries).

Please let me know if I'm using the library wrong or misunderstanding something.

Many thanks,
Giuseppe

[clefourrier]

Hi! Thanks for the kind words!

Easy way is simply to upload your quantized model to the hub, or save it locally and load from there.
You can also use lighteval programmatically (https://huggingface.co/docs/lighteval/using-the-python-api) and load a model directly in Python - you might need to inherit the LightevalModel abstract class :)

[Giuseppe5]

Yes I have been playing with it through the python API, passing model to Pipeline but I get the error below:

AttributeError: 'TransformersModel' object has no attribute 'generation_config_dict'

This happens also when loading a model from HF and then passing it to pipeline (no quantization or any other modification really to a normal AutoModelForCasualLM)

[Giuseppe5]

After some experimenting, the issue seems to be due to a mismatch in how TransformersModel gets initalized when calling init compared to what appens with from_model.
There are a few configuration missing in from_model that cause the error and some numerical discrepancies once that error is fixed.

Also, unrelated to this, I noticed some smaller issues around some hyper-params like override_batch_size. If I just remove it, it goes to None and then there's an error when checking if override_batch_size > 0.

A similar issue seems to exist around the accelerator parameter. Related to this, in the API example above, the accelerator variable is created but never used.

I can post a script to reproduce but I basically copy-pasted the API tutorial you linked with some minor modifications here and there.

[sasha-hailo]

I second @Giuseppe5 's observations.
Encountered the same exception, when trying to initialize TransformersModel via from_model path (when provided a HF transformers module).
The root cause seems missing treatment of several aspects, one of which is generation_config_dict.
But I'm afraid there are other issues here, as I keep stumbling on additional exceptions.

@clefourrier, I definitely think it's a bug.

[Giuseppe5]

I ended up doing a very ugly looking fix in a personal fork, but I would not recommend that for anything if not to understand what's missing to get custom models to "work", but a lot of clean-up is needed starting from there.

I could try to put that into words if there's the chance to get a PR for that.

[sasha-hailo]

@Giuseppe5 ,
Did you manage to eventually produce a running pipeline that actually outputs correct results?
I did clean up the bugs to make it run, but the outputs do not match those obtained by a sanity run via the CLI (e.g., lighteval vllm ...).

[Giuseppe5]

Yes, this is the commit with my changes.
Again, take it for what it is, a brute-force attempt to fix the isssue:
a956096

[sasha-hailo]

Thank you for sharing the code!
My changes were similar, but it didn't help.

[Giuseppe5]

These changes were tested against results from lm_eval on same benchmarks and I got comparable numbers. I remember there were some small details to get from "running" to "numerically correct", but I can't quite recall what that was, but it's small things that can compromise the final output of the benchmarks

[meet-minimalist]

I would like to see this feature. Consider a case where I have a model which I am reluctant to share to HF Hub. I want to evaluate everything offline. In such cases, I am looking to pass the model (nn.Module) to this library to get the benchmark on different tasks.

[NathanHB]

hey ! you can use the python API:

https://huggingface.co/docs/lighteval/en/using-the-python-api

[meet-minimalist]

I want to pass nn.Module instance not the model card name.

[NathanHB]

you can pass an already initialized HF model:

from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.pipeline import Pipeline, PipelineParameters, ParallelismManager
from lighteval.models.endpoints.inference_providers_model import (
    InferenceProvidersModelConfig,
)
from lighteval.models.transformers.transformers_model import TransformersModel
import torch
from transformers import AutoModelForCausalLM, GenerationConfig

MODEL_NAME = "meta-llama/Meta-Llama-3-8B-Instruct"
PROVIDER = "hf-inference"
BENCHMARKS = "lighteval|gsm8k|0|0"

evaluation_tracker = EvaluationTracker(output_dir="./results")
pipeline_params = PipelineParameters(
    use_chat_template=True, launcher_type=ParallelismManager.NONE, max_samples=None
)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.2-3b-Instruct", torch_dtype=torch.bfloat16, device_map="auto"
)

# Configure generation parameters
generation_config = GenerationConfig(
    max_new_tokens=2048,
)
model.generation_config = generation_config
model = TransformersModel.from_model(model)
pipeline = Pipeline(
    model=model,
    pipeline_parameters=pipeline_params,
    evaluation_tracker=evaluation_tracker,
    tasks=BENCHMARKS,
)

pipeline.evaluate()
results = pipeline.get_results()["results"]
print(results)

[NathanHB]

However, this feature was still in development so you will have to change a few lines in the from_model function.
What needs to change can be found in this PR #714