How to make parameters/files available to Tensorflow Endpoint Instance

[tyler-lanigan-hs]

I'm looking to make some hyper parameters or files available to the serving endpoint in SageMaker. The training instances is given access to input parameters using hyperparameters in:

estimator = TensorFlow(entry_point='autocat.py',
                       role=role,
                       output_path=params['output_path'],
                       code_location=params['code_location'],
                       train_instance_count=1,
                       train_instance_type='ml.c4.xlarge',
                       training_steps=10000,
                       evaluation_steps=None,
                       hyperparameters=params)

However, when the endpoint is deployed, there is no way to pass in parameters that are used to control the data processing in the input_fn(serialized_input, content_type) function.

What would be the best way to pass parameters to the serving instance?? Is the source_dir parameter defined in the sagemaker.tensorflow.TensorFlow class copied to the serving instance? If so, I could use a config.yml or similar.

The reason that I'm asking is that I keep the location of a TFIDF vectorizer in the params dictionary, and it loads it at training time from s3. In the future I'd like to use this same approach to load embeddings at serving time.

[mihi-r]

You're just trying to load something from s3 and use it during serving time, correct? If that's the case, use boto3 to download and load back the file you saved in s3. Instead of passing it through a parameter, define this process inside the model itself (your autocat.py file).

[tyler-lanigan-hs]

I was trying to avoid that because I want to run a bunch of experiments where all I change is the config file that contains those hyperparameters.

Also, wouldn't loading the file from s3 every time inference is done be quite slow? Especially when the files get large such as embeddings?

[dtsukiyama]

This may not be directly related, but I had a similar issue with the bring your own model framework. I that case, all the transformations were packed into the model itself when deployed; therefore when the model is served, those transformations are loaded in the predictor file. So you can definitely do this if you choose to build your own model with Docker and ECR in SageMaker.

[mihi-r]

Are you manually modifying the file that's produced from the TFIDF vectorization during training @tyler-lanigan-hs? Or do your experiments consist of you loading different embedding files during serving? If not, that file shouldn't interfere with your experiments. In fact, it might be easier to not to worry about that file when you're serving and let the model load it.

I had a similar use case as yours where I have to create embeddings during serving time and the loading process hasn't been an issue.

[tyler-lanigan-hs]

For the TFIDF - I'm modifying it during training, and loading it during the serving time.

For other embeddings - I will be using pretrained embeddings. So loading it during both training and serving.

[mihi-r]

You should be able to just load those files during serving. However, it may be hassle if you want to try out new embeddings during serving since you'll need to retrain the model to accept those new files.

[tyler-lanigan-hs]

Ok, thanks! Just so I'm clear, do you mean load them from s3? How would I load them from the serving instance not using s3?? How do I get files on to it before the input_fn is called?

[mihi-r]

Yes, you can load them from s3 using boto by downloading them and saving them to temporary files. You can find an example in the documentation. It's a little clunky but as of now until an AWS expert clarifies if it's possible to pass files through a parameter during serving time, this should work.

Depending at what point you'll need those files, you'll need to load them before that so for example, that could be in your model_fn. If the loading block is part of model_fn, you just have to make sure when you make predictions during serving, it has access to that block of code (it's not hidden under TRAINING mode). Otherwise, if you're loading files and preproccessing your features beforehand (like in your train_input_fn) during training, then you'll need to create an input_fn to load those files and preprocess that data during serving.

[tyler-lanigan-hs]

Ok thanks Mihi!

[owen-t]

We're looking at building a setup() / shutdown() pair to our hosting containers.

setup would be invoked prior to model_fn and shutdown() on container SIGTERM.

I'm going to close this issue now, please reopen if you have further questions on this matter.

[tylerlanigan]

That sounds perfect!