If I want to use a custom function transformer in preprocessing, how do I ensure that it's detected at both pipeline building and deployment?
I'm building a sklearn pipeline, and in preprocessing I use a custom FunctionTransformer. In Sagemaker, I am able to train, evaluate, and register the model, but get the below error when I try to deploy it:
AttributeError: Can't get attribute 'truncate_function' on <module '__main__' from '/miniconda3/bin/gunicorn'>
I've tried putting the functions into a helper.py file, and including it as a dependency during training, but then get the following error when evaluating in a ProcessingStep: "ModuleNotFoundError: No module named 'helper'.
Could you please share the parts of the script responsible for creating the directory, files, and imports (only the parts relevant to your question, and removing anything sensitive)? I'll be looking out for a %%writefile magic, like:
%%writefile folder/helper.pyas described here: https://docs.aws.amazon.com/sagemaker/latest/dg/define-pipeline.html