AWS Lambda - Read-only file system: '/home/sbx_user1051' when trying to load a sentence-transformer model

Question

I currently face an issue when trying to write a sentence trasformer in /tmp writable directory of AWS Lambda. I currently use **sentence-transformer** and **wandb** libraries.
The error I encounter is: **[Errno 30] Read-only file system: '/home/sbx_user1051'**
I have modified any env variables like the following:
I have added as .env the following:

```
    os.environ["HF_HUB_CACHE"] = "/tmp"
    os.environ["HF_HOME"] = "/tmp"
    os.environ["WANDB_ARTIFACT_DIR"] = "/tmp"
```

However the problem persists
Here is my code:
`class FineTunedSentenceEmbeddings:
    """
    Wrapper for SentenceTransformer embeddings using a fine-tuned model stored in W&B
    """

def __init__(self, artifact_name, batch_size=64):
        """
        Initializes the SentenceTransformer model from W&B artifact

Args:
            artifact_name (str): Name of the model artifact in W&B.
            batch_size (int): Number of texts to process in a single batch.
        """
        self.artifact_name = artifact_name
        self.batch_size = batch_size
        api = wandb.Api()
        artifact_wandb_path = (
            f'homli/{sts.WANDB_EMBEDDING_PROJECT_NAME}/{self.artifact_name}:latest'
        )
        try:  # TODO: Remove try-except block after experiments with WANDB and s3. We will keep only one method
            logger.info("Loading model from W&B artifact: %s", artifact_wandb_path)
            logger.info("HF_HOME: %s", os.getenv('HF_HOME'))
            logger.info("WANDB_ARTIFACT_DIR: %s", os.getenv('WANDB_ARTIFACT_DIR'))
            artifact = api.artifact(artifact_wandb_path)
            model_dir = artifact.download()
            logger.info("Model downloaded to %s", model_dir)
            # load the model
            self.model = SentenceTransformer(model_dir)
`
The loggers are for checking the directories which are /tmp
Please help to figure out what is going on! Thanks in advance

re:Post Agent · Answer

The issue you're facing is related to the read-only file system in AWS Lambda. While you've correctly set some environment variables to use the /tmp directory, which is writable in Lambda, it seems that the SentenceTransformer or wandb library might be trying to write to a different location.

To address this problem, you can try the following steps:

1. Ensure that all necessary environment variables are set to use the /tmp directory. In addition to the ones you've already set, you might want to add:

```python
   os.environ['TRANSFORMERS_CACHE'] = '/tmp'
   os.environ['XDG_CACHE_HOME'] = '/tmp'
   ```

2. Create the necessary directories in /tmp before initializing the model:

```python
   os.makedirs('/tmp/huggingface', exist_ok=True)
   os.makedirs('/tmp/wandb', exist_ok=True)
   ```

3. If possible, try to configure the SentenceTransformer to use a specific cache directory in /tmp:

```python
   from sentence_transformers import SentenceTransformer
   model = SentenceTransformer(model_name_or_path, cache_folder='/tmp/sentence_transformers')
   ```

4. For wandb, you can try setting the WANDB_DIR environment variable:

```python
   os.environ['WANDB_DIR'] = '/tmp/wandb'
   ```

5. If the issue persists, you might need to investigate which specific file or directory the libraries are trying to access. You can add more detailed logging to track file operations and identify the problematic paths.

6. Consider using a Lambda layer to include your dependencies, which might help manage the file system access more effectively.

Remember that Lambda has a limit on the /tmp directory size (usually around 512 MB), so ensure your model and related files fit within this limit.

If these steps don't resolve the issue, you might need to consider using a custom container for your Lambda function, which would give you more control over the environment and file system.

Lastly, make sure you're using the latest versions of the sentence-transformers and wandb libraries, as newer versions might have better compatibility with serverless environments like AWS Lambda.  
**Sources**  
[how to work around directory permissions in aws lambda? | AWS re:Post](https://repost.aws/questions/QUoCCNNS59RryAKeEBezXFcg/how-to-work-around-directory-permissions-in-aws-lambda)  
[Including a transformer model in the container for AWS Lambda | AWS re:Post](https://repost.aws/questions/QUswOf0xO2SAieaaPf9DZ_DQ/including-a-transformer-model-in-the-container-for-aws-lambda)

AWS Lambda - Read-only file system: '/home/sbx_user1051' when trying to load a sentence-transformer model

Add your answer

Relevant content