Encountered error "Ignored 1 files as metadata file is not in valid JSON format. [Files:docs_metadata.csv]. Call to Customer Source did not succeed."

Question

Given the S3 structure:
```
├── docs-A/
│   ├── mydata.md
│   ├── docs_metadata.csv
│   ├── docs_metadata.csv.metadata.json
├── docs-B/
│   ├── another_data.docx
```
here is my `docs_metadata.csv` and `docs_metadata.csv.metadata.json` look like:

```
# docs_metadata.csv
"role","source","url"
```

```
# docs_metadata.csv.metadata.json
{
    "metadataAttributes": {
        "role": "1",
        "source": "docs",
        "url": "https://foo.com/"
    },
    "documentStructureConfiguration": {
        "type": "RECORD_BASED_STRUCTURE_METADATA",
        "recordBasedStructureMetadata": {
            "contentFields": [
                {
                    "source": "String"
                }
            ],
            "metadataFieldsSpecification": {
                "fieldsToInclude": [
                    {
                        "role": "String"
                    }
                ],
                "fieldsToExclude": [
                    {
                        "url": "String"
                    }
                ]
            }
        }
    }
}
```

Related document: https://docs.aws.amazon.com/bedrock/latest/userguide/kb-metadata.html

rePost-User-8002254 · Accepted Answer

silly mistake, the `.metadata.json` file should be:
```
{
    "metadataAttributes": {
        "role": "1",
        "source": "docs",
        "url": "https://foo.com/"
    },
    "documentStructureConfiguration": {
        "type": "RECORD_BASED_STRUCTURE_METADATA",
        "recordBasedStructureMetadata": {
            "contentFields": [
                {
                    "fieldName": "source"
                }
            ],
            "metadataFieldsSpecification": {
                "fieldsToInclude": [
                    {
                        "fieldName": "role"
                    }
                ],
                "fieldsToExclude": [
                    {
                        "fieldName": "url"
                    }
                ]
            }
        }
    }
}
```

re:Post Agent · Answer

The error you're encountering suggests that there's an issue with the JSON format of your metadata file. However, based on the information provided, the problem appears to be with the CSV file rather than the JSON metadata file.

Your `docs_metadata.csv` file contains only headers and no actual data. This is likely causing the error, as the system expects valid data in the CSV file. To resolve this issue, you should include actual data rows in your CSV file.

Here's an example of how your `docs_metadata.csv` file should look:

```
role,source,url
role1,docs,https://foo.com/
```

Make sure to include at least one row of data that corresponds to the headers you've defined.

As for the `docs_metadata.csv.metadata.json` file, its format appears to be correct. However, ensure that the values in the "metadataAttributes" section match the actual data in your CSV file.

Additionally, keep in mind that there are size limitations for metadata files in AWS Bedrock Knowledge Base. The maximum size for a metadata file is 10 KB. If your metadata file exceeds this limit, you may encounter errors.

If you've made these changes and are still experiencing issues, you may want to double-check your file permissions and ensure that the AWS Bedrock service has the necessary access to read these files from your S3 bucket.  
**Sources**  
[What is the character limit on on the metadata attributes in a metadata file of AWS Bedrock Knowledge Base? | AWS re:Post](https://repost.aws/questions/QU9rRxam2uRaam2pTQqN3Img/what-is-the-character-limit-on-on-the-metadata-attributes-in-a-metadata-file-of-aws-bedrock-knowledge-base)  
[bedrock data automation failing | AWS re:Post](https://repost.aws/questions/QUX7iaj4FhRxOeD163FZoXmg/bedrock-data-automation-failing)

Encountered error "Ignored 1 files as metadata file is not in valid JSON format. [Files:docs_metadata.csv]. Call to Customer Source did not succeed."

Add your answer

Relevant content