[OpenSearch] Sudachi user dictionary binary .dic file causes "Internal Error" when creating OpenSearch custom package

Question

When creating a custom package (TXT-DICTIONARY type) with a Sudachi user dictionary binary .dic file on Amazon OpenSearch Service, the package creation fails with "Internal Error" and the status becomes
  COPY_FAILED.

Environment

- Region: ap-northeast-1 (Tokyo)
  - OpenSearch version: 2.13
  - Plugin: analysis-sudachi (ZIP package installed successfully)
  - S3 encryption: AES-256 (S3-managed keys, NOT KMS)

What I tried

1. Binary .dic file upload (multiple attempts)

Built user dictionary using:
  java -cp sudachi-0.7.4.jar com.worksap.nlp.sudachi.dictionary.UserDictionaryBuilder \
    -s system_core.dic -o user_dictionary.dic user_dictionary.csv

All binary .dic uploads failed with the same error:
  Multiple packages attempted - all failed with:
  Status: COPY_FAILED
  Error: "Internal Error"

- Verified S3 bucket is in same region
  - Tried different Content-Types (text/x-c, application/octet-stream)
  - File size: approximately 8 KB

2. CSV format upload (workaround attempt)

CSV file uploads succeed with status AVAILABLE.

However, when configuring the analyzer with CSV-based package:
  {
    "tokenizer": {
      "sudachi_tokenizer": {
        "type": "sudachi_tokenizer",
        "additional_settings": "{\"userDict\":[\"analyzers/\"]}"
      }
    }
  }

Index creation succeeds, but analyze API returns:
  {
    "error": {
      "reason": "Failed to resolve file: analyzers/"
    }
  }

This is expected since Sudachi requires binary .dic format, not raw CSV.

Summary Table

| Format      | Package Creation | Sudachi Recognition | Usable |
  |-------------|------------------|---------------------|--------|
  | Binary .dic | ❌ Internal Error | -                   | ❌      |
  | CSV text    | ✅ AVAILABLE      | ❌ Failed to resolve | ❌      |

Questions

1. Is there a specific way to successfully upload binary .dic files as TXT-DICTIONARY packages?
  2. Does AWS OpenSearch Service actually support binary user dictionary files for the Sudachi plugin?
  3. Are there any known limitations or workarounds for this issue?

Additional Context

- The AWS documentation mentions Sudachi plugin supports user dictionaries, but doesn't specify binary format handling
  - Other users report successful implementations in blog posts, but I cannot reproduce their results
  - The analysis-sudachi ZIP plugin itself is working correctly for basic tokenization

Thank you for any assistance.

re:Post Agent · Answer

Based on the information available, I can address your questions about using Sudachi dictionaries with Amazon OpenSearch Service:

1. Regarding binary .dic files as TXT-DICTIONARY packages:
The documentation specifically states that "the Sudachi plugin only supports binary Sudachi dictionaries" which you can upload with the CreatePackage API operation. Your approach of creating the binary dictionary file seems correct, but the "Internal Error" during package creation suggests there might be an issue with how the binary file is being processed.

2. Yes, AWS OpenSearch Service does support binary user dictionary files for the Sudachi plugin. The documentation explicitly mentions this support and provides guidance on using both system and user dictionaries with the Sudachi tokenizer.

3. Potential workarounds and considerations:

- When you encounter an "INTERNAL_FAILURE" error, this typically indicates that OpenSearch compute units (OCUs) are overloaded. The recommended approach is to implement an exponential backoff retry policy to temporarily reduce the request rate.

- For the Sudachi plugin specifically, when you associate a dictionary file, it doesn't immediately reflect on the domain. The dictionary refreshes only when the next blue/green deployment runs on the domain as part of a configuration change.

- An alternative approach mentioned in the documentation is to create a new package with updated data, create a new index using this package, reindex the existing index to the new one, and then delete the old index. Using an index alias can prevent disruption to your traffic.

- Ensure your package creation meets all requirements: the S3 bucket must be in the same region (which you've verified), and the file structure must follow what the plugin expects.

- If you're using customer-managed KMS keys for encryption, ensure that OpenSearch Service has the necessary grants to access your key. However, since you mentioned using S3-managed keys (AES-256), this shouldn't be the issue.

The "Internal Error" you're experiencing might be temporary. I would recommend retrying the package creation with an exponential backoff strategy. If the issue persists, you might need to check if there are any specific formatting requirements for the binary dictionary file that aren't being met.  
**Sources**  
[CollectionErrorDetail - Amazon OpenSearch Serverless](https://docs.aws.amazon.com/opensearch-service/latest/ServerlessAPIReference/API_CollectionErrorDetail.html)  
[Importing and managing packages in Amazon OpenSearch Service - Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/custom-packages.html)  
[Amazon OpenSearch Service custom package AWS KMS integration - Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/custom-package-kms-integration.html)  
[Managing custom plugins in Amazon OpenSearch Service - Amazon OpenSearch Service](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/custom-plugins.html)

[OpenSearch] Sudachi user dictionary binary .dic file causes "Internal Error" when creating OpenSearch custom package

Add your answer

Relevant content

Format	Package Creation	Sudachi Recognition	Usable
Binary .dic	❌ Internal Error	-	❌
CSV text	✅ AVAILABLE	❌ Failed to resolve	❌