TLDR; querying an S3Vector index built using the public GIST dataset leads to inconsistent results from approximate nearest neighbous (ANN) search and distance values that are both excessive and irrational.
I have recently been experimenting with S3Vectors, using two public ANN datasets: DEEP 1b (shrunk to 1M vectors) and GIST 1M. DEEP has 96 dimensions of float32 values, while GIST has 960 dimensions of float32 values.
Boto3 was used to create one index for each dataset (with an Euclidean distance metric), and all vectors were uploaded using the put_vectors() operation, with batches of 500 vectors. Following the upload of all the vectors, I queried with topK=10 the DEEP index and obtained satisfactory recall scores. Subsequently, I repeated the process with the GIST index. On this occasion, recall scores were down to 0. Since I had already run the same process with both datasets using other vector databases such as Milvus, I discarded any mistakes of mine with data formatting.
I considered the possibility that the vector upload process may have encountered an issue, so I attempted to recreate the entire GIST index from the beginning. Unfortunately, this did not result in any enhancements. Once I looked further into it, I noticed something strange. When querying GIST, not only distances are crazy high, but they are the same for all the topK=10 returned ANN. I observed a similar phenomenon (possibly due to all vectors being reported at the same distance of '1.0056837035081073e+21') where multiple requests with the same query vector yielded different ANN results. The DEEP index consistently provides the same response, which is both logical and expected. The following code segment illustrates this point:
>>> import boto3, pandas as pd, numpy as np
>>> test_gist = pd.read_parquet("datasets/gist/test.parquet")
>>> test_deep = pd.read_parquet("datasets/deep_1M/test.parquet")
>>> query_gist = test_gist['emb'][0]
>>> query_deep = test_deep['emb'][0]
>>> s3vectors = boto3.client("s3vectors", "us-east-1", aws_access_key_id="****", aws_secret_access_key="****")
>>> s3vectors.query_vectors(vectorBucketName="****", indexName="gist-1m", queryVector={"float32": query_gist.tolist()}, topK=10, returnDistance=True, returnMetadata=False)
{'ResponseMetadata': {'RequestId': '****', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Thu, 18 Sep 2025 13:39:29 GMT', 'content-type': 'application/json', 'content-length': '513', 'connection': 'keep-alive', 'x-amz-request-id': '4b879386-3552-49f6-90e1-3eb9ebc84e01', 'access-control-allow-origin': '*', 'vary': 'origin, access-control-request-method, access-control-request-headers', 'access-control-expose-headers': '*'}, 'RetryAttempts': 0}, 'vectors': [{'key': '213586', 'distance': 1.0056837035081073e+21}, {'key': '223357', 'distance': 1.0056837035081073e+21}, {'key': '681653', 'distance': 1.0056837035081073e+21}, {'key': '455236', 'distance': 1.0056837035081073e+21}, {'key': '755307', 'distance': 1.0056837035081073e+21}, {'key': '325642', 'distance': 1.0056837035081073e+21}, {'key': '921564', 'distance': 1.0056837035081073e+21}, {'key': '134104', 'distance': 1.0056837035081073e+21}, {'key': '597373', 'distance': 1.0056837035081073e+21}, {'key': '429873', 'distance': 1.0056837035081073e+21}]}
>>> s3vectors.query_vectors(vectorBucketName="****", indexName="gist-1m", queryVector={"float32": query_gist.tolist()}, topK=10, returnDistance=True, returnMetadata=False)
{'ResponseMetadata': {'RequestId': '****', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Thu, 18 Sep 2025 13:39:35 GMT', 'content-type': 'application/json', 'content-length': '513', 'connection': 'keep-alive', 'x-amz-request-id': '718f3ced-ce97-40a1-8575-fc7b6339721c', 'access-control-allow-origin': '*', 'vary': 'origin, access-control-request-method, access-control-request-headers', 'access-control-expose-headers': '*'}, 'RetryAttempts': 0}, 'vectors': [{'key': '429873', 'distance': 1.0056837035081073e+21}, {'key': '681653', 'distance': 1.0056837035081073e+21}, {'key': '223357', 'distance': 1.0056837035081073e+21}, {'key': '455236', 'distance': 1.0056837035081073e+21}, {'key': '213586', 'distance': 1.0056837035081073e+21}, {'key': '597373', 'distance': 1.0056837035081073e+21}, {'key': '134104', 'distance': 1.0056837035081073e+21}, {'key': '921564', 'distance': 1.0056837035081073e+21}, {'key': '755307', 'distance': 1.0056837035081073e+21}, {'key': '325642', 'distance': 1.0056837035081073e+21}]}
>>>
>>>
>>> s3vectors.query_vectors(vectorBucketName="****", indexName="deep-1m", queryVector={"float32": query_deep.tolist()}, topK=10, returnDistance=True, returnMetadata=False)
{'ResponseMetadata': {'RequestId': '****', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Thu, 18 Sep 2025 13:40:25 GMT', 'content-type': 'application/json', 'content-length': '484', 'connection': 'keep-alive', 'x-amz-request-id': 'df3bf080-9d7a-41ce-8b51-0890f3076eed', 'access-control-allow-origin': '*', 'vary': 'origin, access-control-request-method, access-control-request-headers', 'access-control-expose-headers': '*'}, 'RetryAttempts': 0}, 'vectors': [{'key': '358460', 'distance': 0.3194265365600586}, {'key': '743231', 'distance': 0.3194948136806488}, {'key': '145424', 'distance': 0.3463098108768463}, {'key': '896261', 'distance': 0.3465639054775238}, {'key': '518009', 'distance': 0.3480227589607239}, {'key': '688179', 'distance': 0.34803152084350586}, {'key': '653623', 'distance': 0.3497021198272705}, {'key': '378861', 'distance': 0.3508843779563904}, {'key': '630358', 'distance': 0.3524482846260071}, {'key': '214441', 'distance': 0.3544580042362213}]}
>>> s3vectors.query_vectors(vectorBucketName="****", indexName="deep-1m", queryVector={"float32": query_deep.tolist()}, topK=10, returnDistance=True, returnMetadata=False)
{'ResponseMetadata': {'RequestId': '****', 'HostId': '', 'HTTPStatusCode': 200, 'HTTPHeaders': {'date': 'Thu, 18 Sep 2025 13:40:31 GMT', 'content-type': 'application/json', 'content-length': '484', 'connection': 'keep-alive', 'x-amz-request-id': 'ec907813-6f28-4d1f-9577-8a33f873cc11', 'access-control-allow-origin': '*', 'vary': 'origin, access-control-request-method, access-control-request-headers', 'access-control-expose-headers': '*'}, 'RetryAttempts': 0}, 'vectors': [{'key': '358460', 'distance': 0.3194265365600586}, {'key': '743231', 'distance': 0.3194948136806488}, {'key': '145424', 'distance': 0.3463098108768463}, {'key': '896261', 'distance': 0.3465639054775238}, {'key': '518009', 'distance': 0.3480227589607239}, {'key': '688179', 'distance': 0.34803152084350586}, {'key': '653623', 'distance': 0.3497021198272705}, {'key': '378861', 'distance': 0.3508843779563904}, {'key': '630358', 'distance': 0.3524482846260071}, {'key': '214441', 'distance': 0.3544580042362213}]}
To ensure absolute certainty, I retrieved the so-called closest vector using the query_vectors() function, specifying the index name and the key value of the vector. I then manually calculated the Euclidean distance to the query vector using the following code: np.linalg.norm(query_vector - retrieved_vector). The distance obtained (an order of 1e+8) was significantly lower than the one indicated by S3Vectors, re-affirming my suspicions that while data is uploaded correctly, somehow the distance calculation for ANN search is not working as expected.
The one thing I can't understand is the reason while S3Vectors is failing vector search on this dataset specifically. The only differences between the two datasets would be the dimension of their vectors and the value range of the embeddings. While DEEP has 96 values ranging from -1 to 1, GIST is composed of 960 int values casted to float32 (e.g. 172839018.0). I have assumed that neither of these would pose a problem, given that:
- The same exact GIST dataset has been successful when used with other vector databases, both for indexing and querying.
- I created a synthetic 768-dimensional index based on the Deep dataset. This was achieved by tiling the original 96D embeddings eight times, in an attempt to create an alternative high-dimensional dataset. The query was successfully submitted and received consistent responses.
- The distance metric applied in all scenarios is the well-established Euclidean distance, which is particularly effective for non-normalized values.
- I have conducted a series of tests using both Boto3 (as described above) and VectorDBBench (the official Zilliztech benchmark tool for vectordb) with the AWS S3Vectors client. In all cases, the GIST dataset underperforms, with recall rates of 0.0 while other datasets perform well.
I would be very grateful if someone with specific S3Vectors knowledge could help me find the root of this intriguing problem.
