- Newest
- Most votes
- Most comments
Hello, there are different ways to retrieve your data from S3.
- Direct access through the S3 console or API/SDK calls over the internet.
- As you said, data can be copied to DynamoDB for lower latency applications, enabling access from there.
- Moreover, AWS Athena facilitates running SQL queries directly on S3-stored data.
- Temporary credentials via access grants allow access to S3 objects.
- S3 File Gateway enables direct access to objects in AWS
- Within a VPC, private access is established using interface VPC endpoints or NAT/internet gateways, controlled by bucket and access point policies.
- Other services like Lambda, EMR, and Glue are capable of accessing S3 data for various data processing tasks.
I know this can gather confusion, but on the long run, disserting what your use case is, you may find the perfect way to access your data in the most efficient why.
Thanks!
The answer from Giovanni Lauria above is good but to really answer the question we need to ask you: What are you trying to do?
There are going to be a large number of ways that you can abstract access to S3. But without knowing what you're trying to achieve it would be easy to get tied up in details which aren't helpful.
Can you suggest which of these ways are favorable, if my data is less than 50 MB and I want to access data in real time with a low latency of about 1 or 2 seconds ?
Overall, considering your specific needs of real-time access with low latency for data less than 50 MB, direct access over the internet via S3 console or API/SDK calls is likely the most favorable option. All the rest is more costly than beneficial, on a multitude of factors.
Relevant content
- asked 2 months ago
- asked 2 years ago
AWS OFFICIALUpdated 6 months ago- AWS OFFICIALUpdated 9 months ago
- AWS OFFICIALUpdated 3 years ago
- AWS OFFICIALUpdated 2 years ago
my data is less than 50 MB in the form of parquet files which is stored in s3 bucket and I want to access data in real time with a low latency of about 1 or 2 seconds through my web application?
That's a good start, but now it becomes about what "access data" means. If you intend to query the data using some sort of SQL query then Glue is definitely the way to go. If you're just going to read it into your application and process it there then a direct S3 connection is probably good enough. You haven't mentioned where the application is hosted nor whether it is the client or server side which will access the data.
I have two use cases of data access : 1 - Simply read the data into my application 2 - Run a sql query to derive an attribute and then accessing it into my application This data will be accessed at client side .
If you have many clients then it makes sense to optimise by only reading the data at the server-side (which is closer to S3 in terms of latency) and doing the query there; then returning the result to the client. This allows to (perhaps) to cache the data (which makes queries faster and perhaps less expensive); and maybe to cache the query responses (again, faster). You could use Glue to do this; but for the size of data that you're talking about it might be simpler to hold it in memory on the server side.