By using AWS re:Post, you agree to the Terms of Use
/Amazon DynamoDB/

Questions tagged with Amazon DynamoDB

Sort by most recent
  • 1
  • 90 / page

Browse through the questions and answers listed below or filter and sort to narrow down your results.

Possible CLI Bug: Dynamo DB endpoint URL does not work locally with active and correct credentials set

**Summary**: Dynamo DB commands from the CLI do not work when real credentials are set up. The 'endpoint-url' flag should work around this and recognize that localhost endpoints can be hit with no credentials given the default setup of the AWS Dynamo Docker image. Output of command after setting credentials: `An error occurred (ResourceNotFoundException) when calling the DescribeTable operation: Cannot do operations on a non-existent table` Is there a fix or workaround for this? **System**: MacOS Monterey version 12.0.1, Macbook Pro - M1 - 2020 ``` 'aws --version' -> aws-cli/2.4.11 Python/3.9.10 Darwin/21.1.0 source/arm64 prompt/off ``` **To reproduce**: -- Start from a terminal that does NOT have AWS Credentials set up via environment variables or anything else -- Start up a local Dynamo DB Instance on Docker: ``` docker pull amazon/dynamodb-local docker run -p 8000:8000 --name=ddblocal -d amazon/dynamodb-local ``` -- Create a table: ``` aws dynamodb create-table --attribute-definitions "[{ \"AttributeName\": \"key\", \"AttributeType\": \"S\"}, { \"AttributeName\": \"valueA\", \"AttributeType\": \"S\"}]" --table-name test_table --key-schema "[{\"AttributeName\": \"key\", \"KeyType\": \"HASH\"}, {\"AttributeName\": \"valueA\", \"KeyType\": \"RANGE\"}]" --endpoint-url "http://localhost:8000" --provisioned-throughput "{\"ReadCapacityUnits\": 100, \"WriteCapacityUnits\": 100}" --region local ``` -- Query the table (to prove it works): ``` aws dynamodb describe-table --table-name test_table --region local --endpoint-url "http://localhost:8000" ``` -- Set your real AWS Credentials: ``` export AWS_ACCESS_KEY_ID="<REAL KEY ID HERE>" export AWS_SECRET_ACCESS_KEY="<REAL SECRET KEY HERE>" export AWS_SESSION_TOKEN="REAL TOKEN HERE>" ``` -- Query the table again (This one fails for me - see output above) ``` aws dynamodb describe-table --table-name test_table --region local --endpoint-url "http://localhost:8000" ```
0
answers
0
votes
6
views
Jaera
asked 4 days ago

The method toJSONPretty() is undefined for the type Map<String,AttributeValue>

Hello I've a jsp with code (from https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/ScanJavaDocumentAPI.html) : ... ``` <%@ page import="com.amazonaws.services.dynamodbv2.AmazonDynamoDB" %> <%@ page import="com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder" %> <%@ page import="com.amazonaws.services.dynamodbv2.document.DynamoDB" %> <%@ page import="com.amazonaws.services.dynamodbv2.document.Item" %> <%@ page import="com.amazonaws.services.dynamodbv2.document.ItemCollection" %> <%@ page import="com.amazonaws.services.dynamodbv2.document.ScanOutcome" %> <%@ page import="com.amazonaws.services.dynamodbv2.document.Table" %> <%@ page import="com.amazonaws.services.dynamodbv2.model.AttributeValue" %> <%@ page import="com.amazonaws.services.dynamodbv2.model.ScanRequest" %> <%@ page import="com.amazonaws.services.dynamodbv2.model.ScanResult" %> ... <% AmazonDynamoDB client = AmazonDynamoDBClientBuilder.standard().build(); DynamoDB dynamoDB = new DynamoDB(client); String tableName = "TicketsWalletUsersDev"; // Table table = dynamoDB.getTable(tableName); Map<String, AttributeValue> lastKeyEvaluated = null; do { ScanRequest scanRequest = new ScanRequest() .withTableName(tableName) .withLimit(10) .withExclusiveStartKey(lastKeyEvaluated); ScanResult result = client.scan(scanRequest); for (Map<String, AttributeValue> item : result.getItems()){ System.out.println(item.toJSONPretty()); } lastKeyEvaluated = result.getLastEvaluatedKey(); } while (lastKeyEvaluated != null); %> ``` It gives the error: The method toJSONPretty() is undefined for the type Map<String,AttributeValue> I want the string containing the true json and not the string returned from item.toString() that is : {addresses={M: {mobile+75@nearit.com={M: {requestId={S: 909b1963-0ed4-4ac1-9b71-846239ed5bf6_1583142338893,}, verified={BOOL: true}},}, federico+11@nearit.com={M: {requestId={S: 909b1963-0ed4-4ac1-9b71-846239ed5bf6_1581332986414,}, verified={BOOL: true}},}, federico+signup3@nearit.com={M: {requestId={S: 909b1963-0ed4-4ac1-9b71-846239ed5bf6_1581330211793,}, verified={BOOL: true}},}, federico+33@nearit.com={M: {requestId={S: 909b1963-0ed4-4ac1-9b71-846239ed5bf6_1581330013122,}, verified={BOOL: true}},}, mobile@nearit.com={M: {requestId={S: 909b1963-0ed4-4ac1-9b71-846239ed5bf6_1583162677615,}, verified={BOOL: true}},}, federico+12@nearit.com={M: {requestId={S: 909b1963-0ed4-4ac1-9b71-846239ed5bf6_1581333042026,}, verified={BOOL: true}},}, federico+2@nearit.com={M: {requestId={S: 909b1963-0ed4-4ac1-9b71-846239ed5bf6_1581332941950,}, verified={BOOL: true}},}},}, created={S: 2020-02-05T10:40:17.887Z,}, userName={S: 909b1963-0ed4-4ac1-9b71-846239ed5bf6,}} Thanks!
0
answers
0
votes
3
views
AWS-User-0689311
asked 4 days ago

Sending Reublished MQTT data to DynamoDB table

I have successfully republished data from my "lorawan/+/uplink" topic to my new topic "datashow". I am now trying send the data I extracted from the original topic to a DynamoDB. I created a rule named "SensorDB_rule" to select the data from the "datashow" topic and send it to a DynamoDB. The data I receive is listed below. ``` { "output": { "DeviceID": "eui-70b3d57ed004c725", "PayloadData": { "Battery": 100, "Sensor": -27.730278573288206, "alt": 63.10, "lat": 38.789196, "lon": -77.073914, "time": "18:52:43" } } } ``` The rule SQL Statement is listed below ``` SELECT output.DeviceID as Device_ID, output.PayloadData.Battery as Battery, output.PayloadData.Sensor as Sensor, output.PayloadData.alt as Altitude, output.PayloadData.lat as Latitude, output.PayloadData.lon as Longitude, output.PayloadData.time as Time FROM 'datashow' ``` I created a DynamoDB table named "SensorDB" and set "DeviceID" as my Partition Key and "time" as my Sort key. With both set to String ``` DeviceID time ``` I'm using the Default settings for my DB table. and created the table Back in the "SensorDB_rule" I selected my table I just created ("SensorDB") and entered the following ``` Partition key- DeviceID Partition key type- STRING Partition key value- $DeviceID Sort Key- time Sort key type- STRING Sort key value- $time ``` and wrote the message to the column "MyMessageData" finally I created a new IAM role names "SensorDB_role" and attached the following permissions AWSIoTLogging AWSIoTRuleActions AWSIoTEventsFullAccess AWSIoTFullAccess AmazonDynamoDBFullAccess I have double checked all my information but I am still not getting anything inside of my table. Any suggestions on how to fix this? (adding policies, correcting my SQL statement, changing my table settings)
2
answers
0
votes
2
views
AWS-User-6259931
asked 8 days ago

Sync DynamoDB to S3

What is the best way to sync my DynamoDB tables to S3, so that I can perform serverless 'big data' queries using Athena? The data must be kept in sync without any intervention. The frequency of sync would depend on the cost, ideally daily but perhaps weekly. I have had this question a long time. I will cover what I have considered, and why I don't like the options. 1) AWS Glue Elastic Views. Sounds like this will do the job with no code, but it was announced 18 months ago and there have been no updates since. Its not generally available, and there is not information on when it might be. 2) Use dynamodb native backup following this blog https://aws.amazon.com/blogs/aws/new-export-amazon-dynamodb-table-data-to-data-lake-amazon-s3/. I actually already use this method for 'one-off' data transfers that I kick-off manually and then configure in Athena. I have two issues with this option. The first is that, to my knowledge, the export cannot be scheduled natively. The blog suggests using the CLI to kick off exports, and I assume the writer intends that the CLI would need scheduling on a cron job somewhere. I don't run any servers for this. I imagine I could do it via a scheduled Lambda with an SDK. The second issue is that the export path in S3 always includes a unique export ID. This means I can't configure the Athena table to point to a static location for the data and just switch over the new data after a scheduled export. Perhaps I could write another lambda to move the data around to a static location after the export has finished, but it seems a shame to have to do so much work and I've not seen that covered anywhere before. 3) I can use data pipeline as described in https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/DynamoDBPipeline.html. This post is more about backing data up than making it accessible to Athena. I feel like this use case must be so common, and yet none of the ideas I've seen online are really complete. I was wondering if anyone had any ideas or experiences that would be useful here?
2
answers
0
votes
8
views
c3R1YXJ0
asked 11 days ago

AWS Backup DynamoDB billing

Hi everyone! I'd like to understand better the billing composition regardless of AWS Backup on DynamoDB resources since I got an unexpected increase in my billing. I'm aware of AWS Backup billing itself thanks to the [documentation](https://aws.amazon.com/backup/pricing/), anyway, when I access the Billing service I can notice an exponential billing pricing in DynamoDB service, on the section `Amazon DynamoDB USE1-TimedBackupStorage-ByteHrs` the description allows me to see that I'll be paying $0.10 per GB-month of storage used for on-demand backup, showing me that I've used 14,247.295 GB-Month (This makes sense with the billing I got) but where my doubt comes from is, **where does all those GB come from?** The last snapshot-size just shows 175.5 GB I've configured my backup plan with the following parameters: ``` { "ruleName": "hourly-basis", "scheduleExpression": "cron(0 * ? * * *)", "startWindowMinutes": 60, "completionWindowMinutes": 180, "lifecycle": { "toDeletedAfterDays": 30 } } ``` I'm also copying snapshots into a second region on `us-west-2` As you can see, I'm handling a schedule expression on an hourly basis backup because of compliance requirements. *Is this enough justification for the high billing?* I'm aware that backups with low RPO are commonly expensive but I just want to be sure that this billing is not higher than it should be because of any wrong Backup configuration. Thanks in advance!
0
answers
0
votes
1
views
AWS-User-2991569
asked 14 days ago

Storing/representing a hierarchical tree used for navigation within an Amplify web app and AppSync GraphQL API layer.

Hi, **TL;DR: Can anyone give a recommend approach to storing customisable n-level hierarchy trees for grouping and navigating results via a frontend Amplify-powered web app (using DynamoDB or any other database solution that can be mapped to AppSync ideally)?** **Some background** I'm building an multi-tenant IoT analytics solution that takes data from some sensors out in the field, uploads to AWS, processes this data and stores in a DynamoDB table (i.e. a very "standard" setup). I'm planning on adding a web frontend (built using Amplify and an AppSync GraphQL layer) that will allow users to navigate a **customisable, n-level** hierarchy tree of assets, in order to view the sensor data we've collected. Examples of valid hierarchies include: Country -> Site -> Building -> Floor -> Room -> Sensor (6-level) or Site -> Building -> Room -> Sensor (4-level) etc. The important thing here, is that this hierarchy tree can differ per customer, and needs to be customisable on a tenant-by-tenant basis, but we don't need to do any complex analysis or navigation of relationships between hierarchy levels (so, to me, something like Amazon Neptune or another graph database feels a bit overkill, but perhaps I’m wrong). My first thought was to try and build a hierarchical relationship inside of a DynamoDB table, possibly making use of a GSI to provide this, but in all of the examples I’ve seen online, the focus is very much on quick retrieval, but not so quick updating of hierarchy trees – now, whilst it’s unlikely that these tree structures would be updated on a regular basis, it is something we need to be able to support, so the idea of possibly updating ‘000s of rows in DynamoDB every time we want to make a change to the hierarchy tree for a given control area doesn’t seem quite right to me. Hence, my question above. I'm ideally looking for guidance on how to structure a DDB table to best support BOTH optimal retrieval of, and updates to, hierarchy trees in our application, but if DDB isn't the right answer here, then suggestions of alternatives would also be greatly appreciated. Many thanks in advance.
1
answers
0
votes
3
views
cgddrd
asked 14 days ago

Which Database service would be best for this use case?

I'm working on an online turn-based strategy game called CivPlanet. There will be dozens of CivPlanet games running at once, each with its own unique gameID. The gamestate consists of several uniquely named JSON objects, henceforth called CivPlanet objects. Players interact with the server via REST requests, which are sent somewhat infrequently. When the player plays the game, they're probably sending around 1 request per minute. In short, I need a database service to store all CivPlanet objects. Each CivPlanet object has a timestamp, gameID, and a name to distinguish it from other objects in that game. CivPlanet objects are never created or deleted once the game is added to the database. However, some can be modified. Also, not all games will share the same set of objects. I need to be able to: * Retrieve a list of all CivPlanet games, along with some metadata about each game such as whether it is accepting new players. * Conditionally retrieve all CivPlanet objects associated with a given gameID that were updated after a given timestamp. * Lock all CivPlanet objects under a given gameID as I prepare to update the gamestate. * Atomically overwrite certain CivPlanet objects that share a gameID. * Release the lock. The server receives a mixture of queries and events from the player. When responding to a query, the server needs to check for any updated data on the database, then process the query and return the result. When responding to the event, it needs to obtain a lock, check for updated data, and process the event. If the event fails, it releases the lock and notifies the player. If it succeeds, it publishes the new data to the database and releases the lock, then notifies the player. My question is, what database service is best suited for these requirements, and what structure should I use within that service? I was looking at DynamoDB. I thought gameID could be the partition key. I'm not sure if I need a sort key, or what that sort key would be. I probably need two databases, one that maps from gameID to metadata, and another that maps from gameID+objectName to JSON. Any thoughts?
0
answers
0
votes
7
views
DrCorchit
asked 16 days ago

How do I set up an AWS Amplify project to query an existing AWS AppSync API?

Hi, I am new to AWS Amplify and would like guidance on how to send a query to an ***existing*** GraphQL API on AWS AppSync. I am unsure how to start as a lot of Amplify coverage creates a *new* AppSync API using the Amplify CLI. ## Objectives * Set up a Node.js project to work with an existing AWS AppSync API, using AWS Amplify as the GraphQL client. * Send a single query to an existing AWS AppSync API. The query lists game results from a DynamoDB table and is called `listGames` in my GraphQL schema. * I need to repeat the query in order to fetch all available database records that satisfy the query. This would mean adding results to an array/object until the `nextToken` is `null` (i.e. no more records can be found for the query). ## Context * This application is deployed in an Amazon ECS container using AWS Fargate. * The ECS service is fronted by an Application Load Balancer (ALB). * A leader board web page fetches game results through a `POST` request to the ALB's DNS name / URL and adds them to a HTML table. ## Notes * For now, API key is my authentication method. I would soon like to switch to a task IAM role in ECS. * The ECS deployment described in 'Context' is working but it sends `POST` requests without AWS libraries. It is my understanding that I would need to use an AWS library in order to use an IAM role for AppSync authentication (used as a [task IAM role in ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html)). Please correct me if I am mistaken. I would greatly appreciate any help you can give me. Thank you for your time!
1
answers
1
votes
4
views
Toby
asked a month ago

Load testing serverless stack using Gatling

Hi, I'm doing some load testing on my serverless app and I see that it is unable to handle some higher loads. I'm using API Gateway. Lambda(Java 8) and DynamoDB. The code that I'm using is the same as this from this [link]([https://github.com/Aleksandr-Filichkin/aws-lambda-runtimes-performance/tree/main/java-graalvm-lambda/src/lambda-java). In my load testing, I'm using Gatling. The load that I configured is that I'm doing a request with 120 users, then in one minute I ramp users from 120 to 400, and then for 2 minutes I'm making requests with 400 constant users per second. The problem is that my stack is unable to handle 400 users per second. Is it normal? I thought that serverless will scale nicely and will work like a charm. Here is my Gatling simulation code: ```java public class OneEndpointSimulation extends Simulation { HttpProtocolBuilder httpProtocol = http .baseUrl("url") // Here is the root for all relative URLs .acceptHeader("text/html,application/xhtml+xml,application/json,application/xml;q=0.9,*/*;q=0.8") // Here are the common headers .acceptEncodingHeader("gzip, deflate") .acceptLanguageHeader("en-US,en;q=0.5") .userAgentHeader("Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:16.0) Gecko/20100101 Firefox/16.0"); ScenarioBuilder scn = scenario("Scenario 1 Workload 2") .exec(http("Get all activities") .get("/activitiesv2")).pause(1); { setUp(scn.injectOpen(atOnceUsers(120), rampUsersPerSec(120).to(400).during(60), constantUsersPerSec(400).during(Duration.ofMinutes(2)) ).protocols(httpProtocol) ); } } ``` Here are the Gatling report results: [Image link](https://ibb.co/68SYDsb) I'm also receiving an error: **i.n.h.s.SslHandshakeTimeoutException: handshake timed out after 10000ms ** -> This is usually approx 50 requests. It is happening when Gatling is starting to inject 400 constant users per second. I'm wondering what could be wrong. It is too much for API Gateway, Lambda and DynamoDB?
2
answers
0
votes
3
views
Maryn
asked a month ago

DynamoDB: Duplicate a lot of data to save a second lookup?

There are two typical uses for my new database. 1. A user will access their record/item via their userID, and they will manage their info and a number of devices, each with a unique devID. 2. A device connects and using its devID, will find the owning userID, then takes action based on attributes in the user item. Two options I could use, each with a single DynamoDB table. A. The table has items that are users or devices, with a partition key of ID and sort key of itemType. User type items have associated attributes like addresses, account and profile info, etc, and a set of deviceIDs. Device type items have associated attributes like their preferences, their type, their capabilities, and an attribute for the userID that owns it. You can access both users and devices really quickly. If you are doing (1) you lookup and find a user, then you will have to use a set attribute that lists the one or more deviceIDs it owns, and then make individual lookups for each device. That's 2 lookups for a user that owns one device and more for multiple devices. Or if you are doing (2), search and find a device, you grab its userID attribute and then lookup the userID item. That's 2 lookups. B. I could reduce the multiple lookups this way: Still one table, but all entries in the table are more homogenous: Every item includes all user related attributes, and includes one device's attributes. The provisioning key is the userID and sort key is deviceID, another indexed attribute is just the deviceID. If you are doing (1) then you lookup the userID and you get one or more records depending on whether they own one device or more. If you are doing (2), then we quickly find the device, and that same item includes all the user info we need and we don't need to do another lookup. The problem with B is that I am duplicating a lot of data about the user in each of the items. Keeping them all synced is going to be problematic too, but that's a lot rarer. So, am I overthinking the lookup costs, and should just go with the multiple lookup as in A, or is the multiple lookups going to be expensive enough that I need to have a better data design?
1
answers
0
votes
4
views
myAWSAdventure
asked 2 months ago

Advice for best database/datastorage for historical data

Hi, I´m doing some reasearch to find the best place to centralize lots of data logs generated by my application considering pricing ,performance and scalabilty. Today all my application data including logs are stored on an Oracle database, but I´m thinking to move all the LOG related data outside it to reduce it´s size and not to worry about storage performance etc... Just put everything on a "infinite" storage apart from my actual database using CDC or a regular batch process **Below are some needs:** - Only inserts are necessary (no updates or deletes) - Customers will need access to this historic data - Well defined pattern of access (one or two indexes at maximum) - Latencies of few seconds is ok - Avoid infrastrucure, DBA, perfomance bottleneck log term... - Infinite Retentiton period (means I don´t want to worry about performance issues, storage size in long term. But something that can handle a few terabytes of data ) **Use case example: ** Historical Sales order by items ( id_item | id_customer | qty_sold | date_inserted ... ), aprox 50 millions records per day Where I would need to see the historical data by item, and by customer for example (two dimensions) I´ve done some research with the options below **S3 + Athena **-> Put everthing on s3, no worries about infrastructure perfomance issues, however as I need query by item and customer, probably it´would be necessary to break files by item or customer , generate millions of partitions to avoid high costs searching on every file etc.. **Postgre** -> Not sure if could be performance bottleneck once tables gets too big even with partition strategies **DynamoDB **-> Not sure if it´s a good alternative to historical data regarding pricing once seconds latency is ok **MongoDB/ DocumentDB **-> Not very familiar with it (I´d prefer SQL language type) but I know it´s has a good scalability **Cassandra**-> dont´know very much **Timeseries db as influxDB, timestream etc..**-> dont´know very much, but it seems appropriate for timeseries What option would you choose ? Sorry in advance if I saying something wrong or impossible :) Thank you!
1
answers
0
votes
5
views
AWS-User-7297854
asked 2 months ago

Filter Expression can only contain non-primary key attributes ?

DynamoDB currently does not allow FilterExpression on your GSI hash or sort keys. Hash or sort key should be specified using KeyCondition only. I understand this is made to protect us from making mistakes, but it also removes the ability of doing some pretty good things. As an example, I am working on a table that has relationships from users to resources. In a simplified example, imagine: ``` PK|user#SoMeId SK|resourceA#rEsOuRcEiD NAME|-nice-slug-name- ``` I want to be able to search in "NAME" using "contains", and I would also like to have my list sorted by it, as it is more natural for users than random ids. If I could use a sort key in both KeyExpression and FilterExpression, I could create a GSI using PL/NAME, append prefix to name as well, and have a nice search index. ``` PK|user#SoMeId SK|resourceA#rEsOuRcEiD NAME|resourceA#-nice-slug-name- query (PK=user#SoMeId, SK=begins_with(resourceA), INDEX=pk-name-index, FILTER={ name: contains(slug) }) ``` That would be very nice. Judging by I understand DynamoDB works, it seems like this should be completely possible from the technical perspective. Why don't you guys force an extra argument to enable functionality? Instead of outright banning it, just offer the ability of setting something like `AllowKeyFilter=true` on the API calls, just to make it an obvious opt-in feature, so no one can complain if they pay more than expected in case they made something wrong? When I asked this question on the past, I have been told to duplicate fields, but that's just cumbersome. Why not allow it to be used by those that opt-in? (Continued from https://forums.aws.amazon.com/thread.jspa?threadID=172530)
0
answers
0
votes
4
views
Ricardo Nolde
asked 2 months ago

ProjectionExpression causes 'Unsupported type passed' error when querying

Hello, I seem to be having a strange issue when querying on my GSI that as soon as I specify the fields I wish to return in the query, I get the "Unsupported type passed" error. Here's the query: ``` { TableName: 'RatingsTable', IndexName: 'username', KeyConditionExpression: '#username = :username', ExpressionAttributeNames: { '#username': 'username' }, ExpressionAttributeValues: { ':username': { S: 'jonathanmsifleet' } }, ProjectionExpression: 'username, imdb_title_id' } ``` The table: ``` RatingsTable: Type: AWS::DynamoDB::Table Properties: TableName: RatingsTable BillingMode: PAY_PER_REQUEST AttributeDefinitions: - AttributeName: imdb_title_id AttributeType: N - AttributeName: username AttributeType: S KeySchema: - AttributeName: imdb_title_id KeyType: HASH - AttributeName: username KeyType: RANGE GlobalSecondaryIndexes: - IndexName: username KeySchema: - AttributeName: username KeyType: HASH Projection: ProjectionType: ALL ``` The strange thing is that if I remove `ProjectionExpression` from the query it executes fine, yet as soon as I specify fields then it throws the error. The error in its entirety in Cloudwatch is `Unsupported type passed: username`, if I remove username then it moves to the next field, `Unsupported type passed: imdb_title_id`. Here's the code I am using to run the query: ``` query = { TableName: 'RatingsTable', IndexName: 'username', KeyConditionExpression: '#username = :username', ExpressionAttributeNames: { '#username': 'username' }, ExpressionAttributeValues: { ':username': { S: username } }, ProjectionExpression: 'username, imdb_title_id' }; console.log('query', query); const result = await dbClient.send(new QueryCommand(query)); ``` Any ideas what I am doing wrong?
1
answers
0
votes
3
views
JonathanSifleet
asked 2 months ago

DynamoDB Hierarchical, Sorted Queries

I would like to be able to query data hierarchically and return the results ordered by another attribute. What is the most efficient way to store and query sorted hierarchical data? For example, if I have a table with four attributes: `customer_id`, `country`, `location`, and `last_updated_date`, where `location` contains hierarchical information such as `state:county:city`, so a few records may look like: ``` ------------|--------|-------------------|-------------| customer_id |country |location |last_updated | ------------|--------|-------------------|-------------| 123456 |USA |WA:King:Seattle |2022-03-18 | 789012 |USA |WA:King:Kent |2022-03-15 | 098765 |USA |NY:Bronx:NYC |2022-02-28 | 432109 |USA |WA:Spokane:Spokane |2022-03-20 | ``` The `PK` of the table is the `customer_id` because most queries will pull information by `customer_id`, but there are other use cases that will want to (a) find all customers within a given location (e.g. `state` or `county`), and (b) return the results sorted (descending) by `last_updated`. To accomplish (a), I have a `GSI`, with `country` as the `PK` and `location` as the `SK`, `query`ing the `GSI` using `location.begins_with`. But I can't figure out how to accomplish (b). My understanding is that ordering operations are usually performed with `scanIndexForward`, but I'm already using the `GSI` for the hierarchical query. Is there a way to do both (a) and (b)? Thanks!
1
answers
0
votes
4
views
ipsherman
asked 2 months ago

Getting an AccessDeniedException when trying to access (read) a DynamoDB table from a completely different AWS account

Hello, I have an application deployed in an EKS cluster in `Account A` that is trying to read an item from a DynamoDB table in `Account B`. I have done the following : * I have created a role in `Account B` called `DynDBReadAccess` with a policy that allows someone to perform the `dynamodb:GetItem` action on the table `arn:aws:dynamodb:us-east-1:<Account B>:table/myTable`. * I then created the role `CrossAccountDynDBAccess` in `Account A` with permissions to perform the `sts:AssumeRole` action and assume the role `arn:aws:iam::<Account B>:role/DynDBReadAccess`. * I updated the trust policy in the `DynDBReadAccess` role to trust the Principal `arn:aws:iam::<Account A>:role/CrossAccountDynDBAccess`. I have a pod with aws cli deployed in the cluster for debugging purposes. Now, I do the following. * I exec into the pod and run `aws sts get-caller-identity`, I see the correct assumed role `CrossAccountDynDBAccess`. * Then I run `aws sts assume-role --role-arn "arn:aws:iam::<Account B>:role/DynDBReadAccess" --role-session-name crossAccountSession` and I get the temporary credentials. * I set the environment variables `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_SESSION_TOKEN` with the temporary credentials I just received. * I run the following command `aws dynamodb get-item --table-name myTable --key 'somekey'` and I get the following error `An error occurred (AccessDeniedException) when calling the GetItem operation: User: arn:aws:sts::<Account B>:assumed-role/DynDBReadAccess/crossAccountSession is not authorized to perform: dynamodb:GetItem on resource: arn:aws:dynamodb:ap-southeast-1:<Account B>:table/myTable` I thought that once the roles, permissions and trust policies were set, cross account access should be possible. **Can someone tell me what is missing?** Some other points to note * The OIDC endpoint and IRSA role has been enabled in the EKS cluster and the service account for the cluster has been created. * The aws cli pod that I deployed has been deployed with the service account mapped to the IRSA role. * I tried doing the same thing with a lambda function where I created a lambda function in `Account A` that will read an item from the same DynamoDB table. The execution role of the lambda function assumes the `arn:aws:iam::<Account B>:role/DynDBReadAccess` and reads an item from the same DynamoDB table. **This works**.
1
answers
0
votes
7
views
AWS-User-3392050
asked 2 months ago
1
answers
0
votes
4
views
Phil Endecott
asked 2 months ago

DynamoDB.DocumentClient Issue - Error: Number 820982911946154500 is greater than Number.MAX_SAFE_INTEGER. Use BigInt.

I am using v3 of the aws-sdk. When I try to use the DynamoDB document client to put items into my DynamoDB table, I receive the error `Number 1213213131312312312314342 is greater than Number.MAX_SAFE_INTEGER. Use BigInt. ` (full stack trace below). My code is as follows: ``` const dynamodb = new DynamoDBClient({ region: 'us-east-1' }); const client = DynamoDBDocumentClient.from(dynamodb, translateConfig); const params = { TableName: 'Orders', Item: { PK: `${PK}`, SK: 'SK', id: id } const putOperation = new PutCommand(params); await client.send(putOperation) ``` Id is a BigInt. I was under the impression that the DocumentDB client would marshall the number to a 'BigInt', however it looks like it is only being marshalled to a 'Number' and since it is too large to be held by a Number it is failing the validation and throwing an exception. I also tried playing around with marshall/unmarshall configurations but saw the same error. Please let me know if you know how I can fix this issue/if it is a bug in the library. Additional information that might be helpful: Lambda Node runtime version specified in CDK ``` runtime: Runtime.NODEJS_14_X, ``` Library versions (all sdk v3) ``` "@aws-sdk/client-dynamodb": "^3.53.0", "@aws-sdk/lib-dynamodb": "^3.53.0", "@aws-sdk/util-dynamodb": "^3.53.0", ``` Full stacktrace. ``` 2022-03-09T20:11:12.601Z 9e321b10-3eb2-4042-b429-a8205e12f5b5 INFO Error: Number 820982911946154500 is greater than Number.MAX_SAFE_INTEGER. Use BigInt. at validateBigIntAndThrow (/var/task/node_modules/@aws-sdk/util-dynamodb/dist-cjs/convertToAttr.js:129:11) at convertToNumberAttr (/var/task/node_modules/@aws-sdk/util-dynamodb/dist-cjs/convertToAttr.js:138:9) at Object.convertToAttr (/var/task/node_modules/@aws-sdk/util-dynamodb/dist-cjs/convertToAttr.js:35:16) at /var/task/node_modules/@aws-sdk/util-dynamodb/dist-cjs/convertToAttr.js:118:36 at convertToMapAttrFromEnumerableProps (/var/task/node_modules/@aws-sdk/util-dynamodb/dist-cjs/convertToAttr.js:122:7) at Object.convertToAttr (/var/task/node_modules/@aws-sdk/util-dynamodb/dist-cjs/convertToAttr.js:23:16) at /var/task/node_modules/@aws-sdk/util-dynamodb/dist-cjs/convertToAttr.js:118:36 at convertToMapAttrFromEnumerableProps (/var/task/node_modules/@aws-sdk/util-dynamodb/dist-cjs/convertToAttr.js:122:7) at Object.convertToAttr (/var/task/node_modules/@aws-sdk/util-dynamodb/dist-cjs/convertToAttr.js:23:16) at Object.marshall (/var/task/node_modules/@aws-sdk/util-dynamodb/dist-cjs/marshall.js:5:53) ```
0
answers
0
votes
3
views
Yash Dalal
asked 2 months ago

Newbie guidance on DynamoDB design

I'm new to DynamoDB and NoSQL, so I'm trying to get my head wrapped around table design. I have a simple app that will accept status reports about a series of locations. Each status report will have a location ID (number), the status (number), and a timestamp (string). I also need to store a description of each location, which will have a location ID (number) and four other values (all strings). The read pattern is to query the most recent status report for a subset of location IDs (that may change each time). I can use the design pattern where you query with a limit of 1 and set Scan Forward to false for each location ID. The report that will be built needs to include the status, the timestamp, and the description for each of the queried locations. The first part of the design is easy -- the partition key is the location ID, and the sort key is the timestamp. Next, though -- how do I store the description data? (Important: the app is supposed to be run via several Lambda functions, so I'm a bit limited in terms of caching the description info. Eventually, the number of locations will be in the thousands.) Option 1: Store the description info in each status report. I will need to do two transactions each time a status report comes in -- read the description info, then write the new status report item including the four description fields. Reads then become easy, as the info is all there. Option 2: Store the store the descriptions as separate items in the same table. Use a Global Secondary Index to query the description info. When a report is run, do two reads for each location, one to get the status and timestamp, one to get the description. Receiving a status report needs only one transaction, a write. If this is the best option, how should I structure the GSI? Option 3: Store the descriptions in a separate table. Everything else is as per Option 2. Any and all advice greatly appreciated.
1
answers
0
votes
9
views
plsuh
asked 2 months ago

Are there any cases lead to "Query condition missed key schema element : sort key" ?

I met a really weird question when I was using DynamoDB in the project (with JAVA SDK). We created a DynamoDB table with a Primary Partition Key 'id', and a Sort Key 'create_time'. And I used `query` operation in my source code with both Partition Key and Sort Key, but I get a exception while querying ``` com.amazonaws.services.dynamodbv2.model.AmazonDynamoDBException: Query condition missed key schema element: create_time (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: K9B8K5DU0CQ1BQA9VJIPTI6EIJVV4KQNSO5AEMVJF66Q9ASUAAJG) ``` I have researched for a day, and almost all the results for searching "Query condition missed key schema element" from google told me that I'm not using the Partition Key or GSI in `KeyConditionExpression` , but I'm sure that we had set up correct Partition Key and Sort key with correct names and used correct in source code, by the way we didn't create any GSI in out DynamoDB. So I just want to know that are there any situations which might cause this exception - Query condition missed key schema element with a **Sort Key** rather than a **Partition Key**. **BACKGROUND STATEMENT ** 1. This `query` operation worked in my develop environment well, but not worked in the production environment. The source code is exactly the same. 2. This table is new created in production environment. (I ever suspected that there must be something wrong while creating the table in the production environment. But I have confirmed for several times, the Partition Key and Sort Key are all correctly named and we do not create any GSI.) 3. The differences between develop and production might be the DynamoDB regions. Develop is created in ap-southeast-1 and the production is in us-west-1.
1
answers
0
votes
4
views
zeko zhang
asked 2 months ago
  • 1
  • 90 / page