By using AWS re:Post, you agree to the Terms of Use
/Caching/

Questions tagged with Caching

Sort by most recent
  • 1
  • 90 / page

Browse through the questions and answers listed below or filter and sort to narrow down your results.

CodeBuild - Extremely long build times/caching

My project is a large JS/Yarn monorepo (using workspaces) that is contributed to dozens of times a day, by multiple devs. Every time a PR is opened, we build, lint, and test the project, which requires running `yarn install` before anything else. Once a merge occurs to the main branch, any open PRs must merge with the new main branch, and the PR checker needs to be run again before _that_ branch is merged. As you can imagine, there can be a pretty large backlog of open PRs when we're in crunch time. I have successfully used (both s3 & local) caching before in smaller projects, however I can't seem to get **local** caching working with our monorepo (this is a large project, so s3 is much too slow for us, as far as I can tell). When we try using local caching, our build fails with `"EEXIST: file already exists, mkdir '/codebuild/output/src11111111/src/github.com/OUR_ORG/OUR_PROJECT/node_modules/@OUR_PACKAGE/common'".`. This behavior is documented across the web: - https://github.com/aws-samples/aws-codebuild-samples/issues/8 - https://stackoverflow.com/questions/55890275/aws-codebuild-does-not-work-with-yarn-workspaces Some of the remedies include using s3, but as mentioned this is a large project and we update packages at least once every two weeks (some weeks , multiple times), and our initial upload more than doubles our time to 40min. Downloading doesn't save us much time either. - Are there any reasonable steps we can take to cache our node_modules directories so we don't have to pull from yarn every time? - Are there any other solutions to speed up these build times (we're currently ~14min. After optimizing other parts of the build process)? - Do you have any example (large) mono repos you can point to as a template? - Any other tips to speed up our build times?
1
answers
1
votes
9
views
asked 23 days ago

Newbie guidance on DynamoDB design

I'm new to DynamoDB and NoSQL, so I'm trying to get my head wrapped around table design. I have a simple app that will accept status reports about a series of locations. Each status report will have a location ID (number), the status (number), and a timestamp (string). I also need to store a description of each location, which will have a location ID (number) and four other values (all strings). The read pattern is to query the most recent status report for a subset of location IDs (that may change each time). I can use the design pattern where you query with a limit of 1 and set Scan Forward to false for each location ID. The report that will be built needs to include the status, the timestamp, and the description for each of the queried locations. The first part of the design is easy -- the partition key is the location ID, and the sort key is the timestamp. Next, though -- how do I store the description data? (Important: the app is supposed to be run via several Lambda functions, so I'm a bit limited in terms of caching the description info. Eventually, the number of locations will be in the thousands.) Option 1: Store the description info in each status report. I will need to do two transactions each time a status report comes in -- read the description info, then write the new status report item including the four description fields. Reads then become easy, as the info is all there. Option 2: Store the store the descriptions as separate items in the same table. Use a Global Secondary Index to query the description info. When a report is run, do two reads for each location, one to get the status and timestamp, one to get the description. Receiving a status report needs only one transaction, a write. If this is the best option, how should I structure the GSI? Option 3: Store the descriptions in a separate table. Everything else is as per Option 2. Any and all advice greatly appreciated.
1
answers
0
votes
10
views
asked 3 months ago

Amplify build failed because the bucket already exists

Hi, I am trying to deploy an amplify app as per https://webapp.serverlessworkshops.io/staticwebhosting/deploy/ but the build is failing because of an existing bucket I am unable to see the bucket in S3. How can I delete the bucket? or how to resolve this issue? ``` 2022-02-25T03:31:32.457Z [INFO]: An error occurred when creating the CloudFormation stack 2022-02-25T03:31:32.459Z [WARNING]: ✖ Root stack creation failed 2022-02-25T03:31:32.464Z [INFO]: AlreadyExistsException: Stack [amplify-wildrydes-prod-32220] already exists at Request.extractError (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/protocol/query.js:50:29) at Request.callListeners (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/sequential_executor.js:106:20) at Request.emit (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/sequential_executor.js:78:10) at Request.emit (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/request.js:686:14) at Request.transition (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/state_machine.js:14:12) at /root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request.<anonymous> (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/request.js:38:9) at Request.<anonymous> (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/request.js:688:12) at Request.callListeners (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/sequential_executor.js:116:18) at Request.emit (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/sequential_executor.js:78:10) at Request.emit (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/request.js:686:14) at Request.transition (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/state_machine.js:14:12) at /root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request.<anonymous> (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/request.js:38:9) at Request.<anonymous> (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/request.js:688:12) at Request.callListeners (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/sequential_executor.js:116:18) at callNextListener (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/sequential_executor.js:96:12) at IncomingMessage.onEnd (/root/.nvm/versions/node/v14.18.1/lib/node_modules/@aws-amplify/cli/node_modules/aws-sdk/lib/event_listeners.js:335:13) at IncomingMessage.emit (events.js:412:35) at IncomingMessage.emit (domain.js:475:12) at endReadableNT (internal/streams/readable.js:1334:12) at processTicksAndRejections (internal/process/task_queues.js:82:21) { code: 'AlreadyExistsException', time: 2022-02-25T03:31:32.456Z, requestId: '92bb3d71-1610-43c4-a385-54cb25e459cd', statusCode: 400, retryable: false, retryDelay: 81.16703459454344 } 2022-02-25T03:31:32.487Z [ERROR]: !!! Build failed 2022-02-25T03:31:32.487Z [ERROR]: !!! Non-Zero Exit Code detected 2022-02-25T03:31:32.488Z [INFO]: # Starting environment caching... 2022-02-25T03:31:32.488Z [INFO]: # Environment caching completed Terminating logging... ``` Thank you
0
answers
0
votes
6
views
asked 3 months ago

Why does CloudFront still strip brotli from the accept-encoding header?

I want to dynamically send GZIP or Brotli compressed files back to the user's browser based on the "accept-encoding" header. If it contains "br", I want to send Brotli. If it does not, like for IE11, I want to send GZIP. Additionally, I want to cache in CloudFront based on the "Accept-Encoding" header. I set up a POC on my personal account to test this out. I have a Cloudfront distribution pointed to an Elastic Beanstalk Node/Express web application. When I hit the Elastic Beanstalk Node/Express application, it properly receives the accept-encoding header values as "gzip, deflate, br". When I access it through CloudFront, it appears to be stripping the header, so I only get "gzip". As such, my Elastic Beanstalk application has no way of knowing that the request supports brotli. And of course additionally, this means the accept-encoding has no value. Edit- I edited this question as I found the answer. <https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/ServingCompressedFiles.html> Answer is here: "CloudFront can compress some types of files for you (see File Types that CloudFront Compresses), by using gzip. But if you want to compress other file types, or if you want to use a compression algorithm that isn’t gzip, such as brotli, you can compress the files on your own server, and then serve the files by using CloudFront. To use CloudFront to serve a file with a compression algorithm that isn’t gzip, set up CloudFront to cache based on the Accept-Encoding header. When you do this, CloudFront does not make any changes to the Accept-Encoding header and your origin can return an appropriate compressed file for the viewer. When your origin returns a compressed file to CloudFront, include a Content-Encoding header to indicate to CloudFront that the file is already compressed. Then CloudFront simply returns the compressed file to the viewer." Ensure to whitelist the "accept-encoding" header to stop CloudFront from modifying the header. Edited by: JPress on Jun 18, 2019 7:44 PM
1
answers
0
votes
2
views
asked 3 years ago
  • 1
  • 90 / page