Common Crawl中文数据处理速度过慢

0

访问及处理Common Crawl中文数据速度过慢，尤其是2020年及之前的部分，相对2022年及之后的数据，处理速度慢很多倍

Topics

Analytics Machine Learning & AI

Tags

Analytics Machine Learning & AI

Language

中文 (简体)

rePost-User-8248716

asked a year ago197 views

1 Answer

Newest
Most votes
Most comments

Are these answers helpful? Upvote the correct answer to help the community benefit from your knowledge.

0

原因可能狠多。Schema 比2020年复杂了，处理文件数量比2020年多，以及其他原因，都可能造成 Glue Crawler在发现 schema 时，花费更多的时间。

answered 8 months ago

Relevant content

sagemaker美国人口普查数据在中国宁夏区下载提示拒绝访问
LiuChuang
asked a year ago
S3上无法使用Chunked上传文件｜ Unable to use Chunked to upload files on S3
rePost-User-1798329
asked a year ago
Sites on Amplify become very slow recently. Amplify站点最近突然变得很慢。
Zhu
asked 10 months ago
The AWS Lambda program sometimes fails to trigger, while other times it works correctly.
Accepted Answer
HaydenLuo
asked 6 months ago
What do I do if AWS resources are being used to crawl my website?
AWS OFFICIALUpdated 4 years ago
How do I troubleshoot common issues in Amazon CloudWatch Synthetics canary?
AWS OFFICIALUpdated 3 months ago
How do I configure AWS WAF to protect my resources from common attacks?
AWS OFFICIALUpdated 3 years ago
How do I troubleshoot common errors with the import process in Amazon Cognito?
AWS OFFICIALUpdated a year ago
How to use Amazon Polly to resolve common implementation challenges
EXPERT
Abhishek Soni
published 2 years ago
Get the latest from re:Invent 2022
EXPERT
Kamran Khan
published a year ago