By using AWS re:Post, you agree to the Terms of Use
/Windows 10 PC freezes when uploading large file over long period of time/

Windows 10 PC freezes when uploading large file over long period of time

0

I've been working on a python script (python version 3.9.6) using boto3 to upload files to my s3 bucket. I want to back up files to AWS as my offsite backup, plus I want to learn more about AWS since I'm a programmer. The problem is my PC freezes when I'm in the middle of uploading it. I have been trying to debug whether or not it's a Windows issue, but I recently changed my configuration to upload chunks in 1000mb sizes and it froze after 15 minutes. Previously, I was using 200mb chunks. My files are rather large so I want to use large chunks. How do I inow if this is a windows issue, isp issue, or boto3 issue or my issue with coding my script? I have successfully uploaded many small files to my S3 bucket, so I don't think it's my script. By the way, I should add that I'm also working with my ISP (T-Mobile home internet) because I do lose internet connectivity several times a day. I get the usual endpoint connection error when that happens. I don't know if my ISP issue can cause my PC to freeze up though. Also, I use half the thread available to me on my PC.

1 Answers
0

If the whole PC is freezing then it is definitely something on the PC causing that. I would start by disabling any plugins, add-ons, virus scanners (temporarily), security tools (temporarily) to see what is causing the issue. If you have another PC, try the script on that.

If it is just the task that is freezing then you'll need to debug what is happening inside the task that is stopping.

EXPERT
answered a month ago
  • I'm working with Microsoft in their forum staffed by volunteers and they are helping me, but they told me to wipe out my c drive and reinstall windows. I did that and added back my apps and windows froze again when I was running my python script to upload a large file to S3. I also have malwarebytes looking into this, in case it's malware, but I'm wondering if it's related to my AWS upload script. I didn't run it over night and it didn't freeze, but, of course, nothing was going on with my PC at that time either. I have it configured to upload 1000 mb per part and I've tried 200mb parts before, but that froze too. The file I want to upload is 136gb. Also, I do have debug logging set, so I have a lot of logs for this, but usually what it says in the log is that it's uploading a part and then nothing and since I have 6 processes going at the same time, I have no idea what process would be the one that causes it to freeze, if it is doing that. Should I look into limiting my bandwidth usage for the upload processes?

  • I would try with a different version of Python (just in case); and I would try with a different version of Windows (if possible); and also with an install of Windows that is vanilla as possible. That is, just the OS, Python and your script. Nothing else.

  • It's interesting. I switched from T-Mobile Home Internet to Xfinity yesterday. For the first time in a long time I was able to upload a large file to my AWS bucket using the unmodified python script with everything including the python version the same. My PC didn't freeze either. I've always lost internet or my PC froze when running this python script before. I was working with support at T-Mobile Home Internet and he asked me what devices were installed and what operating systems on them, so he had a lot of information. I don't know that they deliberately did anything to my PC, but it's interesting that less than 24 hours after switching ISPs, I'm seeing such a dramatic improvement. I'll keep an eye on my PC and whether it freezes or not, but I may have found the issue. I'll keep you informed. Is there anyway I can know for sure that something was being done to my PC? I did see an entry in my audit log in event viewer in Windows that audit entries were deleted around the time that my PC froze.

    I have a couple of python scripts that run when I backup my drives and they upload to my AWS bucket so I have an offsite backup. I thought that with the war in the Ukraine there might be a cyberwar going on in the future, so I thought this might help protect my PC. I use my PC a lot and I need a good backup. I have the python scripts integrated with EaseUS ToDo Backup, so they run whenever I have a backup scheduled to run. This might just work fine now.

  • Some providers (and I'm not familiar with all of them; especially not the US ones - but in general) run all sorts of "interesting" traffic inspection and interception systems on their networks. Sometimes to detect spam, botnets or other malicious traffic; sometimes for advertising purposes; sometimes for traffic compression - lots of reasons. That could be interfering with your data streams.

  • I think I've found the issue. I'm not sure though. I'll have to test some more. I have been checking how many threads are available on my machine and dividing it by two and passing that to the TransferConfig class like this:

            config = TransferConfig(multipart_threshold=1000 * mb, max_concurrency=threads,
                                    multipart_chunksize=1000 * mb, use_threads=True)

    When I upload a file using that config, my PC freezes. I'm trying it again tonight. I could just try this next:

            config = TransferConfig(multipart_threshold=1000 * mb, max_concurrency=int(psutil.cpu_count() / 2),
                                    multipart_chunksize=1000 * mb, use_threads=True)

    I don't know why the variable doesn't work though. The file I'm uploading is 266 GB in size. It's a backup of my PC and I want to use my AWS bucket as an offsite backup, so I've written a few python scripts to do that. I will post that information at some point in the future when I get it working better. Right now it seems to be working with a hard coded value of 6 for max_concurrency, but I'd like to figure out the config dynamically. I also want to figure out the other parms in that config dynamically, too. I'll work on that.

    It seems like this might be some kind of bug in boto3, but I wanted to see what you thought.

    Threads is defined as int(psutil.cpu_count() / 2) and mb is defined as 1024 ** 2.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions