Joining fragments stored in KVS and copy the file to S3 bucket

Question

For the past 5 days I have been trying to write a simple code, I'll keep it simple.
but when I iterate over the chunks and group and write them into my local hard-drive I get a corrupted file, which shows only roughly 20 seconds out of 10 minutes footage.

```
# Step 1: Get the endpoint for the Kinesis Video Stream
    try:
        endpoint = kvs_client.get_data_endpoint(
            StreamName=stream_name,
            APIName="GET_MEDIA"
        )['DataEndpoint']
    except Exception as e:
        print(f"Error getting KVS data endpoint: {e}")
        return {'statusCode': 500, 'body': f"Failed to get KVS endpoint: {e}"}

print(f'The endpoint is: {endpoint}')
    # Initialize KVS media client with the endpoint
    kvs_media_client = boto3.client('kinesis-video-media', endpoint_url=endpoint)

# Step 2: Get media from the KVS stream
    try:

response = kvs_media_client.get_media(
            StreamName=stream_name,
            StartSelector={'StartSelectorType': 'EARLIEST'}  # Start from the latest available fragment
        )
        pprint(f'The response got : {response}')
    except Exception as e:
        print(f"Error retrieving media from KVS: {e}")
        return {'statusCode': 500, 'body': f"Failed to retrieve media: {e}"}

# Step 3: Save video data to a temporary file
    temp_file_path = '/tmp/10minutes_video_should_be.mp4'
    with open(temp_file_path, 'wb') as file:
        for chunk in response['Payload'].iter_chunks():
            file.write(chunk)
```

So, how come it fails? any suggestions will be great.
I would expect the exact file size of the video file downloaded manually from KVS `media-playback` will match size as the footage I saved locally in the code above.

can someone elaborate how to do it? any reliable resource on github will be great.
Thank you

re:Post Agent · Answer

The issue you're experiencing with your code is likely related to how Kinesis Video Streams (KVS) stores and retrieves video data. KVS stores video as a series of fragments, and when you're retrieving the data, you need to ensure you're getting all the fragments and properly reconstructing the video file.

Here are a few suggestions to improve your code and potentially resolve the issue:

1. Instead of using 'EARLIEST' as your StartSelector, consider using a specific timestamp or fragment number to ensure you're getting the exact 10-minute footage you want.

2. The 'iter_chunks()' method you're using might not be the best approach for reconstructing the video file. Instead, you should parse the MKV (Matroska) container format that KVS uses.

3. You may need to handle the continuation token to ensure you're getting all the fragments of your video.

4. Consider using the GetMedia API in a loop to retrieve all fragments, and then combine them properly.

Here's a rough outline of how you might modify your approach:

1. Use GetDataEndpoint to get the endpoint for GET_MEDIA.
2. Use GetMedia to start retrieving fragments.
3. Parse the MKV container format to extract the video data.
4. Continue calling GetMedia with the continuation token until you've retrieved all fragments.
5. Properly combine the fragments to reconstruct the full video file.

Unfortunately, there isn't a simple, one-size-fits-all solution for this process, as it depends on the specifics of your video stream and how you want to handle the data. You may need to use a library that can handle MKV parsing and reconstruction, or implement the MKV parsing yourself.

For a more reliable implementation, you might want to consider using the AWS SDK for your preferred programming language, which often includes helper functions for dealing with KVS data. Additionally, you could look into using AWS services like Kinesis Video Streams Parser Library or exploring open-source projects on GitHub that specifically deal with KVS video reconstruction.

Remember that working with streaming video data can be complex, and you may need to handle various edge cases and error conditions to create a robust solution.  
**Sources**  
[Capture & Record Video Streams  - Amazon Kinesis Video Streams - AWS](https://aws.amazon.com/kinesis/video-streams/?amazon-kinesis-video-streams-resources-blog.sort-by=item.additionalFields.createdDate&amazon-kinesis-video-streams-resources-blog.sort-order=desc)  
[Extract images from video streams - Amazon Kinesis Video Streams](https://docs.aws.amazon.com/kinesisvideostreams/latest/dg/images.html)

Joining fragments stored in KVS and copy the file to S3 bucket

Relevant content