Some invocations of a Lambda function take much longer despite a warm start

Here is my AWS Lambda function:

import json
import time
import pickle

def ms_now():
    return int(time.time_ns() / 1000000)

class Timer():
    def __init__(self):
        self.start = ms_now()

    def stop(self):
        return ms_now() - self.start

timer = Timer()

from punctuators.models import PunctCapSegModelONNX
model_name = "pcs_en"
model_sentences = PunctCapSegModelONNX.from_pretrained(model_name)

with open('model_embeddings.pkl', 'rb') as file:
    model_embeddings = pickle.load(file)

cold_start = True
init_time = timer.stop()
print("Time to initialize:", init_time, flush=True)

def segment_text(texts):
    sentences = model_sentences.infer(texts)
    sentences = [
        [(s, len(model_embeddings.tokenizer.encode(s))) for s in el]
         for el in sentences]
    return sentences

def get_embeddings(texts):
    return model_embeddings.encode(texts)

def compute(body):
    command = body['command']
    
    if command == 'ping':
        return 'success'

    texts = body['texts']

    if command == 'embeddings':
        result = get_embeddings(texts)
        return [el.tolist() for el in result]
    
    if command == 'sentences':
        return segment_text(texts)
    
    assert(False)

def lambda_handler(event, context):
    global cold_start
    global init_time
    
    stats = {'cold_start': cold_start, 'init_time': init_time}
    cold_start = False
    init_time = 0

    stats['started'] = ms_now()
    result = compute(event['body'])
    stats['finished'] = ms_now()
    return {
        'statusCode': 200,
        'headers': {
            'Content-Type': 'application/json'
        },
        'body': {'result': result, 'stats': stats}
    }

This Lambda function, along with the packages and the models (so that those don't need to be downloaded), is deployed as a docker image.

In addition to the timestamps of when the function started and finished (not including the cold start initialization), the response contains the information about whether it was a cold start and how long it took to initialize. I have another function, which invokes this function 15 times in parallel.

The anomaly happens with the first of these parallel invocations. Usually, it takes ~300ms (computed as the difference of the timestamps in the response). But sometimes it takes 900ms and longer (with the same input).

This does not happen due to a cold start, since I have init_time==0 in the response (when a cold start occurs, init_time>6000). It happens both with command == 'embeddings' and with command == 'sentences'.

What could be the explanation for these spikes? With a warm start, what can cause a Lambda function to take much longer than usual?

P.S. The question at SO