Unable to retrieve all EC2 instances

0

I'm writing a code to get EC2 On-Demand prices for a particular OS in a determined region (eg. 'Ubuntu Pro' and 'us-east-1') but I only get 661 instances out of the 705 available on the pricing list. Is there a way that I can retrieve all the instances available for said region and OS - with the correct price?

Pricing list:

Enter image description here

Database:

Enter image description here

Python Code:

import boto3
import json
import os
from database import insert_into_table

def ec2_prices(region, op_system, database_name, table_name):
    desired_region = region
    pricing_region = 'us-east-1'

    ec2            = boto3.client('ec2', region_name=desired_region)
    pricing_client = boto3.client('pricing', region_name=pricing_region)

    response       = ec2.describe_instance_type_offerings(LocationType='region')
    instance_types = [instance_type['InstanceType'] for instance_type in response['InstanceTypeOfferings']]
      
    for instance_type in instance_types:
        response = pricing_client.get_products(
            ServiceCode='AmazonEC2',
            Filters=[
                {'Type': 'TERM_MATCH', 'Field': 'instanceType', 'Value': instance_type},
                {'Type': 'TERM_MATCH', 'Field': 'operatingSystem', 'Value': op_system},
            ],
        )
        for product in response['PriceList']:
            product_dict         = json.loads(product)
            attributes           = product_dict.get('product').get('attributes')
            
            instance_region      = attributes.get('regionCode')
            instance_region_name = attributes.get('location')
            instance_memory      = attributes.get('memory')[:-3]
            instance_vcpu        = attributes.get('vcpu')
            instance_os          = attributes.get('operatingSystem')
            tenancy              = attributes.get('tenancy')
            
            if instance_region == desired_region and tenancy == 'Shared':
                terms = product_dict.get('terms', {}).get('OnDemand', {})
  
                for term_key, term_value in terms.items():
                    price_dimensions = term_value['priceDimensions']

                    for _, price_dimension_value in price_dimensions.items():
                        price       = price_dimension_value['pricePerUnit']  
                        description = price_dimension_value['description']                      
                        value       = float(price.get('USD', 0))
                        
                        if value > 0 and f'{op_system} {instance_type}' in description:                            
                            insert_into_table(
                                database_name, 
                                table_name, 
                                instance_type, 
                                value, 
                                instance_region, 
                                instance_vcpu, 
                                instance_memory, 
                                instance_os
                            )

There are some tags (tenancy and description) that are used for retrieving the right price for the instance.

Maria
asked 4 months ago148 views
2 Answers
0

Hi,

Your code looks correct. But, you make 2 important API calls: instance list and pricing info,

Did you check how many answers you get in your for instance_type in instance_types: ? 705 or something different ?

Then, if you have all instance types, the pricing for a given instance may be missing.

So, I would all some counters in your code to see if you get all instance types or not and then check if issue comes with pricing and collect the types for which it's missing to understand.

Best,

Didier

profile pictureAWS
EXPERT
answered 4 months ago
  • Hello!

    It gets 761 instances - the problem is with the pricing, it's probably missing. If I delete the tenancy flag, it gets 701 instances, but those don't have the correct price.

    Apparently the API can't retrieve all the instance prices available on the Pricing List, but I'm not entirely sure.

0

The output from that API call might be paginated. One thing your code isn't doing is checking for the existence of NextToken in the output - that can be sent to the next API call to continue to get results.

Ref: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ec2/client/describe_instance_type_offerings.html

profile pictureAWS
EXPERT
answered 3 months ago
  • Unfortunately there's no NextToken parameter in the output. The results are all there, so the API isn't really getting all the instances available in the price list for determined region and OS.

You are not logged in. Log in to post an answer.

A good answer clearly answers the question and provides constructive feedback and encourages professional growth in the question asker.

Guidelines for Answering Questions