What will be difference in cost of infrastructure between PySpark vs Flink Scala

When comparing PySpark and Flink Scala for your use case, there are several factors to consider in terms of cost and performance.

For PySpark:

1. Performance: PySpark may have some performance overhead due to serialization/deserialization between Python and JVM, especially when using custom Python functions. This can impact processing speed and increase resource requirements.

2. Infrastructure: You might need larger or more instances to compensate for the Python overhead, potentially increasing costs.

3. Skill set: Your team's Python skills align well with PySpark, which could lead to faster development and easier maintenance.

For Flink Scala:

1. Performance: Flink Scala typically offers better performance for stream processing due to reduced serialization/deserialization overhead. It's well-suited for near real-time use cases.

2. Infrastructure: The improved performance could translate to smaller or fewer instances needed, potentially reducing infrastructure costs.

3. Skill set: Your team may need time to adapt to Scala, which could increase initial development costs.

Considering your requirements:

• 60 MB/sec data volume is manageable for both systems.
• Near real-time processing with a 60-second delay tolerance is achievable with both, but Flink might have an edge in efficiency.
• The need for custom Python functions favors PySpark but may impact performance.

Cost difference: The exact percentage difference in cost is difficult to predict without detailed benchmarking, as it depends on specific workload characteristics and optimization levels. However, you might expect Flink Scala to be potentially 20-30% more cost-effective due to better performance and lower resource requirements. This estimate assumes proper optimization in both cases.

Recommendations:

1. If cost is the primary concern and your team can invest in learning Scala, Flink Scala might be more cost-effective in the long run.
2. If development speed and team productivity are priorities, sticking with PySpark might be better despite potentially higher infrastructure costs.
3. Consider running benchmarks with your specific workload to get a more accurate cost comparison.
4. Explore optimization techniques for PySpark, such as using Spark SQL or DataFrame operations where possible, to minimize the performance gap.

Remember that the actual cost difference will depend on your specific use case, data patterns, and the level of optimization applied to each solution.
Sources
Optimize user-defined functions -
Key topics in Apache Spark -
Real-time cost savings for Amazon Managed Service for Apache Flink | AWS Big Data Blog