What's the best way to filter out duplicated records in a Glue ETL Job with bookmarking enabled?

If I understand well, you are afraid that some of the new files (the old one are filtered out by the bookmark) contain duplicates, is my assumption correct?

If that is correct you could have a couple of different approaches:

1. If you want to remove duplicate based on the entire row (all columns match) you can use spark distinct() function
2. If you want to remove rows that are duplicated based on matching your key, but you are not interested in getting a specific row you can use the spark function dropDuplicates()
3. In case you have another column that you could use to find the row you want to keep (for example a datetime column with last_updated values), you could use the spark window function row_number() partitioning by your key and ordered by the datetime column desc and then keep only the row with rownum=1.

for example on the first 2 points you can look here.

hope this helps