IoT Core, Kinesis Data Streams, Firehose, IoT Rules and Lambda... the list of services that AWS Solution architects will recommend you use to build your IoT data ingestion pipelines. But what if I told you that you could replace all of these services with a single S3 bucket? Without sacrificing any abilities to process, analyze, or visualize your data.
In this post we'll take a look at how to use the new automatic compaction feature in AWS Glue, and how it can help you optimize your Iceberg tables. Use my handy helper script to create an Iceberg table, load sample data into it, and test out this new feature in a hands-on way.
A quick look at how to use partition projections in Amazon Athena accompanied by some hands on examples that cover the basics through to more advanced use cases.
Resolving ICEBERG_VACUUM_MORE_RUNS_NEEDED errors by using AWS Step Functions to retry vacuuming Amazon Athena Iceberg tables until they are clean.