So you've started using Apache Iceberg and you're loving the features it offers. But have you thought about the cost and performance implications of using Iceberg? In this post we'll look at the new optimization features in AWS Glue that help you manage Iceberg tables.
A guide to setting up an Iceberg table in Amazon Athena, creating an Amazon Data Firehose delivery stream, and delivering data to the Iceberg table using a Preview feature in Amazon Data Firehose.
Explore how to federate S3 Access Grants using IAM Identity Center Trusted Identity Propagation in this in-depth guide. Learn to set up a secure and scalable solution for managing user and group-based permissions to S3 data across multiple client applications and AWS services. This post covers the necessary prerequisites, setting up an OAuth 2.0 authorizer, deploying S3 Access Grants and subsequent AWS IAM Identity Center applications with Terraform, and running a simple client application to interface with the federated authentication. Follow along to enhance your understanding of modern AWS security practices and ensure secure data access management.
Explore the newly supported AWS Glue table statistics feature for Apache Iceberg tables. This post provides a high-level overview of how to generate column-level aggregation statistics to significantly accelerate query performance. Follow along to set up your environment, create and load an Iceberg table, and generate valuable statistics that can optimize query execution.
Let's try out the new AWS CodeBuild feature that allows you to run GitHub Actions inside AWS accounts using the same, familiar GitHub Actions workflows.
A pretty good Terraform module template that includes semantic versioning with releases, GitHub Actions, automatic README generation and pre-commit hooks to ensure quality.
Using the new Anthropic Claude 3 Haiku model hosted by Amazon Bedrock to watch a security camera feed and alert when something interesting happens.
IoT Core, Kinesis Data Streams, Firehose, IoT Rules and Lambda... the list of services that AWS Solution architects will recommend you use to build your IoT data ingestion pipelines. But what if I told you that you could replace all of these services with a single S3 bucket? Without sacrificing any abilities to process, analyze, or visualize your data.
In this post, we'll look at the new AWS S3 Access Grants service and compare it to the traditional methods of managing access to S3 data such as S3 Access Points or IAM/Bucket policies. If you've struggle with massive policy documents in the past, or you're looking for a way to grant access to federated identities, then this post is for you!
In this post we'll take a look at how to use the new automatic compaction feature in AWS Glue, and how it can help you optimize your Iceberg tables. Use my handy helper script to create an Iceberg table, load sample data into it, and test out this new feature in a hands-on way.
Walkthrough of how a groups attribute can be customized for Entra ID SAML integration with Amazon Managed Grafana, allowing for group based access control to Grafana.
Automating the daily download of Garmin Connect data to S3 using AWS SAM and Lambda. The eventual goal is to build a dashboard to visualize our health data using AWS.
A quick look at how to use partition projections in Amazon Athena accompanied by some hands on examples that cover the basics through to more advanced use cases.
Resolving ICEBERG_VACUUM_MORE_RUNS_NEEDED errors by using AWS Step Functions to retry vacuuming Amazon Athena Iceberg tables until they are clean.
A quick look at the AWS Greengrass Testing Framework by adding it into an existing Greengrass component to efficiently test how a component behaves when actually deployed.
Improve your AWS Greengrass v2 observability by trying some of these simple tips. We explore the aws.greengrass.LocalDebugConsole and aws.greengrass.LogManager components.
A complete guide to deploying containers to Greengrass v2
I'm going to take you on the journey of building a fine-tuned GPT model that answers technical questions with answers that exclusively recommend AWS CDK.
Amazon Managed Grafana has an unfortunate limitation where API keys created have a maximum expiration of 30 days - making continuous deployments difficult without manual intervention. This post will show you how to write some simple Terraform to automatically rotate an API key that can be used in your CI/CD pipeline.
Microsoft Power Automate can simplify the process of sorting incoming emails from AWS Config and AWS Security Hub. The tool can automate the sorting, categorizing and prioritizing of alerts, freeing up your time to focus on other important tasks.
Be notified when a GitHub Action build fails by illuminating an LED connected to a Raspberry Pi, controlled over AWS IoT
A Demonstration of how you could utilize GitHub's configurable OIDC Claims to mostly automate AWS OIDC role changes.
Building a simple Amazon CodeCatalyst workflow to deploy a GatsbyJS static site to s3. This guide is meant to facilitate how you might migrate an existing CI/CD pipeline to Amazon CodeCatalyst
Learn how to set up a Continuous deployment pipeline for AWS IoT Greengrass v2 from GitHub actions. Leverage the power of the Greengrass Development Kit - Command Line Interface (GDK CLI) to streamline your workflow and deploy more often with more confidence.
Learn how to set up a simple AWS IoT Greengrass v2 Learning environment that can be used in a wider range of tutorials to get you fully educated on how to use Greengrass on projects.
Setup GitHub Codespaces with AWS IAM Roles Anywhere and authenticate your development environment with AWS securely without the need to have long-lived access tokens and ad-hoc user accounts.
When using Windows Subsystem for Linux, it is often necessary to enter the SSH passphrase on reboot. Leveraging Windows stored credentials we can automate the process of authorizing the ssh daemon on reboot.
Export data from Amazon WorkMail programmatically using CloudFormation and the AWS CLI. No longer do you have to download some shady tool from the internet to export your data.
Want to feel like a command line guru? Check out some of my recommendations to supercharge your CLI experience.
Creating a Cypress testing workflow for VSCode Devcontainers and Github Codespaces. You will learn how to setup a .devcontainer configuation that enables you to develop out of any webbrowser.
Learn how to deploy a simple python project to AWS Lambda using the Serverless Framework. We will use the LinkedIn Professional Jargon bot as a sample project.
In this post we go through creating a small function that posts artifically generated status messages on LinkedIn. These messages are generated by taking trends from the ThoughtWorks technology radar and running sentences into DeepAI's text generator.
Learn how to generate tech jargon by taking trends from the ThoughtWorks technology radar and running sentences into DeepAI's text generator.
Learn how to generate tech jargon by taking trends from the ThoughtWorks technology radar and running sentences into DeepAI's text generator.
In this post we explore the new AWS IoT Greengrass V2 and create a recipe and component for controlling an IoT cat feeder.
In this brief post we cover what it takes to keep your GatsbyJS projects up to date and functioning whilst also taking advantages of more recent features.
Do you own a Nintendo Switch that you wish you could re-purpose into a portable container cluster? Since the answer is inevitably yes, join me as we take a look at how we can turn a popular handheld console into a scalable PaaS in your pocket.
When the services I maintain day to day come under attack by a bad actor I decide to take it personally. In this post I outline how you can fight back against sites that might be high-jacking your API for their own use uses.
With the developer preview launch of AWS Amplify Flutter comes all the magic of Amplify Auth, storage, and analytics to the world of Flutter cross-platform development. Join me in this post as I stand up a simple cat photo app with this brand-new support
I got my hands on a LilyGo T-Watch 2020 and decided to build the ultimate cat lover accessory. Join me as I take you through building the ultimate cat watch.
On August 16th, 2019 Selfie2Anime went live to unexpected success. Unfortunately, the code deployed wasn't anywhere near ready for the load we experienced and some bad decisions were made in the heat of the moment to get everything up and running. 8 months on and we're dealing with the consequences of those decisions every day. In this post, I tell the tale of how we came to terms with our problems and learnt to deal with them.
Example for publishing images from an ESP32-CAM to AWS IoT over MQTT
I recently passed my Certified Kubernetes Administrator (CKA) exam and along the journey made note of a number of useful resources. Hopefully some of this information can better prepare you for your own exam.
Building a Dog Bark Detector to help understand what my dogs do while I'm not home. In this post we cover how to deploy a simple frontend that leverages AppSync resolvers for DynamoDB and build a simple Vue dashboard for interacting with the bark events.
Building a Dog Bark Detector to help understand what my dogs do while I'm not home. In this post we cover how to extract and process audio from video clips using a Serverless lambda processor and FFMPEG layers.
Building a Dog Bark Detector to help understand what my dogs do while I'm not home. In this post we cover the training and deployment of a Machine Learning module using the UrbanSound8K dataset.
Kubeflow can better equips your Data science team with a self service access to all the resources they might need to build out Machine learning pipelines and applications. Pair this with Cognito and you have a secure way to work on data projects from anywhere in the world collaboratively.
Developing simple AWS IoT projects using the Espressif IoT Development Framework (ESP-IDF) and PlatformIO
Deploy Folding@Home on Amazon Fargate quick and easily with AWS CDK
Are you a beginner to developing for The Internet of Things? Perhaps you're an expert but are curious about what new things there are to focus on in? Let's chat about what it means to develop IoT projects in 2020.
Google's Lighthouse CI is a programmatic tool that allows you to analyse your web apps and generate detailed reports around best practices building for the web. In this post we're going to explore how we might integrate these tests into our GatsbyJS build pipeline.
I recently passed my AWS Certified Solutions Architect Professional exam and along my journey of studying I scribbled down a bunch of notes I took. I thought it would be a good idea to share these random bits of information so that they can be used as flash cards / bite sized chunks of information
Join us as we go through the A to Z of IoT from an AWS perspective. We won't be using buzzwords or hand-waving, just talking about the awesome services and how they can be used more effectively.
AWS AppSync is a highly flexible API service backed with GraphQL. Recently X-Ray support was launched for AppSync, so we go through the setup process and some general usage patterns for this excellent service.
The AWS Cloud Development Kit (CDK) is a powerful way of programmatically deploying resources. I'm going to show you how you can level up your CDK skills with the VSCode debugging.
Maintaining Search Engine optimizing is crucially important migration to be considered a success. We discuss some of the methods I used to ensure SEO was not effected.
GatsbyJS backed websites need to be statically generated on changes. We work to design an awesome architecture to perform this changes for us automatically using CDK on AWS.
There were 48 WordPress powered blog posts currently hosted that need to be recreated in GatsbyJS. We look into ways of converting these posts Markdown in an Automated way.
Migration away from WordPress can be perceived as a difficult task, however utilizing GatsbyJS this process can be achieved in an afternoon. Join us as we introduce how!
When Amazon Alexa launched in 2014 as the first major home assistant platform I was captivated by the opportunities it offered someone like me. By the end of this article, you will have everything you need to build your gadget by using the source code I've provided to build this simple Alexa-powered cat feeder.
Last year around this time I was just starting to create my first blog post on DevOpStar. Take a trip with me to review this amazing year; discussing the good time and the bad. Then let's chat about 2020!
AWS IoT Secure Tunneling is a managed proxy meant for devices positioned behind secure firewalls on remote sites. A secure tunnel can be created using temporary credentials allowing access to the device on configurable ports. The secure tunneling process requires a bidirectional link to be established before communication can proceed.
AWS IoT Greengrass recently got Docker support; a much needed addition to the suite of edge tooling available at our disposal. As with most new IoT Greengrass feature, it can be a bit bumped to get setup with. I would know, I fought through the setup process on the day of release and ran into a couple quirks that I figure I'll share with the world in the form of an unofficial guide.
Deploy a learning environment called Greener Grass that effectively boostraps all the AWS IoT and Greengrass configuration for you. This allows you to be able to learn about the advanced IoT features in a friendly & repeatable way.
Dive into Detect; a sub feature of Device Defender Detect that allows you to craft your own detection profiles using both cloud-side and device-side metrics.
For this deployment we'll be describing how to setup Greengrass Core on a Raspberry Pi. This is part of a bigger post called Greener Grass.
In this guide, we'll be looking at how to build and bundle ROS applications specifically for the Raspberry Pi ARMHF architecture using Cloud9. The final goal is to have the bundled application deployed over AWS RoboMaker.
Selfie2Anime was a successful use of Image-to-Image translation using UGATIT. In this post we learn how to setup a development environment capable of running UGATIT, then train out own variation of a model.
Learn how to deploy an AWS IoT Greengrass Core node onto a Raspberry Pi with AWS CloudFormation.
Flutter CI/CD with Codemagic, Slyph & AWS Device Farm. Easy & Reliable integration testing on real hardware devices; learn how to ensure your Flutter apps run on all device types (iOS & Android)
Learn to use NoSQL Workbench with Amazon DynamoDB to create a leaderboard system with global secondary indexes.
AWS RoboMaker is a development, deployment, and simulation platform for ROS (Robot Operating System). We explore why Simulating robots on AWS is a big deal, and how you can do it too!
Serverless Certificate Vending Machine is a pattern for managing AWS IoT Devices in a secure and repeatable way. Learn how to deploy your own CVM and onboard new devices.
The development of Selfie2Anime has had its ups and downs. This post is an overview of the iterations we've made along this journey so far and what the plans are for the future
Learn how to use Conda environments to manage hundreds of different & varying python environments. Clean up your projects and make them more repeatable.
WA Animals Amazon EventBridge rule to alert Slack with a Serverless lambda when someone signs in to AWS Account
Configure a number of different Istio Ingress Gateways allowing traffic in for our various services.
Configure Istio VirtualService components to route Kiali, Jaeger, Prometheus and Grafana endpoints to the correct services.
Initialising the Istio Custom Definitions and installing Istio onto our fresh Kubernetes cluster.
A brief introduction into Istio and its role in modern production Kubernetes. We prepare for the future instalments of Practical Istio.
Deployment of Kubernetes onto GKE making use of GCP Deployment Manager. The process insures we end up with a robust private cluster to work with moving forward.
Dynamically update AWS Route53 hosted zones from your Google, Azure or On-Prem Kubernetes using external-dns.
Using Serverless to consume and processing WA Animals adoption forms using Amazon Textract and placing that data in DynamoDB
Generating Pokemon cards using a mixture of StyleGAN and RNN to create beautiful & vibrant cards ready for battle!
Facebook Messenger bot that does Object Detection on the fly using AWS Rekognition. Hosted on AWS Fargate and deployed using CloudFormation
Run OpenCV 4.0 on Lambda using Serverless framework. Extract images from scanned documents
Google Cloud Platform Deployment Manager bootstrap for GKE. Infrastructure as code for the difficult task of deploying and managing Kubernetes & Google Cloud resources
ICface is a generic face animator that is able to control the pose and expressions of a given face image. We explore how to use it for fun!
Containerising PyTorch models in a repeatable way. Deploy OpenAI's GPT-2 model and expose it over a Flask API. Finally deploy it to GCP repositories and publish it on a k8s cluster using Cloud Run.
Going through the steps required to migrate and host Animal Shelter Manager (ASM3) on Google Cloud. Includes creating and managing a GKE Kubernetes instance and images stored on GCR
Automatic coloring and shading of Dune coloring book using manga-style lineart model from deepcolor. Making use of TensorFlow and CGAN
Containerising PyTorch models in a repeatable way. Deploy OpenAI's GPT-2 model and expose it over a Flask API. Finally deploy it to AWS Fargate container hosting using CloudFormation.
Generate Fake Facebook conversations using OpenAI's GPT-2 training model and a large number of your existing facebook message data. Perform training using Amazon SageMaker GPU instances and CloudFormation.
A short kickstart project for working with Open Distro for Elasticsearch in a practical way. Load in podcast data from The Dollop and analyse the data visually using dashboards
Amazon Alexa controller for AWS IoT connected devices. Learn how to safely expose an embedded ESP8266 over MQTT to control an RGB LED.
Using Dyson Pure Cool Link python library over MQTT & Serverless framework we implement a simple endpoint to control our Dyson device
Generating Cat images using StyleGAN on AWS SageMaker. Developing a simple and repeatable Data Science pipeline for Generative Adversarial Network outputs.
Build a Serverless Resume that makes use of AWS Amplify to dynamically display work history
Take control of a magical box in AWS Sumerian using an ESP8266 embedded microcontroller connected to AWS IoT
Utilise Amazon SageMaker Ground Truth to offload manual data labelling to the Cloud or other members of your organisation.
Deploy a Serverless function that performs video watermark conversions. Create a pipeline using Lambda Layers fronting FFmpeg.
Fargate is an AWS offering that allows developers to worry even less about the systems their code runs on. Using CloudFormation you can orchestrate scalable containerised services entirely as code.
Deploy a Private VPN to EC2 by Triggering a CloudFormation deploy using your AWS IoT Button. Finally have your VPN Credentials sent to you via SNS.
Recently at Flutter Live there were a series of announcements and advancements in the development ecosystem of Flutter. Specifically Nevercode was revealed to have partnered with Google to launch a Dedicated CI/CD tool for Flutter apps. In this post we're going to go through the process of setting up an existing repository with Codemagic, this new service!
Deploy your very own Private PPTP VPN to AWS EC2 using CloudFormation. Learn how to deploy using AWS CLI or the CloudFormation GUI
Reduce security and operational complexity by managing your SSH key-pairs, usernames and hosts in the config file within your home SSH directory.