AWS FOR THE REAL WORLD
โฑ๏ธ
Reading time: 8 minutes
๐ฏ
Main Learning: Migrating from Edgio to CloudFront
๐
๐ป
Hey Reader ๐๐ฝ
After a busy week in Prague, both Tobi and I (Sandro) delivered our talks and we got quite some good feedback! We will share them in a separate newsletter soon. But this newsletter is all about accessing S3 within a VPC via Gateway endpoints vs. Internet routing. We know these networking issues are not the fanciest onces (looking at you AI) but everybody who works professionally with AWS knows that these fundamentals are crucial. So, let's dive into it. IntroductionIn this blog post, we will explore the differences between using gateway endpoints and internet routing for S3 access. We'll also have a look at some common pitfalls and how to avoid them. As with most of our articles, we'll also provide you with a complete project that you can run and deploy to your own AWS account. ![]()
๐ A project demonstrating gateway endpoints s3
โญ
Stars
๐ด
Forks
๐
Issues
s3
Feel free to clone the repository and play around with the code!
What are Gateway Endpoints?Gateway endpoints are a type of VPC endpoint that allows you to access S3 (and other services like DynamoDB) from within your VPC. The answer is: mostly due to better security. But back to the question: why is it better? Well, if you use the public internet, you're exposed to all the security threats that come with it. ![]() So this is the public internet routing. How would this look like with a gateway endpoint? ![]() With a gateway endpoint, we'll route the traffic through the VPC endpoint. The best part of this is: you can use gateway endpoints at no additional cost. Setting up an Gateway Endpoint for Amazon S3Let's have a look at how to set up a gateway endpoint for Amazon S3. We'll look at a trivial example: a Lambda function that lists the contents of an S3 bucket. The Lambda function will reside in a private subnet and will only be able to access the bucket through our gateway endpoint. We'll make sure that the bucket is not accessible elsewhere by restricting access to the gateway endpoint. Let's get started! PrerequisitesSurprise surprise: we'll need to have an AWS account and our CLI needs to be configured. If you have done that, you can follow the documentation specific to your operating system. If you are using macOS, you can easily install the CLI through Homebrew by executing the command: Your credentials can be configured by running the command: Creating our VPC and SubnetsNow to the interesting part: creating our VPC and subnets. SST will do a lot of heavy lifting for us:
As you see, it will create a VPC with three private subnets in three different availability zones. Before we create our gateway endpoint, we'll also need to find the route tables of our private subnets:
Now we're ready to create our gateway endpoint! Creating a Gateway EndpointLet's do exactly that:
As you can see, we're creating a VPC endpoint with the type Creating an S3 Bucket and Restricting Access to the Gateway EndpointNow we'll create an S3 bucket and restrict access to the gateway endpoint:
In the bucket policy, we're denying access to the bucket from all principals except the gateway endpoint. You should be able to verify this when accessing your bucket in the web console: ![]() If you see this message, our bucket policy is working as expected. Creating a Lambda Function that Accesses our Bucket from our private SubnetLet's continue to the last and final step: creating a Lambda function that accesses our bucket from our private subnet through our new gateway endpoint. But first, let's create a security group that only allows outbound traffic through the HTTPS port:
Now we can create our Lambda function:
The Lambda function resides in the private subnets and will only be able to access the bucket through our gateway endpoint. Let's find out if this works as expected! Testing our SetupFor validation purposes, let's adjust our bucket policy manually once so we can upload a single file to the bucket. Afterward you can upload any file to the bucket. ![]() Don't forget to re-adjust the bucket policy after you're done testing! Now we're good to go. Let's jump to our Lambda function in the web console and invoke it via the We'll always only list the bucket's contents. ![]() As we can see, the function is able to list the bucket's contents. Common PitfallsAs always with networking configurations, there are some common pitfalls that you should be aware of. Too Restrictive Security Group ConfigurationIf your resource is not able to access the gateway endpoint, it might be due to a too restrictive security group configuration. If this is the case, your resource can't connect to the gateway endpoint. Gateway Endpoint MisconfigurationIf your gateway endpoint is misconfigured, it might be due to a wrong service name or a wrong VPC endpoint type. Confusion between Gateway and Interface EndpointsLet's wrap our heads around this one. Gateway Endpoint:
Interface Endpoint:
This means, if you need to access other services than S3 and DynamoDB, you'll need to use interface endpoints. Final ThoughtsIn this article, we've learned about the differences between using gateway endpoints and internet routing for S3 access. We've also covered some common pitfalls that you should be aware of. Generally, it's advisable to keep your traffic inside the AWS network if possible. Gateway endpoints are one great way to do that. SummaryThat's it for this newsletter! We've learned about using gateway endpoints vs internet routing for S3 access. We talked through why you should pick gateway endpoints and how to set them up step by step. We also pointed out common pitfalls and showed you how to steer clear of them. Plus, we walked you through a full demo project on GitHub with SST and the need to deploy it to your AWS account. Have you tried setting up a gateway endpoint before? We'd love to hear about your experience. See you soon ๐๐ฝ Sandro & Tobias |
We teach AWS for the real world - not for certifications. Join more than 10,500 developers learning how to build real-world applications on AWS.
AWS FOR THE REAL WORLD โฑ๏ธ Reading time: 8 minutes ๐ฏ Main Learning: How to automatically generate professional AWS architecture diagrams using Amazon Q with MCP servers ๐ Blog Post Hey Reader ๐๐ฝEver tried drawing an AWS architecture diagram and ended up spending hours just lining up boxes? Yeah, same here. Thatโs why Iโm pretty pumped about Amazon Qโs new diagram feature. You just describe your stack, andโboomโinstant architecture diagram. Itโs wild how much time this might save.Letโs take a...
AWS FOR THE REAL WORLD โฑ๏ธ Reading time: 6 minutes ๐ฏ Main Learning: Understanding Amazon Aurora DSQL pricing structure including DPUs, storage costs, and free tier ๐ Blog Post ๐ฅ YouTube Video Hey Reader ๐๐ฝin December 2024 AWS announced the preview of the serverless database DSQL and just in May it was generally available. With the announcement of the general availability, the pricing was also announced. The pricing is not super straightforward, so let's dive into it. Click to watch this video...
AWS FOR THE REAL WORLD โฑ๏ธ Reading time: 8 minutes ๐ฏ Main Learning: Building & Testing Lambda@Edge Functions with LocalStack ๐ Blog Post Hey Reader ๐๐ฝwe hope you have a great week! End of last year the CDN Edgeio was deprecated. Tobi's client company used it and the only thing that saved his day was CloudFront and Lambda@Edge Functions. But we realized that testing them and executing them locally is a pain. When we started to look for a solution we found Localstack - let's see how we can use...