aws datasync vpc endpoint

AWS DataSync is a powerful service to move data between a range of sources and destinations. Being a managed service, it frees us from having to manage and maintain the infrastructure as well as the data transfer logic. While experimenting with AWS DataSync for a use case, I walked straight into a feature gap (or a bug, however you see it), that DataSync is unable to work with shared VPC endpoints. This blog post explains the details.

Background

The AWS accounts under consideration are connected via a transit gateway that sits in the shared services account, and the VPC endpoints required by each account for various services are created and shared from the shared services account. So individual work load accounts (say, dev, test, prod, sandbox, demo etc. etc.) don't have to manage vpc endpoints individually within those accounts. This architecture is explained in this AWS blog.

Use case

I need to copy files that are frequently generated/updated to NFS/SMB that are accessible to VPC to an EFS located in the private subnets of my VPC.

The AWS account I am working with doesn't have any VPC endpoints. Rather it uses shared VPC endpoints that are created and shared from the central network (hub) account (or the Shared Services Account)
The private subnets in the VPCs have not Internet access.
I need to avoid the data traversing the public Internet at any time during the transfer

Experimenting AWS DataSync

AWS DataSync is the one tool that tick all the boxes. I started my experiment by referring to this useful AWS guide.

Setup

In the AWS account (where I have access to the source SMB/NFS and my target EFS) as per the doc, I created

An EC2 instance using a datasync AMI (AMI ID: ami-014700ee3286f266f, AMI Name: aws-datasync-2.0.1645818139.1-x86_64-gp2) in the private subnet where I can access the SMB/NFS (Refer - https://docs.aws.amazon.com/datasync/latest/userguide/deploy-agents.html#ec2-deploy-agent)
Once the instance is up and running, logged into the instance's console via Session Manager. Once in the terminal, switch to user 'admin'.

It then presents with an 'AWS DataSync configuration' options in the console as shown below

Option 2 from the list can be used to test the network connectivity (with Public endpoints or FIPS endpoints or VPC endpoint via Private Link). In my case it is vpc endpoint and I passed the two IP addresses from the AWS DataSync VPC endpoint from the operations account.

So, the ec2 instance that I am going to deploy DataSync agent has connectivity to the VPC endpoint. I can now get an activation key from the instance and create the agent in the AWS DataSync console. From the instance terminal Option 0 let us get the activation code. I got the activation key without any issue

So far, things are going as smooth as it can. Time to create the agent in the AWS DataSync console. In AWS DataSync console, Agents -> Create Agent to create the agent.

Selected Hypervisor to deploy the agent as Amazon EC2
Service Endpoint as VPC Endpoints using AWS Private Link
Then the VPC endpoint - The dropdown suggests I don't have any DataSync VPC endpoint in this account. That is true, but I have access to the shared DataSync VPC endpoint and the EC2 instance I am using to deploy the agent confirmed that can access it (otherwise I wouldn't have got the activation key).

I left the VPC endpoint drop down empty, and manually entered the activation key (as shown in the screenshot above) and clicked the 'Create Agent' button (hoping that the dropdown not showing the VPC endpoint is a UI only issue). But that disappointed me.

Then I looked at cloudformation for creating DataSync agent. Now, that looks a bit promising, because it requires to pass the VPC Endpoint ID. So I wrote a simple cloudformation (snippet below) and tried to deploy it

But much to my disappointment, it came back with a similar error as the console did

At this point, I concluded that the shared VPC endpoints may be not supported by AWS DataSync. But I raised a support request with AWS, just to confirm that I am not doing anything wrong (luckily the AWS accounts I have been working with are covered by AWS Enterprise Support). After a couple of usual to and fro messages (asking have you done this and that and replying those questions) AWS Support has come back with this message.

The service team have responded and stated that unfortunately, shared VPCs are currently not supported by the Datasync service. We already currently have a feature request logged to combat this issue. Here are some possible workarounds:

Use public endpoints

Using public endpoints would work as it bypasses all the limitations with public/private endpoints.

Data would be encrypted in transit with a per-agent client certificate and will be mutually authenticated on each end of the transfer.

Use two transfers

The idea here is to first transfer to the account that owns the VPC (transit account) and then transfer from there to the destination.

Create a VPC Endpoint and a Filesystem in the transit account.

Create a DataSync agent in the transit account and activate it against the VPC Endpoint.

Create a source to destination sync from the source to the transit account.

Create another destination sync from the transit account's to the new destination.

So now I have the official confirmation. It took me some time to learn this, but it was interesting and time well spent. And it is good to know that AWS already has a ticket to address this short coming.