Scenario - From Network Fileshare/device
This is the scenario where the files are generated or updated to some file share/network drives.
Sometimes, the the files may be generated to a network device that your VPC may have access to, like an on-premise file share. But unfortunately the only file storage type that Fargate can mount is EFS (as of this writing). In such a scenario (assuming that there isn't any mechanism by which the files can be uploaded to S3 bucket) the two methods discussed before wouldn't be sufficient.
One way is to use an EC2 instance, with its sole purpose is to mount the external drive as well as the EFS and copy/sync files from the external drive to EFS. The life cycle of the EC2 instance may be determined based on how often the external drives to EFS sync needs to happen. It can be an instance that is running all the time or something that comes up at scheduled intervals, does the sync and then disappears.
- Obviously, this is not a 'serverless' approach as we are dealing with an EC2 instance. We will need to worry about the AMI, the instance's security posture, patching etc. etc. depending on the organization's policy.
- On the positive side, it can be a cheaper option to achieve what we are trying to accomplish. Also, being an instance one could login there (Session Manager/SSH) and inspect the files if needed.
The options discussed so far are DYI and none of it will be of much help if we are dealing with huge amount of data and/or the files from the source are made available to Amazon FSx, object storage systems other than S3 etc. AWS provides a powerful managed solution in the form of AWS DataSync, which can be used to fulfill the scenarios that were covered so far, plus a lot more. In the next part of this series, we will explore AWS Datasync.