File Sync
Categories:
3 minute read
File syncing allows a user to transfer files to the data store as well as upload files from the data store.
Transfer Files
A user can populate the data store by transferring files from an Amazon S3 bucket or from an HTTP endpoint.
Amazon S3 Transfer
A user can select to transfer a file from an S3 URI.
Sync File
| Field Name | Description |
|---|---|
| S3 URI | The S3 URI for the location of the file to transfer. |
| Datastore Destination | The data store location where the file will be transferred to. |
Configure Authentication
In order to be able to utilize the S3 File Sync option, valid AWS credentials need to be configured on the node for the applicable S3 buckets.
Configure Bandwidth Throttling
The S3 File Sync service will always try to utilize as much bandwidth as possible to perform the transfer of the file. If the amount of bandwidth needs to be limited, a user can configure the amount of bandwidth allowed to be utilized in Kbps/Mbps/Gbps.
| Field Name | Description |
|---|---|
| Max Concurrent Requests | The maximum number of concurrent requests allowed. |
| Max Bandwidth | The maximum bandwidth that will be consumed for uploading and downloading data to and from Amazon S3. |
| Bandwidth Rate | Rate associated to the Max Bandwidth specified. Supported values are Kbps, Mbps, and Gbps. |
HTTP Endpoint Transfer
A user can select to transfer a file from an HTTP endpoint. The endpoint has to be unauthenticated since the node currently doesn’t support providing credentials.
Sync File
| Field Name | Description |
|---|---|
| HTTP URI | The HTTP URI for the location of the file to transfer. |
| Datastore File Name | The transferred file will be saved with this name under the datastore destination. |
| Hash (Optional) | Optional fields used to validate the integrity of the transferred file. |
| Hashing Algorithm | If a hash is specified, this specifies the hashing algorithm used during validation of the file. |
Upload Files
A user can extract files from the datastore by uploading them to an Amazon S3 bucket or to an HTTP endpoint.
Amazon S3 Upload
A user can select to transfer a file to an S3 URI.
File Upload
| Field Name | Description |
|---|---|
| S3 Bucket URI | The S3 bucket URI for the location where the file will be uploaded. |
| Datastore File | The data store file to upload. |
HTTP Endpoint Upload
A user can select to upload a file to an HTTP endpoint. The endpoint has to be unauthenticated since the node currently doesn’t support providing credentials.
File Upload
| Field Name | Description |
|---|---|
| HTTP URI | The HTTP URI for the location where the file will be uploaded. |
| Datastore File | The data store file to upload. |
| Use Multipart | Used to denote if multipart should be used when uploading the file. True for use multipart and false otherwise. |
Feedback
Was this page helpful?
Glad to hear it! Please tell us how we can improve.
Sorry to hear that. Please tell us how we can improve.