M-Stream™ File Transfer Acceleration
last updated on: April 21, 2022
Transferring large volumes of data can be very difficult. Historically it would take users hours or even days to move very large files. The file would be first downloaded from the source system onto their local desktop (if there’s enough room). Only when that’s successful can an upload to the target server begin and fingers crossed an unreliable network doesn’t derail the transfer.
M-Stream™ File Transfer Acceleration, available with the Enterprise File Fabric™ platform, allows users to transfer files much more quickly. M-Stream transfers files in parallel streams, speeding up large file downloads, uploads, and movement between storage tiers.
M-Stream File Transfer Acceleration is activated when moving and copying large files, folders and objects. It's supported both in desktop tools (uploading and downloading) and in the appliance itself (server-to-server copies). It may also be used by applications using the Enterprise File Fabric API.
The minimum file size and number of threads used for M-Stream are configurable. The number of segments may be determined based on the total size of the file and the specific target provider.
To change the minimum file size and number of threads, log in as the Appliance Administrator (appladmin) and navigate to Settings > Site Functionality.
The setting M-Stream minimum file size is the size in megabytes above which a file will split into segments for copy/move operations using multiple system threads. For best performance, we recommend a minimum file size of 100 MB. A value of '0' disables M-Stream.
The setting M-Stream number of threads controls the number of threads that will be used during an M-Stream copy or move operation. For best performance, we recommend one thread per CPU core. The default is 2 and the maximum number of threads that can be created is 10.
Both “M-Stream™ number of threads” and “M-Stream™ upload number of threads” are internally capped at 10. If you set a larger value, the effective value will be 10.
M-Stream File Transfer Acceleration is supported for storage providers that support multipart uploading and range reads (S3 and OpenStack Swift APIs) as well as file systems that support random I/O such as block-storage providers like CIFS and NFS.
Chunking must be disabled for the target provider. (An Organization Administrator can change from the Dashboard via the Provider Settings.)
For the following provider types to support M-Stream multi-stream uploads of large files, whether directly or as the destination of copy or move operations, MPU must be enabled in the provider settings.
- Mini Object Storage
- Dell EMC ECS S3
- Dell EMC Elastic Cloud Storage
- Backblaze B2 Cloud Storage
- Caringo Swarm (DataCore)
For Swift/OpenStack providers to support M-Stream multi-stream uploads of large files, whether directly or as the destination of copy or move operations, DLO or SLO must be enabled in the provider settings.
See Multipart Uploading (MPU, DLO) for more information on configuring storage providers for M-Stream.