Differences
This shows you the differences between two versions of the page.
| Both sides previous revisionPrevious revisionNext revision | Previous revision | ||
| multipartuploading [2024_02_09 18:53] – steven | multipartuploading [2024_02_28 01:03] (current) – external edit 127.0.0.1 | ||
|---|---|---|---|
| Line 1: | Line 1: | ||
| - | # M-Stream | + | # Multipart Uploading (MPU, DLO) |
| + | ##### last updated on: March 27, 2022 | ||
| + | Multipart uploading (MPU) refers to the ability to create a single large file or object on a server by uploading it in separate segments. Multipart uploading is supported by many object storage platforms and is preferred, and even required, when creating objects over a certain size. | ||
| - | Last updated | + | Access Anywhere can use multipart uploading for object storage providers that support the MPU (S3 Compatible) or DLO (OpenStack Swift). It can be enabled |
| - | Transferring large volumes of data can be very difficult. Historically it would take users hours or even days to move very large files. The file would be first downloaded from the source system onto their local desktop (if there’s enough room). Only when that’s successful can an upload to the target server begin and fingers crossed an unreliable network doesn’t derail the transfer. | + | ##Appliance Configuration |
| - | M-Stream® File Transfer Acceleration allows users to transfer files much more quickly. M-Stream transfers files in parallel streams, | + | A file will be uploaded in parts for an S3-compatible provider when greater than a specified “upload size”. This file size is set per provider but an appliance default can be configured. |
| - | {{ :: | + | The appliance default can be set by the Appliance Administrator (appladmin) under Settings > Site Functionality. If the field is blank a setting of 4096 is used. |
| - | M-Stream File Transfer Acceleration is activated when moving and copying large files, folders and objects. It's supported both in desktop tools (uploading and downloading) and in the appliance itself (server-to-server copies). It may also be used by applications using the Enterprise File Fabric API. | + | {{ :: |
| - | ## When is M-Stream Used | + | ##Provider Configuration |
| - | M-Stream is used when moving data between any of the following | + | Multipart uploading will only be used if enabled for a storage |
| - | * CIFS/SMB | + | {{ : |
| - | * Swift API (SwiftStack, | + | |
| - | * S3 API (certified on AWS S3, Cloudian, IBM COS) | + | |
| - | * Azure Blob | + | |
| - | * Filesystem (FS, Local block storage, StorNext, Ceph FS and many others) | + | |
| - | M-Stream will also accelerate moves, copies and renames (because renames require copying | + | The Multipart upload size setting is optional. If blank the appliance " |
| - | M-Stream | + | <WRAP center round info 100%> |
| + | If you set an explicit MPU size for a provider then that size will always be used with that provider. | ||
| + | </ | ||
| - | ## M-Stream for Upload | ||
| - | M-Stream streams parts in parallel directly to storage | + | OpenStack Swift providers |
| - | For example when: | + | {{ :multipartuploading: |
| - | * Direct upload is used (when both client and provider have ' | ||
| - | * Chunking is enabled (uncommon) | ||
| - | * File uploaded through FTP (as the file size is not provided) | ||
| - | * Encryption is enabled | ||
| - | * Antivirus scanning is enabled. | ||
| - | ## Enabling M-Stream | ||
| - | To enable M-Stream follow the steps below: | ||
| - | |||
| - | ### From Appladmin Account | ||
| - | |||
| - | Go to Settings >> Site Functionality, | ||
| - | |||
| - | * M-Stream minimum file size: | ||
| - | * If set to 0, M-Stream will be disabled | ||
| - | * If set too small, then too much overhead will be added, adversely affecting the performance. | ||
| - | * If set too large, then full potential of M-Stream will not be utilized | ||
| - | * Default multipart upload size (This only applies to S3 based storage and needs to be between 5MB and 4GB) | ||
| - | * Ranged read block size for FS and HPSS providers: This only applies to FS or HPSS providers and specifies the block size for reading data. We recommend 4096 (4K) to (4M). Must be a base 2 integer. | ||
| - | * Write block size for FS and HPSS providers: (Same as point 4, but controls writing to FS or HPSS) | ||
| - | |||
| - | * M-Stream number of threads: | ||
| - | * Should be set to 10 or (no. of web node vCPUs - 2), whichever is lower | ||
| - | * Enable Direct Upload: Should be set to Off | ||
| - | |||
| - | ### From Team Admin Account | ||
| - | |||
| - | Go to Dashboard >> Provider Settings: | ||
| - | |||
| - | * Set Chunking " | ||
| - | * Turn MPU/DLO " | ||
| - | |||
| - | Go to Policies >> Encryption: | ||
| - | |||
| - | * Disable Encryption. (Accelerated transfer will not be used for files encrypted with SME encryption. SSE encryption or other storage encryption allow accelerated transfer.) | ||
| - | |||
| - | Go to Policies >> Security: | ||
| - | |||
| - | * " | ||
| - | |||
| - | ## Enabling For Windows Tools | ||
| - | |||
| - | From Cloud Tools >> Control Panel >> Bandwidth: | ||
| - | |||
| - | * Enable Allow multistream downloads | ||
| - | * Pick minimum file size (100 - 4096 MB) | ||
| - | * Pick up to 4 streams of data (Max 1 per CPU) | ||
| - | * Determine if Cloud Drive should also use M-Stream | ||
| - | |||
| - | ## Notes | ||
| - | |||
| - | * Provider must support MPU/DLO and range read. | ||
| - | * File system must support Random I/O. (Controlled by the read / write block size) | ||
| - | * M-Stream requires no user interaction. It works automatically when conditions allow. | ||
| - | * If a file is being uploaded using M-Stream then MD5 of the file will not be calculated, even if the feature " | ||
| - | * For provider configuration see [[multipartuploading]]. | ||