Provider Synchronization

The Enterprise File Fabric indexes storage providers to provide a number of enhanced file services including fast browsing and search. The File Fabric has its own indexing engine to provide these services.

Applications may update/upload/delete objects through the File Fabric or in a bi-modal fashion directly through storage APIs such as S3 and SMB. When objects or files are updated directly using storage APIs the File Fabric metadata must also be updated. There are a number of ways that the metadata can be updated:

• On Demand – A user can select “Cloud Refresh” when browsing via the web, mobile or desktop apps. This user driven event will re-sync the folder as needed. Alternatively, by enabling “Real-time refresh”, metadata is updated automatically, as folders are browsed. Finally, a third option utilizing “Folders auto-refresh” triggers asynchronous folder refreshes.
• Periodically – To update the metadata for an entire S3 account (provider), a “Provider Re-sync” can be initiated. Running as a background task the File Fabric re-syncs metadata for ALL files/folders within the S3 provider. Periodic re-syncs can be scheduled to run (daily, weekly, etc.) via the Dashboard.
• On S3 Event – The S3 bucket creates an event on object creation or deletion which is sent and processed by the File Fabric.

When the Enterprise File Fabric initially connects to your storage platform, it performs a full provider synchronization. In this process, the File Fabric builds a metadata model for your storage platform, recording information such as the files and folders that are there. At this stage, the File Fabric does not download or read any of the files, instead, it simply records the metatdata of the files.

Out of the box, the File Fabric's synchronization process will be confined to a single system process. As of File Fabric release 1901.06, the performance speed of the initial synchronization process is configurable which will help you reduce the time it takes for initial synchronization.

To set the number of threads to be used for tis initial synchronization, set / replace the following line inside your config.inc.php file.

var $initialsyncthreads = 2;

The number set here will control the number of concurrent processes permitted to run for a single synchronization task running on your appliance.

You should take care when choosing an appropriate value, as high values could cause performance impacts to the overall platform.