This article is meant to help regarding MXFS, namely to highlight known issues, provide recommended setups for specific workflows and other helpful tips.


Settings


 

Global Settings

  1. Service TCP port
    • This is the port used to the MXFS service.
    • Defaults to 32100. This value can be changed if necessary

  2. Show MatrixStore login dialog
    • When true, users can use User Management to log in and connect to vaults
    • When false, users must supply vault credentials (normally stored in .vault files) to connect to a vault

  3. Mount volumes as 'local' volumes [Mac only]
    • Use this setting to enable fuse setting 'local' to workaround a known fuse bug (on some setups, unable to copy large files to the volume - see below for more details)
    • This setting should have minimal impact on MXFS, however, Fuse developers state that this setting is 'experimental' and that they cannot know the full affects of this setting as OSX is not open source. There are currently no known issues with the setting. If concerned, please refer to the 'known issues' below for other workarounds.

  4. Use first drive letter available [Windows only]
    • When mounting a volume, MXFS will ignore the value in the volume settings and attempt to mount the volume to the first available drive letter (excluding A and B)
    • This can be helpful when there are many volumes to track in the MXFS GUI
    • This feature does not work with auto-mounting

  5. Always remove disconnected network drives  [Windows only]
    • When you attempt to mount a volume over a disconnected network drive, MXFS will prompt you to remove the drive. With this setting enabled, the drive will be removed without prompting
    • This feature does not work with auto-mounting
    • It is still possible to experience issues when this setting is enabled. There are times when the network drive will be undiscoverable with the Windows ‘wmic’ tool (if ‘reconnect at login’ is used), in this case, you can still experience strange behaviour with the volume mounting on top of the network drive.



Volume Settings

  1. Read-Only
    • Will mount the volume in read-only mode, regardless or user and vault permissions

  2. Metadata caching
    • Enable this setting for improved performance (enabled by default)
    • When on, FileInfo returned from the server will be cached. Most filesystems make a lot of requests to access this information so caching provides a large performance increase.
    • There is an upper limit of 50,000 entries on this cache, so large directories may still have issues
    • Changes to files made elsewhere may not appear immediately

  3. Dir caching timeout
    • Set to 30 seconds by default. To disable the cache, set the value to 0.
    • When on, listed files will be cached so that subsequent listings of the same folder will be quicker.
      • The first listing will block it's thread until the files are returned from the server and processed, then repeat listings will return the cached information whilst it performs the search in the background - the server will only be queried again if the directory timeout has passed since the last listing.
    • There is an upper limit of 50,000 entries on the directory cache to prevent out of memory errors.
    • If a file is created or deleted elsewhere, the change may not appear immediately

  4. Read block size
    • This is the amount of data we return from the cluster in one read request. So for reading large files, a higher size would result in less trips to the server, but for small files it will result in returning buffers being mainly empty and wasting bandwidth

    • A larger size may also result in more memory being used. Larger sizes should be used carefully.

  5. Automount [Service only]
    • When a volume has this setting enabled, MXFS will attempt to mount the volume when the service is turned on (generally on the boot of the machine)
    • Automount will only work if the specified drive letter is available in the system - setting 'use first drive letter' does not work on automount.

  6. Extended Attributes [Mac only]
    • Mac extended attributes are a feature of Mac OS X, known as xattr
      • These attributes are additional attributes to the system attributes, such as 'com.apple.Finder', 'com.app.quarantine' or any user-supplied attributes
    • Some of these official attributes are requested often (on listing or highlighting of a file) and can cause a high load on the server. Other applications, such as QuickTime Player, may use their own extended attributes.
    • When this setting is disabled, MXFS will return dummy values instead of contacting the server for real values. If extended attributes are not needed by the customer, disabling extended attribute can provide a large improvement on performance.
      • Most applications can run with these attributes disabled as we return that the call was successful
      • Using 'xattr' on the command line will return that the command was successful, but will not return true data. Please be careful and remember your settings when using this command as no warning will be given that the extended attributes are disabled.
      • It should be noted that when this setting is off, extended attributes will not be preserved when archiving or retrieving files from the vault. Again, no warning will be given. This lack of warning is the reason that the setting is disabled by default.

  7. Spotlight Indexing [Mac only]
    • Spotlight Indexing is a feature of Mac OS X. It indexes volumes to allow for quicker searches and the ability to search for file content.
    • When this setting is enabled, Spotlight will be able to index to the content of the volume. It also allows 'tags' to be used on items in the volume.
      • It should be noted that Spotlight will re-index the content of the volume every time that it is mounted. The larger the volume, the longer it will take Spotlight to index the volume. MXFS may become slow and unresponsive until this indexing is complete
      • If this setting is disabled, existing tags will not be lost. However, they will not be preserved on files that are retrieved or archived in the volume whilst the setting is off.

  8. Disable Mac OS X Bundles [Mac Only]
    • When this setting is enabled, MXFS will ignore any requests to search for a file/folder named ‘Contents’ in a folder with a dot in the name.
    • This feature has been included because it was found that any folder that has a dot in its name will be treated as a bundle by Mac. This means that for each of these folders, Mac will be making unnecessary extra calls to the server to search for the expected package structure.
      • These extra calls can quickly add up when there are several of these folders int he same location. We saw this affect on customers who stored RED cards, which use ‘folder.RDC’ in their own structures. This made listing unnecessarily slow, something that should have taken a couple of seconds could take over a minute.
    • It is important to note that when this setting is enabled, MXFS will ignore ALL attempts to find ‘Contents’ in a folder with a dot in the name, even calls made for genuine bundles. These bundles will incorrectly display as empty whilst the setting is enabled. This is why the setting has been disabled by default, despite the large performance improvement.





Workflows - Generic Workflows

These are the recommended setups for some known use cases.

For all workflows, it is recommended to turn off ‘Extended Attributes’ if it is possible as it will increase performance. However, please be sure that these attributes are not necessary first.

If directory caching is turned off, it is recommended that the user does not use this volume (or at least this mount) to navigate the volume in explorer or finder as each search will be blocking it's thread.

It is still possible to experience issues when this setting is enabled. There are times when the network drive will be undiscoverable with the Windows ‘wmic’ tool (if ‘reconnect at login’ is used), in this case, you can still experience strange behaviour with the volume mounting on top of the network drive.  


Here you can find some more information on this topic:



Default


In most circumstances, the default setup is going to be appropriate.


    Read-only: false

    Metadata caching: false

    Extended Attributes: true

    Spotlight Indexing: false

    Disable Mac OS X Bundles: false

    Dir Caching Timeout: 30

    Automount: false

    Read Block Size: 16 MB



Heavy Searches - FS scanners, Anti-virus, etc.


If the volume is being heavily scanned by an application (i.e. anti-virus) then you will experience extreme slowdown when using the volume, and possibly the cluster. In this case, using as much caching as possible is recommended to reduce the load on the server.


In general, however, scanning of the volumes is not recommended and should be avoided where possible.


    Dir Caching Timeout: 300



Examples: Anti-virus software, MediaSpy, Windows Software Indexer


Watching Folders in the Volume

Applications may regularly scan a folder to check for changes. In these cases, we would recommend using a much smaller value for the directory cache. This would provide the benefit of repeated listings working in the background, whilst still updating in a timely manner.

Dir Caching Timeout: 10


Note: If performance is not an issue for the customer, they could set the value to 0 to get the latest results from the server on every call.

Examples: Vantage, Dalet


Accessing a Volume from Multiple Client Machines

The settings for this use case will largely be determined based on how quickly the customer needs to see the data.

In general, the default 30 seconds timeout should be appropriate. However, if it is vital that the customer doesn't see any stale data, then they can set the directory cache timeout to 0 and disable metadata cache (WARNING: Disabling the metadata cache will severely drop performance, this should only be changed if it is absolutely vital that the customer sees the updated file information straight away, such as size or access time. If they only need to see the creation/deletion of the file straight away then they can disable only the directory cache.)


Copy/Backup Tools

MXFS default settings should be appropriate for use with 3rd party transfer applications.

If problems are experienced, it is recommended to enable retries or reduce the workload on the transfer application's settings.


Examples: Syncovery, FastCopy



Workflows - Specific Workflows

This section describes specific workflows or applications using MXFS


Media Composer

Media Composer is known to make a lot of unnecessary calls to the filesystem, in particular, when auto-detecting plugins it will search for a lot of non-existent files.

To minimise the impact of this, we recommend using at least 30 seconds on the directory cache, though customers may find that increasing the value further helps their performance (Note: 300 is the maximum value).

Other settings that will greatly help this workflow are;

In Media Composer, disable AMA Management by using the command 'manageama off'
Deleting or renaming the folder 'AMA Management' (This is important is Media Composer will keep scanning this directory, even with ManageAMA off)
Windows: C:\Users\Public\Documents\Avid Media Composer\AMA Management
        Mac: 'HD/Users/Shared/AvidMediaComposer/AMA Management'


You can find or information following this link.


Mac OS X Spotlight

Enabling Spotlight Indexing on the volume will noticeably reduce performance. It will also be necessary to give Spotlight time to index the volume whenever it mounts (the large the volume is, the longer this period of time will be), therefore it is recommended to leave volumes mounted as much as possible. To avoid locking issues, Spotlight should only be enabled on a volume on one machine.

- Extended Attributes: true
   - Spotlight Indexing: true


Searching the volume using Finder/Explorer

This use of the volume will be as detrimental as scanning the volume (as long as the search is still in progress) as it will go through every item in the volume. On large volumes, this will be a very expensive action.

If you need to find a specific item in a volume, it is recommended to use DropSpot or Vision instead.


Bundles / Folders with dots in the name

The folder structure within RED cards use folders with dots in the name. We have found that Mac OS X will treat these folders differently, which can cause large performance drops when listing a parent folder with many of these folders. There is a volume setting ‘Disable Mac OS X Bundles’ that will prevent this performance cost, however, it comes at the expense of being unable to display the content of real bundles (or any Mac package type; .app, .framework, plugin etc.). As long as genuine bundles are not stored in the volume, it is safe to use this setting.

- Disable Mac OS X Bundles: true

Examples: Red cards



Apache HTTPD

Setting up Apache on a Linux SE server may run into permissions issues. To give Linux the permissions it requires to work with Apache, use the following setup;

1) Enforcement must be off

- Check with getenforce
- Set with setenforce 0

2) Update '/usr/lib/systemd/system/httpd.service'

- Comment out PrivateTmp=true followed by restarting Apache


With these setting changed, you should be able to use resources stored in the volume.



MXFS vs CIFS/SMB (Hub)

Users may choose to mount a share from a MatrixStore Hub and access to vault via the smb protocol instead of using MXFS locally.

Benefits of this are;

- It is compatible with applications that require UNC paths
Examples: MAM systems or automated systems. EVS workflows.
- Changes to files or folders are automatically visible to all clients
- Clients can receive FS notifications
- When a change is made on the hub, all clients are notified. All explorers/finders will have the same view.

Potentials issues are:

It creates a bottleneck in performance, all traffic is routed through the Hub
    It will be easier to overwork the hub as all clients will be accessing the vault through the same connection



MXS-SMB (Hub) Authentication

Authenticating Samba shares when Active Directory is enabled

Assuming that the username is 'TestUser' and the domain name of the Active Directory is 'OM'

  1. To authenticate with an AD user from any machine, use 'DOMAIN/user':
    OM/testuser

  2. To authenticate with a custom user from a Windows machine that is part of the AD domain, use just the username (lowercase)
    testuser

  3. To authenticate with a custom user from a machine that is not part of the AD domain, use any non-domain prefix (i.e. 'NOTDOMAIN/user')
    1. Connecting through a GUI
      NETBIOS/testuser or nodomain/testuser
    2. Connecting through smbclient
      smbclient //hub/testvault -U A/testuser


Active Directory security is enabled by supplying the following line in the smb.conf file

security = ads


Authenticating Samba shares when Active Directory is not enable

Assuming that the username is 'TestUser'

  1. To authenticate with a custom user from any machine, use the username (lowercase)
    testuser


The default authentication for samba is 'user level' security. To use this security, do not supply a value for 'security = ' in the smb.conf file

For details about samba security modes see: https://www.samba.org/samba/docs/man/Samba-HOWTO-Collection/ServerType.html



Samba Authentication Trouble Shooting


On Windows:


  1. You can only authenticate shares with one user at a time. To connect to shares with another user, you must first disconnect the existing shares and then try again.
  2. If you have a share authenticated with a user, then add a new vault to the hub with the same user, you will be unable to authenticate to the new vault. To connect to the new share you must first disconnect the existing shares using one of the following methods 
    - In command prompt use 'net use \\samba_host\share_name /delete' or 'net use X: /delete' (replacing X: with the desired drive letter)
    - Log out of the current Windows session and log back in again
    - Reboot the machine






Dokan

  1. When running as service, mounted volume does not always appear in the explorer but is accessible via the command prompt
    • Workarounds:
      1. Restart the explorer process. This will display any currently mounted volumes but will need to be restarted again if new volumes are mounted
        'taskkill -f -im explorer.exe' followed by 'explorer.exe'
      2. When a volume is mounted via the GUI, it will open a new Windows Explorer. This particular explorer will receive all notifications and will be able to display all mounted volumes, included ones that are mounted later. Keeping this window open will allow you to access all mounted volumes.

  2. Early versions of Dokan (previous to 0.8.0) incorrectly set the NetworkProvider name. This prevented use of the 'net use' command
    • Workaround:
      1. Upgrade to at least Dokan 0.8.0 and the compatible version of MXFS
      2. Change the names in the registry to match. IMPORTANT: Only use this option if you know what you are doing. Mistakes made in the registry can lead to serious problems
        • HKLM/SYSTEM/CurrentControlSet/Services/Dokan/NetworkProvider
        • HKLM/SYSTEM/CurrentControlSet/control/NetworkProvider/Order
  3. Dokan 1.0.0 onwards requires patch “KB3033929” on Windows 7 



You can find more information about Dokan and MXFS following this link and this link



Fuse

  1. On some systems, you may get an error stating that you are unable to copy a file due to unsupported volume. Fuse developers think this is a bug in Apple, where if another application sets 'bSupports2TBFiles' to false, the response is cached and applied to other applications such as MXFS
    • Workaround:
      1. Attempt to isolate and uninstall to application supplying this response
      2. Mount the volume using the 'olocal' parameter
        • In MXFS 3.2.6.5 and onwards, you can use 'Preferences' → 'Volumes' and enabled setting 'Mount volume and 'local' volume'
        • In builds previous to 3.2.6.5, you can mount the volume with setting ’Spotlight Indexing’ enabled to use the 'olocal' paramter. However, using this setting can worsen performance and should only be as part of a workflow. If you experience this issue and use an earlier build of MXFS, please request an upgrade.

  2. On Fuse versions 3.0.3-3.6.1, mounting will eventually fail as unmounting does not properly release the osxfuse devices until the application is restarted (23 or 63 mounts, depending on the fuse version).


Simultaneous Files Access from multiple clients

Users should be careful when accessing objects from multiple machines. Server lock permissions are;

  • If an api client has a read lock
    • The current client can request another read lock or a write lock
    • Another client can request another read lock
    • Another client will be denied a write lock
  • If the api client has a write lock
    • The current client can request a read lock
    • The current client will deny another write lock
    • Another client will deny a read lock
    • Another client will deny a write lock

It is important to remember that content and metadata share the same lock, so updating attributes such as timestamps will request a write lock on the server.

Server locks will timeout out 30 minutes after the last access to the object. This is to prevent situations where the lock may be held indefinitely, i.e. a machine crashed whilst holding the lock.

Subsequently, MXFS will release (and if necessary re-open) a stream 25 minutes after it's last use.


It is important to note that a single MXFS does not re-use it's connection to the cluster. This means that if MXFS crashes whilst it holds a read lock on a file, if you restart MXFS on the same machine you will be unable to write to that file until the original read lock times out - this new MXFS session is not the same connection that held that previous read lock.


General Tips

Cross-platform file and directory naming recommendations

The MatrixStore does not prevent applications from specifying any character as part of a file or directory name since it's an object store. It also does not rename any characters since this may break the compatibility with that application or others.

However, as MXFS is available for many operating systems, use of special characters should be avoided in file and directory names, to ensure best cross-platform compatibility.

Special characters include file path separators, punctuation and parentheses, which are reserved characters in file and directory names on some operating systems.

For best cross-platform compatibility, the use of the following characters should be avoided

  • File path separators e.g. ":" (colon), "/" (forward-slash) and "\" (back-slash), which are used as path separators on some platforms. Consider substituting these with "_" (underscore) or "-" (hyphen).
  • Punctuation, parentheses, quotation, brackets and operators e.g. . , [ ] { } ( ) ! ; " ' * ? < > | as these are often reserved or used for special functions when scripting / programming
  • White space characters e.g. spaces, tabs, new lines and embedded returns. Although supported on some platforms, scripts and applications may not handle them as expected. Consider substituting these with "_" (underscore) or "-" (hyphen).

In addition, below are some platform specific requirements.

Windows

Windows has it's own specific naming requirements detailed on the below link

https://msdn.microsoft.com/en-gb/library/windows/desktop/aa365247(v=vs.85).aspx

If you wish for files and directories to behave as expected in MXFS on Windows, the following should be considered

  • The filename cannot end with a space
  • The filename cannot end with a period
  • Names are case-insensitive
  • Avoid reserved characters including < > : / \ | ? * "
  • Avoid reserved names (detailed in above link)

In addition, MXFS requires that filenames should be less than 128 characters in length.

MacOS / OS X

MacOS / OS X has the following requirements

  • "/" (forward-slash) is not allowed in file and directory names (Finder will let you but will write a ":" in it's place)
  • ":" (colon) although allowed, should be avoided as it's display will vary depending on the application used, e.g. terminal will show ":", but Finder will display "/"

Unix

Unix platforms have the following requirements

  • "/" (forward-slash) and the NULL ("\0") character are not allowed in file and directory names