Working with large data files

In this guide we show you how you can actively handle large data files in your ARC using ARC Commander.

💡 If you use ARCitect to manage your ARCs, make sure to select or unselect the boxes LFS (in the "Download ARC" panel) or Download LFS Files (in the "Versions" panel) in order to allow or prevent syncing large files (LFS = large file storage).

☑️ You have created an ARC before using the ARCitect or ARC Commander
☑️ You have a DataPLANT account
☑️ Your computer is linked to the DataHUB via personal access token

ARCs and the DataHUB come with a mechanism to sync and store large files called Large File Storage (LFS). LFS is an efficient way to store your large data files. These files are called "LFS objects". Rather than checking every file during every arc sync (ARC Commander) or DataHUB Sync (ARCitect), the tools first check whether there was a change at all. And only if this is the case, it scans what was changed. This way it saves time and computing power compared to always scanning all large files for possible changes.

The ARCitect offers to activate or deactivate the use of LFS:

• in the "Download ARC" (1) menu via the "LFS" checkbox (2)

• as well as in the "DataHUB Sync" menu (1) via the "Use Large File Storage" checkbox (2), which are available once an ARC has been open in ARCitect.

In addition you can set a threshold (2) in megabytes (MB) for what you consider a large file in the "Commit" menu (1).

Finally, you can individually download large files via right-click → "Download LFS File" (1)

By default, the ARC Commander tracks the following files via LFS:

1. All files stored in an assay's dataset folder, and
2. All files with a size larger than 150 MB.

The threshold of 150 MB can easily be adjusted using the ARC Commander. For instance, if you want to decrease it to 5 MB (i.e. 5000000 bytes), run

💡 The LFS system is also the reason why git LFS needs to be installed prior to using the ARC Commander.

In addition to the defaults, you can also actively choose, which files to track via LFS.

1. Update your local ARC via arc sync
2. Add large files or folders by copying or moving them to your ARC
3. Track files via

4. Sync your ARC to the DataHUB via arc sync
5. Open your ARC in the DataHUB and navigate to the folder with LFS objects and see them flagged as "LFS" (1).

⚠️ Please avoid uploading large files without git LFS (i.e. accidentally with pure git, when git-lfs is not available).

Sometimes you may want to download your ARC to a smaller computer, where you do not need a full copy of your ARC including all its large data files. For instance, you just want to work with smaller derived data sets or want to update ISA metadata. In this case, you can add the -n or --nolfs flag to your arc get command:

For example, have a look at the ARC https://gitlab.nfdi4plants.de/shiltemann/physcomitrium-patens-light-signaling-2022/. In the DataHUB this ARC has a storage volume of ~84GB (December 2023), most of which comes from the large RNASeq data files flagged as "LFS".

You can download this ARC without the LFS objects via

If at some point you wish to selectively download one or more of the LFS objects of your ARC to that machine, you can do so via git lfs pull --include "<path/to/fileOrFolder>"

For example, the following command will download one of the large RNASeq data files.

If at some point you wish to download all LFS files of your ARC, you can use the following command

If at some point you would like to check how much free storage you have for your ARC, you can easily do so by navigating to your ARC in the DataHUB and clicking on "Project Storage" in the right sidebar (1).

DataPLANT Support