Working with big files

Storing large (a few megabytes) binary files in Git is usually a bad idea. They are contained in the history forever and blow up your repository size, which annoys your contributors and increases storage cost on Codeberg.

But there is a good solution: Using Git LFS (Large File Storage), you can easily manage binary files, and remove them permanently when they are no longer necessary.

The files are still being tracked with Git but the actual content is stored elsewhere on the Codeberg server side.

Installation

git-lfs is a command line plugin for Git.

You can install it from the git-lfs-Website or via the package manager of your distribution. A more detailed installation description can be found at the Installation article of the git-lfs wiki.

After the installation of the git-lfs plugin, it needs to be enabled in your Git configuration.

This is achieved by calling:

git-lfs install

You need to do this only once for every os user you want to use git-lfs with.

Tracking large files with git-lfs

To track files with git-lfs the file- and/or path-names must be configured to be managed by git-lfs.

git lfs track $fileorpathname

You can add filenames, directories or even wild-card representations of files (e.g. images/** for all files in the images/ directory, or **.dat for all files ending with .dat).

This creates a .gitattributes file containing (apart from other non lfs-related stuff) the configuration of git-lfs. Be sure to add it to your repository as well.

git add .gitattributes

Now you can work with Git as usual (and actually forget about the lfs stuff). git-lfs will transparently handle the use of lfs on the configured files for you.

Which files are currently tracked?

To find out which file patterns are currently configured to be tracked with lfs:

git lfs track

To list the actual file name which are tracked with lfs:

git lfs ls-files

What is locking?

When you push your changes to Codeberg it might occur, that git-lfs tells you to enable locking:

Locking support detected on remote "origin". Consider enabling it with:
$ git config lfs.https://codeberg.org/your-user-name/your-repository.git/info/lfs.locksverify true

Locking ensures that the files you are trying to push not modified at the same time.

Codeberg LFS supports locking and it is probably a good idea to activate it. Further details on the locking mechanism can be found in the git-lfs wiki article on File Locking.

git-lfs and the web interface

Unfortunately git-lfs files cannot be edited in the web interface. However, that should not be a problem as the files configured to be tracked by lfs should be big in terms of file size anyway.

Reduce local repository size

You can reduce the size of the local repository (aka the .git folder) by calling git lfs prune. This will remove all files that are not in the current HEAD.

Caveat: This means that older versions of the files are no longer stored on your local disk. If you check out an older commit that still references the file, it will be downloaded from the lfs server. If the server side is not available the requested version of the file will be inaccessible.

Enabling and Disabling LFS

Use git lfs migrate to enable or disable the tracking of files using lfs in an existing repository. All changes done by lfs-migrate are done in your local working copy. Nothing will be done on the server side unless you push your changes to the server.

Note: git lfs migrate will rewrite the history of your repository. So be sure that no one else is working with the repository during your changes. Also keep in mind, that all users must refresh their local repositories prior to doing changes to the altered repository.

Enabling LFS in an existing repository

If you already have **a big file ** in your repository, it will not automatically be moved to lfs by just enabling the tracking.

You can first run git lfs migrate info to see which files take up the most space in your repository. Add --everything to see the sizes of all files (even the smaller ones).

Then you can run git lfs migrate import to import some or all files into using lfs.

As the man page of git-lfs-migrate(1) already contains a lot more detailed description on how to import files, you are encouraged to read it to learn more about importing before starting the migration.

Disabling LFS

If you want to disable lfs in your repository or for specific files, you can use git lfs migrate export.

Note you must always specify a reference or --everything when using the export command so that git-lfs knows whether you want to stop using lfs for the files in a specific reference or in your whole repository.

Example:

git lfs migrate export --include=\*.rnd --everything

This disables lfs for all appearances of files ending with .rnd in your repository.

The man page of git-lfs-migrate(1) further hints on how to export files to disable the tracking of the files by lfs.

Further reading


Hey there! 👋 Thank you for reading this article!

Is there something missing or do you have an idea on how to improve the documentation? Do you want to write your own article?

You're invited to contribute to the Codeberg Documentation at its source code repository, for example by Adding a pull request or joining in on the discussion in the issue tracker.

For an introduction on contributing to Codeberg Documentation, please have a look at the Contributor FAQ.

© Codeberg Docs Contributors. See LICENSE