Documentation/git-maintenance.txt

   1 git-maintenance(1)
   2 ==================
   3
   4 NAME
   5 ----
   6 git-maintenance - Run tasks to optimize Git repository data
   7
   8
   9 SYNOPSIS
  10 --------
  11 [verse]
  12 'git maintenance' run [<options>]
  13 'git maintenance' start [--scheduler=<scheduler>]
  14 'git maintenance' (stop|register|unregister) [<options>]
  15
  16
  17 DESCRIPTION
  18 -----------
  19 Run tasks to optimize Git repository data, speeding up other Git commands
  20 and reducing storage requirements for the repository.
  21
  22 Git commands that add repository data, such as `git add` or `git fetch`,
  23 are optimized for a responsive user experience. These commands do not take
  24 time to optimize the Git data, since such optimizations scale with the full
  25 size of the repository while these user commands each perform a relatively
  26 small action.
  27
  28 The `git maintenance` command provides flexibility for how to optimize the
  29 Git repository.
  30
  31 SUBCOMMANDS
  32 -----------
  33
  34 run::
  35         Run one or more maintenance tasks. If one or more `--task` options
  36         are specified, then those tasks are run in that order. Otherwise,
  37         the tasks are determined by which `maintenance.<task>.enabled`
  38         config options are true. By default, only `maintenance.gc.enabled`
  39         is true.
  40
  41 start::
  42         Start running maintenance on the current repository. This performs
  43         the same config updates as the `register` subcommand, then updates
  44         the background scheduler to run `git maintenance run --scheduled`
  45         on an hourly basis.
  46
  47 stop::
  48         Halt the background maintenance schedule. The current repository
  49         is not removed from the list of maintained repositories, in case
  50         the background maintenance is restarted later.
  51
  52 register::
  53         Initialize Git config values so any scheduled maintenance will start
  54         running on this repository. This adds the repository to the
  55         `maintenance.repo` config variable in the current user's global config,
  56         or the config specified by --config-file option, and enables some
  57         recommended configuration values for `maintenance.<task>.schedule`. The
  58         tasks that are enabled are safe for running in the background without
  59         disrupting foreground processes.
  60 +
  61 The `register` subcommand will also set the `maintenance.strategy` config
  62 value to `incremental`, if this value is not previously set. The
  63 `incremental` strategy uses the following schedule for each maintenance
  64 task:
  65 +
  66 --
  67 * `gc`: disabled.
  68 * `commit-graph`: hourly.
  69 * `prefetch`: hourly.
  70 * `loose-objects`: daily.
  71 * `incremental-repack`: daily.
  72 --
  73 +
  74 `git maintenance register` will also disable foreground maintenance by
  75 setting `maintenance.auto = false` in the current repository. This config
  76 setting will remain after a `git maintenance unregister` command.
  77
  78 unregister::
  79         Remove the current repository from background maintenance. This
  80         only removes the repository from the configured list. It does not
  81         stop the background maintenance processes from running.
  82 +
  83 The `unregister` subcommand will report an error if the current repository
  84 is not already registered. Use the `--force` option to return success even
  85 when the current repository is not registered.
  86
  87 TASKS
  88 -----
  89
  90 commit-graph::
  91         The `commit-graph` job updates the `commit-graph` files incrementally,
  92         then verifies that the written data is correct. The incremental
  93         write is safe to run alongside concurrent Git processes since it
  94         will not expire `.graph` files that were in the previous
  95         `commit-graph-chain` file. They will be deleted by a later run based
  96         on the expiration delay.
  97
  98 prefetch::
  99         The `prefetch` task updates the object directory with the latest
 100         objects from all registered remotes. For each remote, a `git fetch`
 101         command is run. The configured refspec is modified to place all
 102         requested refs within `refs/prefetch/`. Also, tags are not updated.
 103 +
 104 This is done to avoid disrupting the remote-tracking branches. The end users
 105 expect these refs to stay unmoved unless they initiate a fetch.  However,
 106 with the prefetch task, the objects necessary to complete a later real fetch
 107 would already be obtained, making the real fetch faster.  In the ideal case,
 108 it will just become an update to a bunch of remote-tracking branches without
 109 any object transfer.
 110
 111 gc::
 112         Clean up unnecessary files and optimize the local repository. "GC"
 113         stands for "garbage collection," but this task performs many
 114         smaller tasks. This task can be expensive for large repositories,
 115         as it repacks all Git objects into a single pack-file. It can also
 116         be disruptive in some situations, as it deletes stale data. See
 117         linkgit:git-gc[1] for more details on garbage collection in Git.
 118
 119 loose-objects::
 120         The `loose-objects` job cleans up loose objects and places them into
 121         pack-files. In order to prevent race conditions with concurrent Git
 122         commands, it follows a two-step process. First, it deletes any loose
 123         objects that already exist in a pack-file; concurrent Git processes
 124         will examine the pack-file for the object data instead of the loose
 125         object. Second, it creates a new pack-file (starting with "loose-")
 126         containing a batch of loose objects. The batch size is limited to 50
 127         thousand objects to prevent the job from taking too long on a
 128         repository with many loose objects. The `gc` task writes unreachable
 129         objects as loose objects to be cleaned up by a later step only if
 130         they are not re-added to a pack-file; for this reason it is not
 131         advisable to enable both the `loose-objects` and `gc` tasks at the
 132         same time.
 133
 134 incremental-repack::
 135         The `incremental-repack` job repacks the object directory
 136         using the `multi-pack-index` feature. In order to prevent race
 137         conditions with concurrent Git commands, it follows a two-step
 138         process. First, it calls `git multi-pack-index expire` to delete
 139         pack-files unreferenced by the `multi-pack-index` file. Second, it
 140         calls `git multi-pack-index repack` to select several small
 141         pack-files and repack them into a bigger one, and then update the
 142         `multi-pack-index` entries that refer to the small pack-files to
 143         refer to the new pack-file. This prepares those small pack-files
 144         for deletion upon the next run of `git multi-pack-index expire`.
 145         The selection of the small pack-files is such that the expected
 146         size of the big pack-file is at least the batch size; see the
 147         `--batch-size` option for the `repack` subcommand in
 148         linkgit:git-multi-pack-index[1]. The default batch-size is zero,
 149         which is a special case that attempts to repack all pack-files
 150         into a single pack-file.
 151
 152 pack-refs::
 153         The `pack-refs` task collects the loose reference files and
 154         collects them into a single file. This speeds up operations that
 155         need to iterate across many references. See linkgit:git-pack-refs[1]
 156         for more information.
 157
 158 OPTIONS
 159 -------
 160 --auto::
 161         When combined with the `run` subcommand, run maintenance tasks
 162         only if certain thresholds are met. For example, the `gc` task
 163         runs when the number of loose objects exceeds the number stored
 164         in the `gc.auto` config setting, or when the number of pack-files
 165         exceeds the `gc.autoPackLimit` config setting. Not compatible with
 166         the `--schedule` option.
 167
 168 --schedule::
 169         When combined with the `run` subcommand, run maintenance tasks
 170         only if certain time conditions are met, as specified by the
 171         `maintenance.<task>.schedule` config value for each `<task>`.
 172         This config value specifies a number of seconds since the last
 173         time that task ran, according to the `maintenance.<task>.lastRun`
 174         config value. The tasks that are tested are those provided by
 175         the `--task=<task>` option(s) or those with
 176         `maintenance.<task>.enabled` set to true.
 177
 178 --quiet::
 179         Do not report progress or other information over `stderr`.
 180
 181 --task=<task>::
 182         If this option is specified one or more times, then only run the
 183         specified tasks in the specified order. If no `--task=<task>`
 184         arguments are specified, then only the tasks with
 185         `maintenance.<task>.enabled` configured as `true` are considered.
 186         See the 'TASKS' section for the list of accepted `<task>` values.
 187
 188 --scheduler=auto|crontab|systemd-timer|launchctl|schtasks::
 189         When combined with the `start` subcommand, specify the scheduler
 190         for running the hourly, daily and weekly executions of
 191         `git maintenance run`.
 192         Possible values for `<scheduler>` are `auto`, `crontab`
 193         (POSIX), `systemd-timer` (Linux), `launchctl` (macOS), and
 194         `schtasks` (Windows). When `auto` is specified, the
 195         appropriate platform-specific scheduler is used; on Linux,
 196         `systemd-timer` is used if available, otherwise
 197         `crontab`. Default is `auto`.
 198
 199
 200 TROUBLESHOOTING
 201 ---------------
 202 The `git maintenance` command is designed to simplify the repository
 203 maintenance patterns while minimizing user wait time during Git commands.
 204 A variety of configuration options are available to allow customizing this
 205 process. The default maintenance options focus on operations that complete
 206 quickly, even on large repositories.
 207
 208 Users may find some cases where scheduled maintenance tasks do not run as
 209 frequently as intended. Each `git maintenance run` command takes a lock on
 210 the repository's object database, and this prevents other concurrent
 211 `git maintenance run` commands from running on the same repository. Without
 212 this safeguard, competing processes could leave the repository in an
 213 unpredictable state.
 214
 215 The background maintenance schedule runs `git maintenance run` processes
 216 on an hourly basis. Each run executes the "hourly" tasks. At midnight,
 217 that process also executes the "daily" tasks. At midnight on the first day
 218 of the week, that process also executes the "weekly" tasks. A single
 219 process iterates over each registered repository, performing the scheduled
 220 tasks for that frequency. Depending on the number of registered
 221 repositories and their sizes, this process may take longer than an hour.
 222 In this case, multiple `git maintenance run` commands may run on the same
 223 repository at the same time, colliding on the object database lock. This
 224 results in one of the two tasks not running.
 225
 226 If you find that some maintenance windows are taking longer than one hour
 227 to complete, then consider reducing the complexity of your maintenance
 228 tasks. For example, the `gc` task is much slower than the
 229 `incremental-repack` task. However, this comes at a cost of a slightly
 230 larger object database. Consider moving more expensive tasks to be run
 231 less frequently.
 232
 233 Expert users may consider scheduling their own maintenance tasks using a
 234 different schedule than is available through `git maintenance start` and
 235 Git configuration options. These users should be aware of the object
 236 database lock and how concurrent `git maintenance run` commands behave.
 237 Further, the `git gc` command should not be combined with
 238 `git maintenance run` commands. `git gc` modifies the object database
 239 but does not take the lock in the same way as `git maintenance run`. If
 240 possible, use `git maintenance run --task=gc` instead of `git gc`.
 241
 242 The following sections describe the mechanisms put in place to run
 243 background maintenance by `git maintenance start` and how to customize
 244 them.
 245
 246 BACKGROUND MAINTENANCE ON POSIX SYSTEMS
 247 ---------------------------------------
 248
 249 The standard mechanism for scheduling background tasks on POSIX systems
 250 is cron(8). This tool executes commands based on a given schedule. The
 251 current list of user-scheduled tasks can be found by running `crontab -l`.
 252 The schedule written by `git maintenance start` is similar to this:
 253
 254 -----------------------------------------------------------------------
 255 # BEGIN GIT MAINTENANCE SCHEDULE
 256 # The following schedule was created by Git
 257 # Any edits made in this region might be
 258 # replaced in the future by a Git command.
 259
 260 0 1-23 * * * "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=hourly
 261 0 0 * * 1-6 "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=daily
 262 0 0 * * 0 "/<path>/git" --exec-path="/<path>" for-each-repo --config=maintenance.repo maintenance run --schedule=weekly
 263
 264 # END GIT MAINTENANCE SCHEDULE
 265 -----------------------------------------------------------------------
 266
 267 The comments are used as a region to mark the schedule as written by Git.
 268 Any modifications within this region will be completely deleted by
 269 `git maintenance stop` or overwritten by `git maintenance start`.
 270
 271 The `crontab` entry specifies the full path of the `git` executable to
 272 ensure that the executed `git` command is the same one with which
 273 `git maintenance start` was issued independent of `PATH`. If the same user
 274 runs `git maintenance start` with multiple Git executables, then only the
 275 latest executable is used.
 276
 277 These commands use `git for-each-repo --config=maintenance.repo` to run
 278 `git maintenance run --schedule=<frequency>` on each repository listed in
 279 the multi-valued `maintenance.repo` config option. These are typically
 280 loaded from the user-specific global config. The `git maintenance` process
 281 then determines which maintenance tasks are configured to run on each
 282 repository with each `<frequency>` using the `maintenance.<task>.schedule`
 283 config options. These values are loaded from the global or repository
 284 config values.
 285
 286 If the config values are insufficient to achieve your desired background
 287 maintenance schedule, then you can create your own schedule. If you run
 288 `crontab -e`, then an editor will load with your user-specific `cron`
 289 schedule. In that editor, you can add your own schedule lines. You could
 290 start by adapting the default schedule listed earlier, or you could read
 291 the crontab(5) documentation for advanced scheduling techniques. Please
 292 do use the full path and `--exec-path` techniques from the default
 293 schedule to ensure you are executing the correct binaries in your
 294 schedule.
 295
 296
 297 BACKGROUND MAINTENANCE ON LINUX SYSTEMD SYSTEMS
 298 -----------------------------------------------
 299
 300 While Linux supports `cron`, depending on the distribution, `cron` may
 301 be an optional package not necessarily installed. On modern Linux
 302 distributions, systemd timers are superseding it.
 303
 304 If user systemd timers are available, they will be used as a replacement
 305 of `cron`.
 306
 307 In this case, `git maintenance start` will create user systemd timer units
 308 and start the timers. The current list of user-scheduled tasks can be found
 309 by running `systemctl --user list-timers`. The timers written by `git
 310 maintenance start` are similar to this:
 311
 312 -----------------------------------------------------------------------
 313 $ systemctl --user list-timers
 314 NEXT                         LEFT          LAST                         PASSED     UNIT                         ACTIVATES
 315 Thu 2021-04-29 19:00:00 CEST 42min left    Thu 2021-04-29 18:00:11 CEST 17min ago  git-maintenance@hourly.timer git-maintenance@hourly.service
 316 Fri 2021-04-30 00:00:00 CEST 5h 42min left Thu 2021-04-29 00:00:11 CEST 18h ago    git-maintenance@daily.timer  git-maintenance@daily.service
 317 Mon 2021-05-03 00:00:00 CEST 3 days left   Mon 2021-04-26 00:00:11 CEST 3 days ago git-maintenance@weekly.timer git-maintenance@weekly.service
 318 -----------------------------------------------------------------------
 319
 320 One timer is registered for each `--schedule=<frequency>` option.
 321
 322 The definition of the systemd units can be inspected in the following files:
 323
 324 -----------------------------------------------------------------------
 325 ~/.config/systemd/user/git-maintenance@.timer
 326 ~/.config/systemd/user/git-maintenance@.service
 327 ~/.config/systemd/user/timers.target.wants/git-maintenance@hourly.timer
 328 ~/.config/systemd/user/timers.target.wants/git-maintenance@daily.timer
 329 ~/.config/systemd/user/timers.target.wants/git-maintenance@weekly.timer
 330 -----------------------------------------------------------------------
 331
 332 `git maintenance start` will overwrite these files and start the timer
 333 again with `systemctl --user`, so any customization should be done by
 334 creating a drop-in file, i.e. a `.conf` suffixed file in the
 335 `~/.config/systemd/user/git-maintenance@.service.d` directory.
 336
 337 `git maintenance stop` will stop the user systemd timers and delete
 338 the above mentioned files.
 339
 340 For more details, see `systemd.timer(5)`.
 341
 342
 343 BACKGROUND MAINTENANCE ON MACOS SYSTEMS
 344 ---------------------------------------
 345
 346 While macOS technically supports `cron`, using `crontab -e` requires
 347 elevated privileges and the executed process does not have a full user
 348 context. Without a full user context, Git and its credential helpers
 349 cannot access stored credentials, so some maintenance tasks are not
 350 functional.
 351
 352 Instead, `git maintenance start` interacts with the `launchctl` tool,
 353 which is the recommended way to schedule timed jobs in macOS. Scheduling
 354 maintenance through `git maintenance (start|stop)` requires some
 355 `launchctl` features available only in macOS 10.11 or later.
 356
 357 Your user-specific scheduled tasks are stored as XML-formatted `.plist`
 358 files in `~/Library/LaunchAgents/`. You can see the currently-registered
 359 tasks using the following command:
 360
 361 -----------------------------------------------------------------------
 362 $ ls ~/Library/LaunchAgents/org.git-scm.git*
 363 org.git-scm.git.daily.plist
 364 org.git-scm.git.hourly.plist
 365 org.git-scm.git.weekly.plist
 366 -----------------------------------------------------------------------
 367
 368 One task is registered for each `--schedule=<frequency>` option. To
 369 inspect how the XML format describes each schedule, open one of these
 370 `.plist` files in an editor and inspect the `<array>` element following
 371 the `<key>StartCalendarInterval</key>` element.
 372
 373 `git maintenance start` will overwrite these files and register the
 374 tasks again with `launchctl`, so any customizations should be done by
 375 creating your own `.plist` files with distinct names. Similarly, the
 376 `git maintenance stop` command will unregister the tasks with `launchctl`
 377 and delete the `.plist` files.
 378
 379 To create more advanced customizations to your background tasks, see
 380 launchctl.plist(5) for more information.
 381
 382
 383 BACKGROUND MAINTENANCE ON WINDOWS SYSTEMS
 384 -----------------------------------------
 385
 386 Windows does not support `cron` and instead has its own system for
 387 scheduling background tasks. The `git maintenance start` command uses
 388 the `schtasks` command to submit tasks to this system. You can inspect
 389 all background tasks using the Task Scheduler application. The tasks
 390 added by Git have names of the form `Git Maintenance (<frequency>)`.
 391 The Task Scheduler GUI has ways to inspect these tasks, but you can also
 392 export the tasks to XML files and view the details there.
 393
 394 Note that since Git is a console application, these background tasks
 395 create a console window visible to the current user. This can be changed
 396 manually by selecting the "Run whether user is logged in or not" option
 397 in Task Scheduler. This change requires a password input, which is why
 398 `git maintenance start` does not select it by default.
 399
 400 If you want to customize the background tasks, please rename the tasks
 401 so future calls to `git maintenance (start|stop)` do not overwrite your
 402 custom tasks.
 403
 404 CONFIGURATION
 405 -------------
 406
 407 include::includes/cmd-config-section-all.txt[]
 408
 409 include::config/maintenance.txt[]
 410
 411
 412 GIT
 413 ---
 414 Part of the linkgit:git[1] suite