android_system_core/fs_mgr/README.overlayfs.md

Android Overlayfs integration with adb remount
==============================================

Introduction
------------

Users working with userdebug or eng builds expect to be able to
remount the system partition as read-write and then add or modify
any number of files without reflashing the system image, which is
understandably efficient for a development cycle.
Limited memory systems that chose to use readonly filesystems like
*squashfs*, or *Logical Resizable Android Partitions* which land
system partition images right-sized, and with filesystem that have
been deduped on the block level to compress the content; means that
either a remount is not possible directly, or when done offers
little or no utility because of remaining space limitations or
support logistics.

*Overlayfs* comes to the rescue for these debug scenarios, and logic
will _automatically_ setup backing storage for a writable filesystem
as an upper reference, and mount overtop the lower.  These actions
will be performed in the **adb disable-verity** and **adb remount**
requests.

Operations
----------

### Cookbook

The typical action to utilize the remount facility is:

    $ adb root
    $ adb disable-verity
    $ adb reboot
    $ adb wait-for-device
    $ adb root
    $ adb remount

Followed by one of the following:

    $ adb stop
    $ adb sync
    $ adb start
    $ adb reboot

*or*

    $ adb push <source> <destination>
    $ adb reboot

Note that the sequence above:

    $ adb disable-verity
    $ adb reboot

*or*

    $ adb remount

can be replaced in both places with:

    $ adb remount -R

which will not reboot if everything is already prepared and ready
to go.

None of this changes if *overlayfs* needs to be engaged.
The decisions whether to use traditional direct filesystem remount,
or one wrapped by *overlayfs* is automatically determined based on
a probe of the filesystem types and space remaining.

### Backing Storage

When *overlayfs* logic is feasible, it will use either the
**/cache/overlay/** directory for non-A/B devices, or the
**/mnt/scratch/overlay** directory for A/B devices that have
access to *Logical Resizable Android Partitions*.
The backing store is used as soon as possible in the boot
process and can occur at first stage init, or at the
mount_all init rc commands.

This early as possible attachment of *overlayfs* means that
*sepolicy* or *init* itself can also be pushed and used after
the exec phases that accompany each stage.

Caveats
-------

- Space used in the backing storage is on a file by file basis
  and will require more space than if updated in place.  As such
  it is important to be mindful of any wasted space, for instance
  **BOARD_<partition>IMAGE_PARTITION_RESERVED_SIZE** being defined
  will have a negative impact on the overall right-sizing of images
  and thus free dynamic partition space.
- Kernel must have CONFIG_OVERLAY_FS=y and will need to be patched
  with "*overlayfs: override_creds=off option bypass creator_cred*"
  if kernel is 4.4 or higher.
  The patch is available on the upstream mailing list and the latest as of
  Feb 8 2019 is https://lore.kernel.org/patchwork/patch/1009299/.
  This patch adds an override_creds _mount_ option to overlayfs that
  permits legacy behavior for systems that do not have overlapping
  sepolicy rules, principals of least privilege, which is how Android behaves.
- *adb enable-verity* will free up overlayfs and as a bonus the
  device will be reverted pristine to before any content was updated.
  Update engine does not take advantage of this, will perform a full OTA.
- Update engine may not run if *fs_mgr_overlayfs_is_setup*() reports
  true as adb remount overrides are incompatible with an OTA resources.
- For implementation simplicity on retrofit dynamic partition devices,
  take the whole alternate super (eg: if "*a*" slot, then the whole of
  "*system_b*").
  Since landing a filesystem on the alternate super physical device
  without differentiating if it is setup to support logical or physical,
  the alternate slot metadata and previous content will be lost.
- If dynamic partitions runs out of space, resizing a logical
  partition larger may fail because of the scratch partition.
  If this happens, either fastboot flashall or adb enable-verity can
  be used to clear scratch storage to permit the flash.
  Then reinstate the overrides and continue.
- File bugs or submit fixes for review.
- There are other subtle caveats requiring complex logic to solve.
  Have evaluated them as too complex or not worth the trouble, please
  File a bug if a use case needs to be covered.
  - The backing storage is treated fragile, if anything else has
    issue with the space taken, the backing storage will be cleared
    out and we reserve the right to not inform, if the layering
    does not prevent any messaging.
  - Space remaining threshold is hard coded.  If 1% or more space
    still remains, overlayfs will not be used, yet that amount of
    space remaining is problematic.
  - Flashing a partition via bootloader fastboot, as opposed to user
    space fastbootd, is not detected, thus a partition may have
    override content remaining.  adb enable-verity to wipe.
  - Space is limited, there is near unlimited space on userdata,
    we have made an architectural decision to not utilize
    /data/overlay/ at this time.  Acquiring space to use for
    backing remains an ongoing battle.
  - First stage init, or ramdisk, can not be overriden.
  - Backing storage will be discarded or ignored on errors, leading
    to confusion.  When debugging using **adb remount** it is
    currently advised to confirm update is present after a reboot
    to develop confidence.