A recent proposal by Marolf, dubbed boundary unitarity, suggests that the diffeomorphism invariance of general relativity comes close to explaining how holography arises in gravitational physics. Boundary unitarity makes use of the fact that the Hamiltonian of a diffeomorphism-invariant theory is a boundary term on-shell, thus the boundary algebra of observables evolves unitarily. We explore the consequences of boundary unitarity, and compare diffeomorphism invariance with other proposals in the literature about the mechanism of holography, such as quantum error correction.