logo

oasis-root

Compiled tree of Oasis Linux based on own branch at <https://hacktivis.me/git/oasis/> git clone https://anongit.hacktivis.me/git/oasis-root.git

zpool-features.7 (34284B)


  1. .\"
  2. .\" Copyright (c) 2012, 2018 by Delphix. All rights reserved.
  3. .\" Copyright (c) 2013 by Saso Kiselkov. All rights reserved.
  4. .\" Copyright (c) 2014, Joyent, Inc. All rights reserved.
  5. .\" The contents of this file are subject to the terms of the Common Development
  6. .\" and Distribution License (the "License"). You may not use this file except
  7. .\" in compliance with the License. You can obtain a copy of the license at
  8. .\" usr/src/OPENSOLARIS.LICENSE or https://opensource.org/licenses/CDDL-1.0.
  9. .\"
  10. .\" See the License for the specific language governing permissions and
  11. .\" limitations under the License. When distributing Covered Code, include this
  12. .\" CDDL HEADER in each file and include the License file at
  13. .\" usr/src/OPENSOLARIS.LICENSE. If applicable, add the following below this
  14. .\" CDDL HEADER, with the fields enclosed by brackets "[]" replaced with your
  15. .\" own identifying information:
  16. .\" Portions Copyright [yyyy] [name of copyright owner]
  17. .\" Copyright (c) 2019, 2023, 2024, Klara, Inc.
  18. .\" Copyright (c) 2019, Allan Jude
  19. .\" Copyright (c) 2021, Colm Buckley <colm@tuatha.org>
  20. .\"
  21. .Dd October 2, 2024
  22. .Dt ZPOOL-FEATURES 7
  23. .Os
  24. .
  25. .Sh NAME
  26. .Nm zpool-features
  27. .Nd description of ZFS pool features
  28. .
  29. .Sh DESCRIPTION
  30. ZFS pool on-disk format versions are specified via
  31. .Dq features
  32. which replace the old on-disk format numbers
  33. .Pq the last supported on-disk format number is 28 .
  34. To enable a feature on a pool use the
  35. .Nm zpool Cm upgrade ,
  36. or set the
  37. .Sy feature Ns @ Ns Ar feature-name
  38. property to
  39. .Sy enabled .
  40. Please also see the
  41. .Sx Compatibility feature sets
  42. section for information on how sets of features may be enabled together.
  43. .Pp
  44. The pool format does not affect file system version compatibility or the ability
  45. to send file systems between pools.
  46. .Pp
  47. Since most features can be enabled independently of each other, the on-disk
  48. format of the pool is specified by the set of all features marked as
  49. .Sy active
  50. on the pool.
  51. If the pool was created by another software version
  52. this set may include unsupported features.
  53. .
  54. .Ss Identifying features
  55. Every feature has a GUID of the form
  56. .Ar com.example : Ns Ar feature-name .
  57. The reversed DNS name ensures that the feature's GUID is unique across all ZFS
  58. implementations.
  59. When unsupported features are encountered on a pool they will
  60. be identified by their GUIDs.
  61. Refer to the documentation for the ZFS
  62. implementation that created the pool for information about those features.
  63. .Pp
  64. Each supported feature also has a short name.
  65. By convention a feature's short name is the portion of its GUID which follows
  66. the
  67. .Sq \&:
  68. .Po
  69. i.e.
  70. .Ar com.example : Ns Ar feature-name
  71. would have the short name
  72. .Ar feature-name
  73. .Pc ,
  74. however a feature's short name may differ across ZFS implementations if
  75. following the convention would result in name conflicts.
  76. .
  77. .Ss Feature states
  78. Features can be in one of three states:
  79. .Bl -tag -width "disabled"
  80. .It Sy active
  81. This feature's on-disk format changes are in effect on the pool.
  82. Support for this feature is required to import the pool in read-write mode.
  83. If this feature is not read-only compatible,
  84. support is also required to import the pool in read-only mode
  85. .Pq see Sx Read-only compatibility .
  86. .It Sy enabled
  87. An administrator has marked this feature as enabled on the pool, but the
  88. feature's on-disk format changes have not been made yet.
  89. The pool can still be imported by software that does not support this feature,
  90. but changes may be made to the on-disk format at any time
  91. which will move the feature to the
  92. .Sy active
  93. state.
  94. Some features may support returning to the
  95. .Sy enabled
  96. state after becoming
  97. .Sy active .
  98. See feature-specific documentation for details.
  99. .It Sy disabled
  100. This feature's on-disk format changes have not been made and will not be made
  101. unless an administrator moves the feature to the
  102. .Sy enabled
  103. state.
  104. Features cannot be disabled once they have been enabled.
  105. .El
  106. .Pp
  107. The state of supported features is exposed through pool properties of the form
  108. .Sy feature Ns @ Ns Ar short-name .
  109. .
  110. .Ss Read-only compatibility
  111. Some features may make on-disk format changes that do not interfere with other
  112. software's ability to read from the pool.
  113. These features are referred to as
  114. .Dq read-only compatible .
  115. If all unsupported features on a pool are read-only compatible,
  116. the pool can be imported in read-only mode by setting the
  117. .Sy readonly
  118. property during import
  119. .Po see
  120. .Xr zpool-import 8
  121. for details on importing pools
  122. .Pc .
  123. .
  124. .Ss Unsupported features
  125. For each unsupported feature enabled on an imported pool, a pool property
  126. named
  127. .Sy unsupported Ns @ Ns Ar feature-name
  128. will indicate why the import was allowed despite the unsupported feature.
  129. Possible values for this property are:
  130. .Bl -tag -width "readonly"
  131. .It Sy inactive
  132. The feature is in the
  133. .Sy enabled
  134. state and therefore the pool's on-disk
  135. format is still compatible with software that does not support this feature.
  136. .It Sy readonly
  137. The feature is read-only compatible and the pool has been imported in
  138. read-only mode.
  139. .El
  140. .
  141. .Ss Feature dependencies
  142. Some features depend on other features being enabled in order to function.
  143. Enabling a feature will automatically enable any features it depends on.
  144. .
  145. .Ss Compatibility feature sets
  146. It is sometimes necessary for a pool to maintain compatibility with a
  147. specific on-disk format, by enabling and disabling particular features.
  148. The
  149. .Sy compatibility
  150. feature facilitates this by allowing feature sets to be read from text files.
  151. When set to
  152. .Sy off
  153. .Pq the default ,
  154. compatibility feature sets are disabled
  155. .Pq i.e. all features are enabled ;
  156. when set to
  157. .Sy legacy ,
  158. no features are enabled.
  159. When set to a comma-separated list of filenames
  160. .Po
  161. each filename may either be an absolute path, or relative to
  162. .Pa /etc/zfs/compatibility.d
  163. or
  164. .Pa /usr/share/zfs/compatibility.d
  165. .Pc ,
  166. the lists of requested features are read from those files,
  167. separated by whitespace and/or commas.
  168. Only features present in all files are enabled.
  169. .Pp
  170. Simple sanity checks are applied to the files:
  171. they must be between 1 B and 16 KiB in size, and must end with a newline
  172. character.
  173. .Pp
  174. The requested features are applied when a pool is created using
  175. .Nm zpool Cm create Fl o Sy compatibility Ns = Ns Ar …
  176. and controls which features are enabled when using
  177. .Nm zpool Cm upgrade .
  178. .Nm zpool Cm status
  179. will not show a warning about disabled features which are not part
  180. of the requested feature set.
  181. .Pp
  182. The special value
  183. .Sy legacy
  184. prevents any features from being enabled, either via
  185. .Nm zpool Cm upgrade
  186. or
  187. .Nm zpool Cm set Sy feature Ns @ Ns Ar feature-name Ns = Ns Sy enabled .
  188. This setting also prevents pools from being upgraded to newer on-disk versions.
  189. This is a safety measure to prevent new features from being
  190. accidentally enabled, breaking compatibility.
  191. .Pp
  192. By convention, compatibility files in
  193. .Pa /usr/share/zfs/compatibility.d
  194. are provided by the distribution, and include feature sets
  195. supported by important versions of popular distributions, and feature
  196. sets commonly supported at the start of each year.
  197. Compatibility files in
  198. .Pa /etc/zfs/compatibility.d ,
  199. if present, will take precedence over files with the same name in
  200. .Pa /usr/share/zfs/compatibility.d .
  201. .Pp
  202. If an unrecognized feature is found in these files, an error message will
  203. be shown.
  204. If the unrecognized feature is in a file in
  205. .Pa /etc/zfs/compatibility.d ,
  206. this is treated as an error and processing will stop.
  207. If the unrecognized feature is under
  208. .Pa /usr/share/zfs/compatibility.d ,
  209. this is treated as a warning and processing will continue.
  210. This difference is to allow distributions to include features
  211. which might not be recognized by the currently-installed binaries.
  212. .Pp
  213. Compatibility files may include comments:
  214. any text from
  215. .Sq #
  216. to the end of the line is ignored.
  217. .Pp
  218. .Sy Example :
  219. .Bd -literal -compact -offset 4n
  220. .No example# Nm cat Pa /usr/share/zfs/compatibility.d/grub2
  221. # Features which are supported by GRUB2 versions from v2.12 onwards.
  222. allocation_classes
  223. async_destroy
  224. block_cloning
  225. bookmarks
  226. device_rebuild
  227. embedded_data
  228. empty_bpobj
  229. enabled_txg
  230. extensible_dataset
  231. filesystem_limits
  232. hole_birth
  233. large_blocks
  234. livelist
  235. log_spacemap
  236. lz4_compress
  237. project_quota
  238. resilver_defer
  239. spacemap_histogram
  240. spacemap_v2
  241. userobj_accounting
  242. zilsaxattr
  243. zpool_checkpoint
  244. .No example# Nm cat Pa /usr/share/zfs/compatibility.d/grub2-2.06
  245. # Features which are supported by GRUB2 versions prior to v2.12.
  246. #
  247. # GRUB is not able to detect ZFS pool if snaphsot of top level boot pool
  248. # is created. This issue is observed with GRUB versions before v2.12 if
  249. # extensible_dataset feature is enabled on ZFS boot pool.
  250. #
  251. # This file lists all read-only comaptible features except
  252. # extensible_dataset and any other feature that depends on it.
  253. #
  254. allocation_classes
  255. async_destroy
  256. block_cloning
  257. device_rebuild
  258. embedded_data
  259. empty_bpobj
  260. enabled_txg
  261. hole_birth
  262. log_spacemap
  263. lz4_compress
  264. resilver_defer
  265. spacemap_histogram
  266. spacemap_v2
  267. zpool_checkpoint
  268. .No example# Nm zpool Cm create Fl o Sy compatibility Ns = Ns Ar grub2 Ar bootpool Ar vdev
  269. .Ed
  270. .Pp
  271. See
  272. .Xr zpool-create 8
  273. and
  274. .Xr zpool-upgrade 8
  275. for more information on how these commands are affected by feature sets.
  276. .
  277. .de feature
  278. .It Sy \\$2
  279. .Bl -tag -compact -width "READ-ONLY COMPATIBLE"
  280. .It GUID
  281. .Sy \\$1:\\$2
  282. .if !"\\$4"" \{\
  283. .It DEPENDENCIES
  284. \fB\\$4\fP\c
  285. .if !"\\$5"" , \fB\\$5\fP\c
  286. .if !"\\$6"" , \fB\\$6\fP\c
  287. .if !"\\$7"" , \fB\\$7\fP\c
  288. .if !"\\$8"" , \fB\\$8\fP\c
  289. .if !"\\$9"" , \fB\\$9\fP\c
  290. .\}
  291. .It READ-ONLY COMPATIBLE
  292. \\$3
  293. .El
  294. .Pp
  295. ..
  296. .
  297. .ds instant-never \
  298. .No This feature becomes Sy active No as soon as it is enabled \
  299. and will never return to being Sy enabled .
  300. .
  301. .ds remount-upgrade \
  302. .No Each filesystem will be upgraded automatically when remounted, \
  303. or when a new file is created under that filesystem. \
  304. The upgrade can also be triggered on filesystems via \
  305. Nm zfs Cm set Sy version Ns = Ns Sy current Ar fs . \
  306. No The upgrade process runs in the background and may take a while to complete \
  307. for filesystems containing large amounts of files .
  308. .
  309. .de checksum-spiel
  310. When the
  311. .Sy \\$1
  312. feature is set to
  313. .Sy enabled ,
  314. the administrator can turn on the
  315. .Sy \\$1
  316. checksum on any dataset using
  317. .Nm zfs Cm set Sy checksum Ns = Ns Sy \\$1 Ar dset
  318. .Po see Xr zfs-set 8 Pc .
  319. This feature becomes
  320. .Sy active
  321. once a
  322. .Sy checksum
  323. property has been set to
  324. .Sy \\$1 ,
  325. and will return to being
  326. .Sy enabled
  327. once all filesystems that have ever had their checksum set to
  328. .Sy \\$1
  329. are destroyed.
  330. ..
  331. .
  332. .Sh FEATURES
  333. The following features are supported on this system:
  334. .Bl -tag -width Ds
  335. .feature org.zfsonlinux allocation_classes yes
  336. This feature enables support for separate allocation classes.
  337. .Pp
  338. This feature becomes
  339. .Sy active
  340. when a dedicated allocation class vdev
  341. .Pq dedup or special
  342. is created with the
  343. .Nm zpool Cm create No or Nm zpool Cm add No commands .
  344. With device removal, it can be returned to the
  345. .Sy enabled
  346. state if all the dedicated allocation class vdevs are removed.
  347. .
  348. .feature com.delphix async_destroy yes
  349. Destroying a file system requires traversing all of its data in order to
  350. return its used space to the pool.
  351. Without
  352. .Sy async_destroy ,
  353. the file system is not fully removed until all space has been reclaimed.
  354. If the destroy operation is interrupted by a reboot or power outage,
  355. the next attempt to open the pool will need to complete the destroy
  356. operation synchronously.
  357. .Pp
  358. When
  359. .Sy async_destroy
  360. is enabled, the file system's data will be reclaimed by a background process,
  361. allowing the destroy operation to complete
  362. without traversing the entire file system.
  363. The background process is able to resume
  364. interrupted destroys after the pool has been opened, eliminating the need
  365. to finish interrupted destroys as part of the open operation.
  366. The amount of space remaining to be reclaimed by the background process
  367. is available through the
  368. .Sy freeing
  369. property.
  370. .Pp
  371. This feature is only
  372. .Sy active
  373. while
  374. .Sy freeing
  375. is non-zero.
  376. .
  377. .feature org.openzfs blake3 no extensible_dataset
  378. This feature enables the use of the BLAKE3 hash algorithm for checksum and
  379. dedup.
  380. BLAKE3 is a secure hash algorithm focused on high performance.
  381. .Pp
  382. .checksum-spiel blake3
  383. .
  384. .feature com.fudosecurity block_cloning yes
  385. When this feature is enabled ZFS will use block cloning for operations like
  386. .Fn copy_file_range 2 .
  387. Block cloning allows to create multiple references to a single block.
  388. It is much faster than copying the data (as the actual data is neither read nor
  389. written) and takes no additional space.
  390. Blocks can be cloned across datasets under some conditions (like equal
  391. .Nm recordsize ,
  392. the same master encryption key, etc.).
  393. ZFS tries its best to clone across datasets including encrypted ones.
  394. This is limited for various (nontrivial) reasons depending on the OS
  395. and/or ZFS internals.
  396. .Pp
  397. This feature becomes
  398. .Sy active
  399. when first block is cloned.
  400. When the last cloned block is freed, it goes back to the enabled state.
  401. .feature com.delphix bookmarks yes extensible_dataset
  402. This feature enables use of the
  403. .Nm zfs Cm bookmark
  404. command.
  405. .Pp
  406. This feature is
  407. .Sy active
  408. while any bookmarks exist in the pool.
  409. All bookmarks in the pool can be listed by running
  410. .Nm zfs Cm list Fl t Sy bookmark Fl r Ar poolname .
  411. .
  412. .feature com.datto bookmark_v2 no bookmark extensible_dataset
  413. This feature enables the creation and management of larger bookmarks which are
  414. needed for other features in ZFS.
  415. .Pp
  416. This feature becomes
  417. .Sy active
  418. when a v2 bookmark is created and will be returned to the
  419. .Sy enabled
  420. state when all v2 bookmarks are destroyed.
  421. .
  422. .feature com.delphix bookmark_written no bookmark extensible_dataset bookmark_v2
  423. This feature enables additional bookmark accounting fields, enabling the
  424. .Sy written Ns # Ns Ar bookmark
  425. property
  426. .Pq space written since a bookmark
  427. and estimates of send stream sizes for incrementals from bookmarks.
  428. .Pp
  429. This feature becomes
  430. .Sy active
  431. when a bookmark is created and will be
  432. returned to the
  433. .Sy enabled
  434. state when all bookmarks with these fields are destroyed.
  435. .
  436. .feature org.openzfs device_rebuild yes
  437. This feature enables the ability for the
  438. .Nm zpool Cm attach
  439. and
  440. .Nm zpool Cm replace
  441. commands to perform sequential reconstruction
  442. .Pq instead of healing reconstruction
  443. when resilvering.
  444. .Pp
  445. Sequential reconstruction resilvers a device in LBA order without immediately
  446. verifying the checksums.
  447. Once complete, a scrub is started, which then verifies the checksums.
  448. This approach allows full redundancy to be restored to the pool
  449. in the minimum amount of time.
  450. This two-phase approach will take longer than a healing resilver
  451. when the time to verify the checksums is included.
  452. However, unless there is additional pool damage,
  453. no checksum errors should be reported by the scrub.
  454. This feature is incompatible with raidz configurations.
  455. .
  456. This feature becomes
  457. .Sy active
  458. while a sequential resilver is in progress, and returns to
  459. .Sy enabled
  460. when the resilver completes.
  461. .
  462. .feature com.delphix device_removal no
  463. This feature enables the
  464. .Nm zpool Cm remove
  465. command to remove top-level vdevs,
  466. evacuating them to reduce the total size of the pool.
  467. .Pp
  468. This feature becomes
  469. .Sy active
  470. when the
  471. .Nm zpool Cm remove
  472. command is used
  473. on a top-level vdev, and will never return to being
  474. .Sy enabled .
  475. .
  476. .feature org.openzfs draid no
  477. This feature enables use of the
  478. .Sy draid
  479. vdev type.
  480. dRAID is a variant of RAID-Z which provides integrated distributed
  481. hot spares that allow faster resilvering while retaining the benefits of RAID-Z.
  482. Data, parity, and spare space are organized in redundancy groups
  483. and distributed evenly over all of the devices.
  484. .Pp
  485. This feature becomes
  486. .Sy active
  487. when creating a pool which uses the
  488. .Sy draid
  489. vdev type, or when adding a new
  490. .Sy draid
  491. vdev to an existing pool.
  492. .
  493. .feature org.illumos edonr no extensible_dataset
  494. This feature enables the use of the Edon-R hash algorithm for checksum,
  495. including for nopwrite
  496. .Po if compression is also enabled, an overwrite of
  497. a block whose checksum matches the data being written will be ignored
  498. .Pc .
  499. In an abundance of caution, Edon-R requires verification when used with
  500. dedup:
  501. .Nm zfs Cm set Sy dedup Ns = Ns Sy edonr , Ns Sy verify
  502. .Po see Xr zfs-set 8 Pc .
  503. .Pp
  504. Edon-R is a very high-performance hash algorithm that was part
  505. of the NIST SHA-3 competition.
  506. It provides extremely high hash performance
  507. .Pq over 350% faster than SHA-256 ,
  508. but was not selected because of its unsuitability
  509. as a general purpose secure hash algorithm.
  510. This implementation utilizes the new salted checksumming functionality
  511. in ZFS, which means that the checksum is pre-seeded with a secret
  512. 256-bit random key
  513. .Pq stored on the pool
  514. before being fed the data block to be checksummed.
  515. Thus the produced checksums are unique to a given pool,
  516. preventing hash collision attacks on systems with dedup.
  517. .Pp
  518. .checksum-spiel edonr
  519. .
  520. .feature com.delphix embedded_data no
  521. This feature improves the performance and compression ratio of
  522. highly-compressible blocks.
  523. Blocks whose contents can compress to 112 bytes
  524. or smaller can take advantage of this feature.
  525. .Pp
  526. When this feature is enabled, the contents of highly-compressible blocks are
  527. stored in the block
  528. .Dq pointer
  529. itself
  530. .Po a misnomer in this case, as it contains
  531. the compressed data, rather than a pointer to its location on disk
  532. .Pc .
  533. Thus the space of the block
  534. .Pq one sector, typically 512 B or 4 KiB
  535. is saved, and no additional I/O is needed to read and write the data block.
  536. .
  537. \*[instant-never]
  538. .
  539. .feature com.delphix empty_bpobj yes
  540. This feature increases the performance of creating and using a large
  541. number of snapshots of a single filesystem or volume, and also reduces
  542. the disk space required.
  543. .Pp
  544. When there are many snapshots, each snapshot uses many Block Pointer
  545. Objects
  546. .Pq bpobjs
  547. to track blocks associated with that snapshot.
  548. However, in common use cases, most of these bpobjs are empty.
  549. This feature allows us to create each bpobj on-demand,
  550. thus eliminating the empty bpobjs.
  551. .Pp
  552. This feature is
  553. .Sy active
  554. while there are any filesystems, volumes,
  555. or snapshots which were created after enabling this feature.
  556. .
  557. .feature com.delphix enabled_txg yes
  558. Once this feature is enabled, ZFS records the transaction group number
  559. in which new features are enabled.
  560. This has no user-visible impact, but other features may depend on this feature.
  561. .Pp
  562. This feature becomes
  563. .Sy active
  564. as soon as it is enabled and will never return to being
  565. .Sy enabled .
  566. .
  567. .feature com.datto encryption no bookmark_v2 extensible_dataset
  568. This feature enables the creation and management of natively encrypted datasets.
  569. .Pp
  570. This feature becomes
  571. .Sy active
  572. when an encrypted dataset is created and will be returned to the
  573. .Sy enabled
  574. state when all datasets that use this feature are destroyed.
  575. .
  576. .feature com.klarasystems fast_dedup yes
  577. This feature allows more advanced deduplication features to be enabled on new
  578. dedup tables.
  579. .Pp
  580. This feature will be
  581. .Sy active
  582. when the first deduplicated block is written after a new dedup table is created
  583. (ie after a new pool creation, or new checksum used on a dataset with
  584. .Sy dedup
  585. enabled).
  586. It will be returned to the
  587. .Sy enabled
  588. state when all deduplicated blocks using it are freed.
  589. .
  590. .feature com.delphix extensible_dataset no
  591. This feature allows more flexible use of internal ZFS data structures,
  592. and exists for other features to depend on.
  593. .Pp
  594. This feature will be
  595. .Sy active
  596. when the first dependent feature uses it, and will be returned to the
  597. .Sy enabled
  598. state when all datasets that use this feature are destroyed.
  599. .
  600. .feature com.joyent filesystem_limits yes extensible_dataset
  601. This feature enables filesystem and snapshot limits.
  602. These limits can be used to control how many filesystems and/or snapshots
  603. can be created at the point in the tree on which the limits are set.
  604. .Pp
  605. This feature is
  606. .Sy active
  607. once either of the limit properties has been set on a dataset
  608. and will never return to being
  609. .Sy enabled .
  610. .
  611. .feature com.delphix head_errlog no
  612. This feature enables the upgraded version of errlog, which required an on-disk
  613. error log format change.
  614. Now the error log of each head dataset is stored separately in the zap object
  615. and keyed by the head id.
  616. With this feature enabled, every dataset affected by an error block is listed
  617. in the output of
  618. .Nm zpool Cm status .
  619. In case of encrypted filesystems with unloaded keys we are unable to check
  620. their snapshots or clones for errors and these will not be reported.
  621. An "access denied" error will be reported.
  622. .Pp
  623. \*[instant-never]
  624. .
  625. .feature com.delphix hole_birth no enabled_txg
  626. This feature has/had bugs, the result of which is that, if you do a
  627. .Nm zfs Cm send Fl i
  628. .Pq or Fl R , No since it uses Fl i
  629. from an affected dataset, the receiving party will not see any checksum
  630. or other errors, but the resulting destination snapshot
  631. will not match the source.
  632. Its use by
  633. .Nm zfs Cm send Fl i
  634. has been disabled by default
  635. .Po
  636. see
  637. .Sy send_holes_without_birth_time
  638. in
  639. .Xr zfs 4
  640. .Pc .
  641. .Pp
  642. This feature improves performance of incremental sends
  643. .Pq Nm zfs Cm send Fl i
  644. and receives for objects with many holes.
  645. The most common case of hole-filled objects is zvols.
  646. .Pp
  647. An incremental send stream from snapshot
  648. .Sy A No to snapshot Sy B
  649. contains information about every block that changed between
  650. .Sy A No and Sy B .
  651. Blocks which did not change between those snapshots can be
  652. identified and omitted from the stream using a piece of metadata called
  653. the
  654. .Dq block birth time ,
  655. but birth times are not recorded for holes
  656. .Pq blocks filled only with zeroes .
  657. Since holes created after
  658. .Sy A No cannot be distinguished from holes created before Sy A ,
  659. information about every hole in the entire filesystem or zvol
  660. is included in the send stream.
  661. .Pp
  662. For workloads where holes are rare this is not a problem.
  663. However, when incrementally replicating filesystems or zvols with many holes
  664. .Pq for example a zvol formatted with another filesystem
  665. a lot of time will be spent sending and receiving unnecessary information
  666. about holes that already exist on the receiving side.
  667. .Pp
  668. Once the
  669. .Sy hole_birth
  670. feature has been enabled the block birth times
  671. of all new holes will be recorded.
  672. Incremental sends between snapshots created after this feature is enabled
  673. will use this new metadata to avoid sending information about holes that
  674. already exist on the receiving side.
  675. .Pp
  676. \*[instant-never]
  677. .
  678. .feature org.open-zfs large_blocks no extensible_dataset
  679. This feature allows the record size on a dataset to be set larger than 128 KiB.
  680. .Pp
  681. This feature becomes
  682. .Sy active
  683. once a dataset contains a file with a block size larger than 128 KiB,
  684. and will return to being
  685. .Sy enabled
  686. once all filesystems that have ever had their recordsize larger than 128 KiB
  687. are destroyed.
  688. .
  689. .feature org.zfsonlinux large_dnode no extensible_dataset
  690. This feature allows the size of dnodes in a dataset to be set larger than 512 B.
  691. .
  692. This feature becomes
  693. .Sy active
  694. once a dataset contains an object with a dnode larger than 512 B,
  695. which occurs as a result of setting the
  696. .Sy dnodesize
  697. dataset property to a value other than
  698. .Sy legacy .
  699. The feature will return to being
  700. .Sy enabled
  701. once all filesystems that have ever contained a dnode larger than 512 B
  702. are destroyed.
  703. Large dnodes allow more data to be stored in the bonus buffer,
  704. thus potentially improving performance by avoiding the use of spill blocks.
  705. .
  706. .feature com.klarasystems large_microzap yes extensible_dataset large_blocks
  707. This feature allows "micro" ZAPs to grow larger than 128 KiB without being
  708. upgraded to "fat" ZAPs.
  709. .Pp
  710. This feature becomes
  711. .Sy active
  712. the first time a micro ZAP grows larger than 128KiB.
  713. It will only be returned to the
  714. .Sy enabled
  715. state when all datasets that ever had a large micro ZAP are destroyed.
  716. .Pp
  717. Note that even when this feature is enabled, micro ZAPs cannot grow larger
  718. than 128 KiB without also changing the
  719. .Sy zap_micro_max_size
  720. module parameter.
  721. See
  722. .Xr zfs 4 .
  723. .
  724. .feature com.delphix livelist yes extensible_dataset
  725. This feature allows clones to be deleted faster than the traditional method
  726. when a large number of random/sparse writes have been made to the clone.
  727. All blocks allocated and freed after a clone is created are tracked by the
  728. the clone's livelist which is referenced during the deletion of the clone.
  729. The feature is activated when a clone is created and remains
  730. .Sy active
  731. until all clones have been destroyed.
  732. .
  733. .feature com.delphix log_spacemap yes com.delphix:spacemap_v2
  734. This feature improves performance for heavily-fragmented pools,
  735. especially when workloads are heavy in random-writes.
  736. It does so by logging all the metaslab changes on a single spacemap every TXG
  737. instead of scattering multiple writes to all the metaslab spacemaps.
  738. .Pp
  739. \*[instant-never]
  740. .
  741. .feature org.zfsonlinux longname no extensible_dataset
  742. This feature allows creating files and directories with name up to 1023 bytes
  743. in length.
  744. A new dataset property
  745. .Sy longname
  746. is also introduced to toggle longname support for each dataset individually.
  747. This property can be disabled even if it contains longname files.
  748. In such case, new file cannot be created with longname but existing longname
  749. files can still be looked up.
  750. .Pp
  751. This feature becomes
  752. .Sy active
  753. when a file name greater than 255 is created in a dataset, and returns to
  754. being
  755. .Sy enabled
  756. when all such datasets are destroyed.
  757. .
  758. .feature org.illumos lz4_compress no
  759. .Sy lz4
  760. is a high-performance real-time compression algorithm that
  761. features significantly faster compression and decompression as well as a
  762. higher compression ratio than the older
  763. .Sy lzjb
  764. compression.
  765. Typically,
  766. .Sy lz4
  767. compression is approximately 50% faster on compressible data and 200% faster
  768. on incompressible data than
  769. .Sy lzjb .
  770. It is also approximately 80% faster on decompression,
  771. while giving approximately a 10% better compression ratio.
  772. .Pp
  773. When the
  774. .Sy lz4_compress
  775. feature is set to
  776. .Sy enabled ,
  777. the administrator can turn on
  778. .Sy lz4
  779. compression on any dataset on the pool using the
  780. .Xr zfs-set 8
  781. command.
  782. All newly written metadata will be compressed with the
  783. .Sy lz4
  784. algorithm.
  785. .Pp
  786. \*[instant-never]
  787. .
  788. .feature com.joyent multi_vdev_crash_dump no
  789. This feature allows a dump device to be configured with a pool comprised
  790. of multiple vdevs.
  791. Those vdevs may be arranged in any mirrored or raidz configuration.
  792. .Pp
  793. When the
  794. .Sy multi_vdev_crash_dump
  795. feature is set to
  796. .Sy enabled ,
  797. the administrator can use
  798. .Xr dumpadm 8
  799. to configure a dump device on a pool comprised of multiple vdevs.
  800. .Pp
  801. Under
  802. .Fx
  803. and Linux this feature is unused, but registered for compatibility.
  804. New pools created on these systems will have the feature
  805. .Sy enabled
  806. but will never transition to
  807. .Sy active ,
  808. as this functionality is not required for crash dump support.
  809. Existing pools where this feature is
  810. .Sy active
  811. can be imported.
  812. .
  813. .feature com.delphix obsolete_counts yes device_removal
  814. This feature is an enhancement of
  815. .Sy device_removal ,
  816. which will over time reduce the memory used to track removed devices.
  817. When indirect blocks are freed or remapped,
  818. we note that their part of the indirect mapping is
  819. .Dq obsolete
  820. – no longer needed.
  821. .Pp
  822. This feature becomes
  823. .Sy active
  824. when the
  825. .Nm zpool Cm remove
  826. command is used on a top-level vdev, and will never return to being
  827. .Sy enabled .
  828. .
  829. .feature org.zfsonlinux project_quota yes extensible_dataset
  830. This feature allows administrators to account the spaces and objects usage
  831. information against the project identifier
  832. .Pq ID .
  833. .Pp
  834. The project ID is an object-based attribute.
  835. When upgrading an existing filesystem,
  836. objects without a project ID will be assigned a zero project ID.
  837. When this feature is enabled, newly created objects inherit
  838. their parent directories' project ID if the parent's inherit flag is set
  839. .Pq via Nm chattr Sy [+-]P No or Nm zfs Cm project Fl s Ns | Ns Fl C .
  840. Otherwise, the new object's project ID will be zero.
  841. An object's project ID can be changed at any time by the owner
  842. .Pq or privileged user
  843. via
  844. .Nm chattr Fl p Ar prjid
  845. or
  846. .Nm zfs Cm project Fl p Ar prjid .
  847. .Pp
  848. This feature will become
  849. .Sy active
  850. as soon as it is enabled and will never return to being
  851. .Sy disabled .
  852. \*[remount-upgrade]
  853. .
  854. .feature org.openzfs raidz_expansion no none
  855. This feature enables the
  856. .Nm zpool Cm attach
  857. subcommand to attach a new device to a RAID-Z group, expanding the total
  858. amount usable space in the pool.
  859. See
  860. .Xr zpool-attach 8 .
  861. .
  862. .feature com.delphix redaction_bookmarks no bookmarks extensible_dataset
  863. This feature enables the use of redacted
  864. .Nm zfs Cm send Ns s ,
  865. which create redaction bookmarks storing the list of blocks
  866. redacted by the send that created them.
  867. For more information about redacted sends, see
  868. .Xr zfs-send 8 .
  869. .
  870. .feature com.delphix redacted_datasets no extensible_dataset
  871. This feature enables the receiving of redacted
  872. .Nm zfs Cm send
  873. streams, which create redacted datasets when received.
  874. These datasets are missing some of their blocks,
  875. and so cannot be safely mounted, and their contents cannot be safely read.
  876. For more information about redacted receives, see
  877. .Xr zfs-send 8 .
  878. .
  879. .feature com.delphix redaction_list_spill no redaction_bookmarks
  880. This feature enables the redaction list created by zfs redact to store
  881. many more entries.
  882. It becomes
  883. .Sy active
  884. when a redaction list is created with more than 36 entries,
  885. and returns to being
  886. .Sy enabled
  887. when no long redaction lists remain in the pool.
  888. For more information about redacted sends, see
  889. .Xr zfs-send 8 .
  890. .
  891. .feature com.datto resilver_defer yes
  892. This feature allows ZFS to postpone new resilvers if an existing one is already
  893. in progress.
  894. Without this feature, any new resilvers will cause the currently
  895. running one to be immediately restarted from the beginning.
  896. .Pp
  897. This feature becomes
  898. .Sy active
  899. once a resilver has been deferred, and returns to being
  900. .Sy enabled
  901. when the deferred resilver begins.
  902. .
  903. .feature org.illumos sha512 no extensible_dataset
  904. This feature enables the use of the SHA-512/256 truncated hash algorithm
  905. .Pq FIPS 180-4
  906. for checksum and dedup.
  907. The native 64-bit arithmetic of SHA-512 provides an approximate 50%
  908. performance boost over SHA-256 on 64-bit hardware
  909. and is thus a good minimum-change replacement candidate
  910. for systems where hash performance is important,
  911. but these systems cannot for whatever reason utilize the faster
  912. .Sy skein No and Sy edonr
  913. algorithms.
  914. .Pp
  915. .checksum-spiel sha512
  916. .
  917. .feature org.illumos skein no extensible_dataset
  918. This feature enables the use of the Skein hash algorithm for checksum and dedup.
  919. Skein is a high-performance secure hash algorithm that was a
  920. finalist in the NIST SHA-3 competition.
  921. It provides a very high security margin and high performance on 64-bit hardware
  922. .Pq 80% faster than SHA-256 .
  923. This implementation also utilizes the new salted checksumming
  924. functionality in ZFS, which means that the checksum is pre-seeded with a
  925. secret 256-bit random key
  926. .Pq stored on the pool
  927. before being fed the data block to be checksummed.
  928. Thus the produced checksums are unique to a given pool,
  929. preventing hash collision attacks on systems with dedup.
  930. .Pp
  931. .checksum-spiel skein
  932. .
  933. .feature com.delphix spacemap_histogram yes
  934. This features allows ZFS to maintain more information about how free space
  935. is organized within the pool.
  936. If this feature is
  937. .Sy enabled ,
  938. it will be activated when a new space map object is created, or
  939. an existing space map is upgraded to the new format,
  940. and never returns back to being
  941. .Sy enabled .
  942. .
  943. .feature com.delphix spacemap_v2 yes
  944. This feature enables the use of the new space map encoding which
  945. consists of two words
  946. .Pq instead of one
  947. whenever it is advantageous.
  948. The new encoding allows space maps to represent large regions of
  949. space more efficiently on-disk while also increasing their maximum
  950. addressable offset.
  951. .Pp
  952. This feature becomes
  953. .Sy active
  954. once it is
  955. .Sy enabled ,
  956. and never returns back to being
  957. .Sy enabled .
  958. .
  959. .feature org.zfsonlinux userobj_accounting yes extensible_dataset
  960. This feature allows administrators to account the object usage information
  961. by user and group.
  962. .Pp
  963. \*[instant-never]
  964. \*[remount-upgrade]
  965. .
  966. .feature com.klarasystems vdev_zaps_v2 no
  967. This feature creates a ZAP object for the root vdev.
  968. .Pp
  969. This feature becomes active after the next
  970. .Nm zpool Cm import
  971. or
  972. .Nm zpool reguid .
  973. .
  974. Properties can be retrieved or set on the root vdev using
  975. .Nm zpool Cm get
  976. and
  977. .Nm zpool Cm set
  978. with
  979. .Sy root
  980. as the vdev name which is an alias for
  981. .Sy root-0 .
  982. .feature org.openzfs zilsaxattr yes extensible_dataset
  983. This feature enables
  984. .Sy xattr Ns = Ns Sy sa
  985. extended attribute logging in the ZIL.
  986. If enabled, extended attribute changes
  987. .Pq both Sy xattrdir Ns = Ns Sy dir No and Sy xattr Ns = Ns Sy sa
  988. are guaranteed to be durable if either the dataset had
  989. .Sy sync Ns = Ns Sy always
  990. set at the time the changes were made, or
  991. .Xr sync 2
  992. is called on the dataset after the changes were made.
  993. .Pp
  994. This feature becomes
  995. .Sy active
  996. when a ZIL is created for at least one dataset and will be returned to the
  997. .Sy enabled
  998. state when it is destroyed for all datasets that use this feature.
  999. .
  1000. .feature com.delphix zpool_checkpoint yes
  1001. This feature enables the
  1002. .Nm zpool Cm checkpoint
  1003. command that can checkpoint the state of the pool
  1004. at the time it was issued and later rewind back to it or discard it.
  1005. .Pp
  1006. This feature becomes
  1007. .Sy active
  1008. when the
  1009. .Nm zpool Cm checkpoint
  1010. command is used to checkpoint the pool.
  1011. The feature will only return back to being
  1012. .Sy enabled
  1013. when the pool is rewound or the checkpoint has been discarded.
  1014. .
  1015. .feature org.freebsd zstd_compress no extensible_dataset
  1016. .Sy zstd
  1017. is a high-performance compression algorithm that features a
  1018. combination of high compression ratios and high speed.
  1019. Compared to
  1020. .Sy gzip ,
  1021. .Sy zstd
  1022. offers slightly better compression at much higher speeds.
  1023. Compared to
  1024. .Sy lz4 ,
  1025. .Sy zstd
  1026. offers much better compression while being only modestly slower.
  1027. Typically,
  1028. .Sy zstd
  1029. compression speed ranges from 250 to 500 MB/s per thread
  1030. and decompression speed is over 1 GB/s per thread.
  1031. .Pp
  1032. When the
  1033. .Sy zstd
  1034. feature is set to
  1035. .Sy enabled ,
  1036. the administrator can turn on
  1037. .Sy zstd
  1038. compression of any dataset using
  1039. .Nm zfs Cm set Sy compress Ns = Ns Sy zstd Ar dset
  1040. .Po see Xr zfs-set 8 Pc .
  1041. This feature becomes
  1042. .Sy active
  1043. once a
  1044. .Sy compress
  1045. property has been set to
  1046. .Sy zstd ,
  1047. and will return to being
  1048. .Sy enabled
  1049. once all filesystems that have ever had their
  1050. .Sy compress
  1051. property set to
  1052. .Sy zstd
  1053. are destroyed.
  1054. .El
  1055. .
  1056. .Sh SEE ALSO
  1057. .Xr zfs 8 ,
  1058. .Xr zpool 8