]> git.ipfire.org Git - thirdparty/e2fsprogs.git/log
thirdparty/e2fsprogs.git
3 years agoe2fsck: Annotating fields in e2fsck_struct pfsck
Saranya Muruganandam [Tue, 17 Nov 2020 17:54:30 +0000 (09:54 -0800)] 
e2fsck: Annotating fields in e2fsck_struct

Adding information on fields in e2fsck_struct
on how they are used when running parallel fsck.

Signed-off-by: Saranya Muruganandam <saranyamohan@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: propagate number of threads
Saranya Muruganandam [Fri, 13 Nov 2020 23:33:05 +0000 (15:33 -0800)] 
e2fsck: propagate number of threads

Sometimes, such as in orphan_inode case, e2fsck_pass1
is called after reading the block bitmaps. This results in
reading the block bitmap sequentially and multithreading
only gets kicked in later. Fix the thread count earlier
while setting up the file system.

Signed-off-by: Saranya Muruganandam <saranyamohan@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: misc cleanups for pfsck
Andreas Dilger [Mon, 28 Sep 2020 19:21:16 +0000 (13:21 -0600)] 
e2fsck: misc cleanups for pfsck

Add -m option description to e2fsck.8 man page.

Rename e2fsck_struct fs_num_threads to pfs_num_threads to avoid
confusion with the ext2_filsys fs_num_threads field, and move
thread_info to be together with the other HAVE_PTHREAD fields.

Move ext2_filsys fs_num_threads to fit into the __u16 "pad" field
to avoid consuming one of the few remaining __u32 reserved fields.

Fix a few print format warnings.

Signed-off-by: Andreas Dilger <adilger@whamcloud.com>
Reviewed-by: Wang Shilong <wshilong@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: fix memory leaks with pfsck enabled
Wang Shilong [Mon, 28 Sep 2020 02:27:12 +0000 (10:27 +0800)] 
e2fsck: fix memory leaks with pfsck enabled

valgrind detected two memory leaks:

1) quota context is not released after merging.
2) @refcount_orig should be released

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agotests: add pfsck test
Wang Shilong [Wed, 23 Sep 2020 15:58:23 +0000 (23:58 +0800)] 
tests: add pfsck test

pfsck run on a clean fs should not return any errors.

Generate an image with possible features enabled,
especially EA shared blocks etc.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: reset @inodes_to_rebuild if restart
Wang Shilong [Thu, 24 Sep 2020 02:00:06 +0000 (10:00 +0800)] 
e2fsck: reset @inodes_to_rebuild if restart

Verify multiple thread on a corrupted images hit following bug:

pass1.c:2902: e2fsck_pass1_thread_prepare:
Assertion `global_ctx->inodes_to_rebuild == NULL' failed.
Signal (6) SIGABRT si_code=SI_TKILL
./e2fsck/e2fsck[0x43829e]
/lib64/libpthread.so.0(+0x14b20)[0x7f3b45135b20]
/lib64/libc.so.6(gsignal+0x145)[0x7f3b44f2c625]
/lib64/libc.so.6(abort+0x12b)[0x7f3b44f158d9]
/lib64/libc.so.6(+0x257a9)[0x7f3b44f157a9]
/lib64/libc.so.6(+0x34a66)[0x7f3b44f24a66]
./e2fsck/e2fsck(e2fsck_pass1+0x1662)[0x423572]
./e2fsck/e2fsck(e2fsck_run+0x5a)[0x41611a]
./e2fsck/e2fsck(main+0x1608)[0x4121b8]
/lib64/libc.so.6(__libc_start_main+0xf3)[0x7f3b44f171a3]
./e2fsck/e2fsck(_start+0x2e)[0x413dde]

@inodes_to_rebuild could be not NULL after we restart pass1

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: update mmp block in one thread
Wang Shilong [Wed, 23 Sep 2020 14:36:41 +0000 (22:36 +0800)] 
e2fsck: update mmp block in one thread

For multiple threads, different threads will try to
update mmp block at the same time, only allow one
thread to update it.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: make default smallest RA size to 1M
Wang Shilong [Tue, 15 Sep 2020 07:06:11 +0000 (15:06 +0800)] 
e2fsck: make default smallest RA size to 1M

If we have a smaller inodes per group, default ra size could
be very small(etc 128KiB), this hurts performances.

Tune above 128K to 1M, i see pass1 time drop down from
677.12 seconds to 246 secons with 32 threads.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: avoid too much memory allocation for pfsck
Wang Shilong [Wed, 27 May 2020 07:14:23 +0000 (07:14 +0000)] 
e2fsck: avoid too much memory allocation for pfsck

e2fsck init memory according to filesystem inodes/dir numbers
recorded in the superblock, this should be aware of filesystem
number of threads, otherwise, oom happen.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: cleanup e2fsck_pass1_thread_join()
Wang Shilong [Wed, 23 Sep 2020 10:05:34 +0000 (18:05 +0800)] 
e2fsck: cleanup e2fsck_pass1_thread_join()

Use e2fsck_reset_context() to free memory to simpify
codes.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: wait fix thread finish before checking
Wang Shilong [Thu, 14 May 2020 14:16:50 +0000 (22:16 +0800)] 
e2fsck: wait fix thread finish before checking

Before proceeding next inodes, waitting existed
fixing finished.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Saranya Muruganandam <saranyamohan@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: set E2F_FLAG_ALLOC_OK after threads
Wang Shilong [Thu, 16 Apr 2020 07:08:31 +0000 (15:08 +0800)] 
e2fsck: set E2F_FLAG_ALLOC_OK after threads

Only flag ALLOC OK after all threads finished without problem.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: simplify e2fsck context merging codes
Wang Shilong [Fri, 13 Mar 2020 10:46:24 +0000 (18:46 +0800)] 
e2fsck: simplify e2fsck context merging codes

We tried to copy thread context to global context directly
and then copy back some saved variables before merging.

Since we have finished almost all necessary variables
in the e2fsck context, we could simplify codes, and
this could help us understand what is missing rather
than hide problems.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: merge extent depth count after threads finish
Wang Shilong [Fri, 20 Mar 2020 01:55:18 +0000 (09:55 +0800)] 
e2fsck: merge extent depth count after threads finish

tests covered by f_extent_htree.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: reset lost_and_found after threads finish
Wang Shilong [Fri, 20 Mar 2020 01:50:56 +0000 (09:50 +0800)] 
e2fsck: reset lost_and_found after threads finish

This should not be kept, the reaons is similar to what
e2fsck_pass1 has done before.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: merge options after threads finish
Wang Shilong [Thu, 19 Mar 2020 14:05:30 +0000 (22:05 +0800)] 
e2fsck: merge options after threads finish

It will be possible that threads might append E2F_OPT_YES,
so we need merge options to global, test f_yesall cover this.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Saranya Muruganandam <saranyamohan@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: fix readahead for pfsck of pass1
Wang Shilong [Wed, 11 Mar 2020 12:48:00 +0000 (20:48 +0800)] 
e2fsck: fix readahead for pfsck of pass1

Several improvments for this patch:

1) move readahead_kb detection to preparing phase.
2) inode readahead should be aware of thread block group
boundary.
3) make readahead_kb aware of multiple threads.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: adjust number of threads
Wang Shilong [Wed, 9 Sep 2020 07:07:10 +0000 (15:07 +0800)] 
e2fsck: adjust number of threads

number of threads should not exceed flex bg numbers,
and output messages if we adjust threads number.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: allow admin specify number of threads
Wang Shilong [Mon, 9 Mar 2020 07:11:10 +0000 (15:11 +0800)] 
e2fsck: allow admin specify number of threads

-m option is introduced to specify number of threads for pfsck.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: kickoff mutex lock for block found map
Wang Shilong [Wed, 11 Mar 2020 02:12:18 +0000 (10:12 +0800)] 
e2fsck: kickoff mutex lock for block found map

Now @block_found_map is no longer shared by multiple threads,
and @block_dup_map need be checked again after threads finish.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Saranya Muruganandam <saranyamohan@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: merge EA blocks properly
Wang Shilong [Thu, 12 Mar 2020 08:05:51 +0000 (16:05 +0800)] 
e2fsck: merge EA blocks properly

EA blocks might be shared, merge them carefully.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Saranya Muruganandam <saranyamohan@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: split and merge invalid bitmaps
Wang Shilong [Sat, 14 Dec 2019 05:12:36 +0000 (13:12 +0800)] 
e2fsck: split and merge invalid bitmaps

Invalid bitmaps are splitted per thread, and we
should merge them after thread finish.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: move some fixes out of parallel pthreads
Wang Shilong [Sat, 7 Mar 2020 14:00:45 +0000 (22:00 +0800)] 
e2fsck: move some fixes out of parallel pthreads

We could only use @found_map_block to find free blocks
after we have collectd all used blocks, so something like
handle_fs_bad_blocks(), ext2fs_create_resize_inode(),
e2fsck_pass1_dupblocks() really should be handled after
all threads has been finished.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Saranya Muruganandam <saranyamohan@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: serialize fix operations
Wang Shilong [Fri, 6 Mar 2020 08:46:47 +0000 (16:46 +0800)] 
e2fsck: serialize fix operations

Allow different threads to fix at the same time could
be dangerous and error-prone now, and most of time
parallel scanning and checking is important.

So this patch adds a mutex to serialize
fix operations during pass1.

And the good benefit of this, we don't need block
allocations and free, superblock updates protection
any more, since only fix operations during pass1
could touch them.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Saranya Muruganandam <saranyamohan@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: merge quota context after threads finish
Wang Shilong [Fri, 6 Mar 2020 15:08:07 +0000 (23:08 +0800)] 
e2fsck: merge quota context after threads finish

Every threads calculate its own quota accounting,
merge them after threads finish.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: merge context flags properly
Wang Shilong [Mon, 24 Feb 2020 05:08:58 +0000 (13:08 +0800)] 
e2fsck: merge context flags properly

e2fsck might restart after pass1, so we should keep
flags if possible, this patch try to fix f_illitable_flexbg failure

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Saranya Muruganandam <saranyamohan@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: merge dirs_to_hash when threads finish
Wang Shilong [Mon, 24 Feb 2020 04:44:06 +0000 (12:44 +0800)] 
e2fsck: merge dirs_to_hash when threads finish

@dirs_to_hash list need be merged after threads finish,
test covered by t_dangerous.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: merge dx_dir_info after threads finish
Wang Shilong [Fri, 6 Dec 2019 15:37:53 +0000 (23:37 +0800)] 
e2fsck: merge dx_dir_info after threads finish

Merge properly.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: merge fs flags when threads finish
Li Xi [Sun, 8 Sep 2019 08:15:35 +0000 (16:15 +0800)] 
e2fsck: merge fs flags when threads finish

merge fs flags properly.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: merge counts after threads finish
Wang Shilong [Tue, 22 Sep 2020 13:14:39 +0000 (21:14 +0800)] 
e2fsck: merge counts after threads finish

Merge counts properly.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Saranya Muruganandam <saranyamohan@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: add debug codes for multiple threads
Li Xi [Thu, 5 Sep 2019 11:40:36 +0000 (19:40 +0800)] 
e2fsck: add debug codes for multiple threads

These debug codes are added to run the multiple pass1 check
thread one by one in order. If all the codes are correct,
fsck of multiple threads should have exactly the same outcome
with single thread.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: merge dblist after thread finishes
Li Xi [Thu, 5 Sep 2019 08:30:40 +0000 (16:30 +0800)] 
e2fsck: merge dblist after thread finishes

Merge dblist properly.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: merge icounts after thread finishes
Li Xi [Wed, 4 Sep 2019 10:47:14 +0000 (18:47 +0800)] 
e2fsck: merge icounts after thread finishes

Merge inode_count and inode_link_info properly after
threads finish.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: merge badblocks after thread finishes
Wang Shilong [Tue, 22 Sep 2020 10:00:39 +0000 (18:00 +0800)] 
e2fsck: merge badblocks after thread finishes

Badblocks should be merged properly after threads finish.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: rbtree bitmap for dir
Wang Shilong [Mon, 21 Sep 2020 11:02:33 +0000 (19:02 +0800)] 
e2fsck: rbtree bitmap for dir

Only rbtree support merge operation now, use it for bitmaps.

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: merge dir_info after thread finishes
Li Xi [Mon, 2 Sep 2019 08:14:38 +0000 (16:14 +0800)] 
e2fsck: merge dir_info after thread finishes

dir_info need be merged after thread finish.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: optimize the inserting of dir_info_db
Li Xi [Fri, 30 Aug 2019 09:56:10 +0000 (17:56 +0800)] 
e2fsck: optimize the inserting of dir_info_db

Binary search is now used when inserting an dir info to the array.
Memmove is now used when moving array. Both of them improves
the performance of inserting.

This patch is also a prepartion for the merging of two dir db
arrays.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: do not change global variables
Li Xi [Fri, 30 Aug 2019 04:20:11 +0000 (12:20 +0800)] 
e2fsck: do not change global variables

Global variables used in pass1 check are changed to local variables
in this patch. This will avoid conflict between threads.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: merge bitmaps after thread completes
Wang Shilong [Wed, 23 Sep 2020 03:17:29 +0000 (11:17 +0800)] 
e2fsck: merge bitmaps after thread completes

A new method merge_bmap has been added to bitmap operations. But
only red-black bitmap has that operation now.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: print thread log properly
Li Xi [Mon, 26 Aug 2019 14:34:06 +0000 (22:34 +0800)] 
e2fsck: print thread log properly

When multi-thread fsck is enabled, logs printed from multiple
threads could overlap with each other. The overlap sometimes
makes the logs unreadable because log_out() is used multiple times
for a single line.

This patch adds leading [Thread XXX] to each logs if multi-thread
is enabed by -m option.

This patch also adds message to show the group ranges and inode
numbers for each thread, which is useful for debuging multi-thread
check.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: split groups to different threads
Li Xi [Thu, 22 Aug 2019 07:35:12 +0000 (15:35 +0800)] 
e2fsck: split groups to different threads

The start/end groups of a thread is calculated according to the
thread number. But still, only one thread is used to check.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: add start/end group for thread
Li Xi [Wed, 21 Aug 2019 13:58:31 +0000 (21:58 +0800)] 
e2fsck: add start/end group for thread

When multi-threads are used for check, each thread needs to jump
to different group in pass1 check. This patch adds the group
jumping support. But still, only one thread is used to check.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: configure one pfsck thread
Li Xi [Thu, 15 Aug 2019 08:33:19 +0000 (16:33 +0800)] 
e2fsck: configure one pfsck thread

This patch creates only one thread to do pass1 check if pthreads are
enabled. The same codes can be used to create multiple threads, but
other functions need to be modified to get ready for that.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: create logs for mult-threads
Li Xi [Tue, 13 Aug 2019 03:55:27 +0000 (11:55 +0800)] 
e2fsck: create logs for mult-threads

When multi-threads are used, different logs should be created
for different threads. Each thread has log files with suffix
of ".$THREAD_INDEX".

And this patch adds f_multithread_logfile test case.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: open io-channel when copying fs
Li Xi [Sat, 10 Aug 2019 07:59:09 +0000 (15:59 +0800)] 
e2fsck: open io-channel when copying fs

This patch also add writethrough flag to the thread io-channel.
When multiple threads write the same disk, we don't want the
data being saved in memory cache. This will be useful in the
future, but even without that flag, the tests can be passed too.

This patch also cleanup the io channel cache of the global
context. Otherwise, after pass1 step, the next steps would use
old data saved in the cache. And the cached data might have
already been overwritten in pass1.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Saranya Muruganandam <saranyamohan@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: copy badblocks when copying fs
Wang Shilong [Wed, 9 Sep 2020 07:07:10 +0000 (15:07 +0800)] 
e2fsck: copy badblocks when copying fs

This patch copies badblocks when the copying fs.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: copy bitmaps when copying context
Li Xi [Wed, 7 Aug 2019 08:54:41 +0000 (16:54 +0800)] 
e2fsck: copy bitmaps when copying context

This patch copies bitmap when the copying context. In the
multi-thread fsck, each thread use different bitmap that copied
from the glboal bitmap. And Bitmaps from multiple threads will
be merged into a global one after the pass1 finishes.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: add assert when copying context
Li Xi [Wed, 7 Aug 2019 07:34:00 +0000 (15:34 +0800)] 
e2fsck: add assert when copying context

Adding the assert would simplify the copying of context.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: clear icache when using multi-thread fsck
Wang Shilong [Thu, 17 Sep 2020 02:58:56 +0000 (10:58 +0800)] 
e2fsck: clear icache when using multi-thread fsck

icache of fs will be rebuilt when needed, so after copying
fs, icache can be inited to NULL.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: copy fs when using multi-thread fsck
Li Xi [Tue, 6 Aug 2019 03:19:15 +0000 (11:19 +0800)] 
e2fsck: copy fs when using multi-thread fsck

This patch only copy the fs to a new one when -m is enabled.
It doesn't actually start any thread. When pass1 test finishes,
the new fs is copied back to the original context.

This patch handles the fs fields in dblist, inode_map and block_map
properly.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: copy context when using multi-thread fsck
Li Xi [Mon, 5 Aug 2019 04:01:17 +0000 (12:01 +0800)] 
e2fsck: copy context when using multi-thread fsck

This patch only copy the context to a new one when -m is enabled.
It doesn't actually start any thread. When pass1 test finishes,
the new context is copied back to the original context.

Since the signal handler only changes the original context, so
add global_ctx in "struct e2fsck_struct" and use that to check
whether there is any signal of canceling.

This patch handles the long jump properly so that all the existing
tests can be passed even the context has been copied. Otherwise,
test f_expisize_ea_del would fail when aborting.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: add -m option for multithread
Li Xi [Sun, 4 Aug 2019 15:28:59 +0000 (23:28 +0800)] 
e2fsck: add -m option for multithread

-m option is added but no actual functionality is added. This
patch only adds the logic that when -m is specified, one of
-p/-y/-n options should be specified. And when -m is specified,
-C shouldn't be specified and the completion progress report won't
be triggered by sending SIGUSR1/SIGUSR2 signals. This simplifies
the implementation of multi-thread fsck in the future.

Completion progress support with multi-thread fsck will be added
back after multi-thread fsck implementation is finished. Right
now, disable it to simplify the implementation of multi-thread fsck.

Signed-off-by: Li Xi <lixi@ddn.com>
Signed-off-by: Wang Shilong <wshilong@ddn.com>
Reviewed-by: Andreas Dilger <adilger@whamcloud.com>
Signed-off-by: Saranya Muruganandam <saranyamohan@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoMerge branch 'maint' into next
Theodore Ts'o [Sat, 23 Jan 2021 06:05:43 +0000 (01:05 -0500)] 
Merge branch 'maint' into next

3 years agoInclude PTHREAD_CFLAGS in LDFLAGS* macros
Theodore Ts'o [Sat, 23 Jan 2021 06:02:19 +0000 (01:02 -0500)] 
Include PTHREAD_CFLAGS in LDFLAGS* macros

PTHREAD_CFLAGS is set by AX_PTHREADS, and these flags need to be
included when linking executables.

Fixes: bdcd5f22203f ("Add configure and build support for the pthreads library")
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoFix clang warnings
Theodore Ts'o [Sat, 23 Jan 2021 05:57:18 +0000 (00:57 -0500)] 
Fix clang warnings

Clang gets unhappy when passing an unsigned char to string functions.
For better or for worse we use __u8[] in the definition of the
superblock.  So cast them these to "char *" to prevent clang
build-time warnings.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agolibext2fs: use compiler built-in offsetof() if available
Theodore Ts'o [Sat, 23 Jan 2021 05:55:25 +0000 (00:55 -0500)] 
libext2fs: use compiler built-in offsetof() if available

This avoids UBSAN sanitizer warnings, since &(0->member) is
technically undefined per the C standard.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoOnly build resize2fs.static when running "make all-static"
Theodore Ts'o [Sat, 23 Jan 2021 05:34:54 +0000 (00:34 -0500)] 
Only build resize2fs.static when running "make all-static"

Fixes: 93df80d2409d ("Teach makefiles... the target all-static")
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agolibext2fs: fix UBSAN warning in ext2fs_mmp_new_seq()
Theodore Ts'o [Fri, 22 Jan 2021 04:27:00 +0000 (23:27 -0500)] 
libext2fs: fix UBSAN warning in ext2fs_mmp_new_seq()

Left shifting the pid by 16 bits can cause a UBSAN warning if the pid
is greater than or equal to 2**16.  It doesn't matter since we're just
using the pid to seed for a pseudo-random number generator, but
silence the warning by just swapping the high and low 16 bits of the
pid instead.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agomke2fs.8: Improve valid block size documentation
Jan Kara [Thu, 21 Jan 2021 23:26:25 +0000 (18:26 -0500)] 
mke2fs.8: Improve valid block size documentation

Explain which valid block sizes mke2fs supports in more detail.

Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agobuild: Add SYSLIBS to e4crypt linking
Hauke Mehrtens [Thu, 21 Jan 2021 23:11:30 +0000 (18:11 -0500)] 
build: Add SYSLIBS to e4crypt linking

The $(SYSLIBS) was missing when linking the e4crypt application. This is
available in the e4crypt.profiled variant, so I assume this was just
missing in the normal variant and is not left out intentionally.

Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agotune2fs: abort clearing the dir_index when the fs needs to be fsck'ed first
Theodore Ts'o [Thu, 21 Jan 2021 22:08:40 +0000 (17:08 -0500)] 
tune2fs: abort clearing the dir_index when the fs needs to be fsck'ed first

We were not checking the return value of check_fsck_needed() when
checking to clear the dir_index feature.  As a result, tune2fs would
print that the file system needed to be checked first, but then go
ahead and clear the dir_index flag.

Addresses-Coverity-Bug: 1467671
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: remove dead code when recreating the journal
Theodore Ts'o [Thu, 21 Jan 2021 21:51:45 +0000 (16:51 -0500)] 
e2fsck: remove dead code when recreating the journal

params.num_journal_blocks is an unsigned value so it can never be less
than zero.

Addresses-Coverity-Bug: 1472250
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agodebugfs: fix the printf specifier when dumping a fast commit block
Theodore Ts'o [Thu, 21 Jan 2021 21:44:39 +0000 (16:44 -0500)] 
debugfs: fix the printf specifier when dumping a fast commit block

Addresses-Coverity-Bug: 1472249
Addresses-Coverity-Bug: 1472253
Addresses-Coverity-Bug: 1472254
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agolibext2fs: fix minor Coverity nits in ext2fs_rw_bitmaps()
Theodore Ts'o [Thu, 21 Jan 2021 21:34:57 +0000 (16:34 -0500)] 
libext2fs: fix minor Coverity nits in ext2fs_rw_bitmaps()

Addresses-Coverity-Bug: 1472252
Addresses-Coverity-Bug: 1472253
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agotune2fs: fix resource leak in handle_quota_options()
Theodore Ts'o [Thu, 21 Jan 2021 21:07:25 +0000 (16:07 -0500)] 
tune2fs: fix resource leak in handle_quota_options()

Addresses-Coverity-Bug: 1467672
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agodebugfs: fix double free in realloc() error path in read_list()
Theodore Ts'o [Thu, 21 Jan 2021 21:01:14 +0000 (16:01 -0500)] 
debugfs: fix double free in realloc() error path in read_list()

Fixes-Coverity-Bug: 1464575
Fixes-Coverity-Bug: 1464571
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agolibext2fs: fix incorrect negative error return in unix and sparse io managers
Theodore Ts'o [Thu, 21 Jan 2021 21:00:01 +0000 (16:00 -0500)] 
libext2fs: fix incorrect negative error return in unix and sparse io managers

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agolibext2fs: fix incorrect negative error return in ext2fs_rw_bitmaps()
Theodore Ts'o [Thu, 21 Jan 2021 20:20:45 +0000 (15:20 -0500)] 
libext2fs: fix incorrect negative error return in ext2fs_rw_bitmaps()

Fixes: e2e58d312804 ("ext2fs: parallel bitmap loading")
Fixes-Coverity-Bug: 147255
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agodebugfs: add fast commit support to logdump
Harshad Shirwadkar [Wed, 20 Jan 2021 21:26:39 +0000 (13:26 -0800)] 
debugfs: add fast commit support to logdump

Add fast commit support for debugfs logdump.

This commit also adds fast_commit.h that contains the necessary
helpers needed for fast commit replay. Note that this file is also
byte by byte identical with kernel's fast_commit.h.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoext4: fix tests to account for new dumpe2fs output
Harshad Shirwadkar [Wed, 20 Jan 2021 21:26:41 +0000 (13:26 -0800)] 
ext4: fix tests to account for new dumpe2fs output

dumpe2fs tool now is capable of reporting number of fast commit
blocks. There were slight changes in the output of dumpe2fs outside of
fast commits. This patch fixes the regression tests to expect the new
output.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoMake userspace tools number of fast commits blocks aware
Harshad Shirwadkar [Wed, 20 Jan 2021 21:26:31 +0000 (13:26 -0800)] 
Make userspace tools number of fast commits blocks aware

This patch makes number of fast commit blocks configurable. Also, the
number of fast commit blocks can now be seen in dumpe2fs output.

$ ./misc/mke2fs -O fast_commit -t ext4 image
mke2fs 1.46-WIP (20-Mar-2020)
Discarding device blocks: done
Creating filesystem with 5120 1k blocks and 1280 inodes
Allocating group tables: done
Writing inode tables: done
Creating journal (1040 blocks): done
Writing superblocks and filesystem accounting information: done

$ ./misc/dumpe2fs image
dumpe2fs 1.46-WIP (20-Mar-2020)
...
Journal features:         (none)
Total journal size:       1040k
Total journal blocks:     1040
Max transaction length:   1024
Fast commit length:       16
Journal sequence:         0x00000001
Journal start:            0

$ ./misc/mke2fs -O fast_commit -t ext4 image -J fast_commit_size=256,size=1
mke2fs 1.46-WIP (20-Mar-2020)
Creating filesystem with 5120 1k blocks and 1280 inodes
Allocating group tables: done
Writing inode tables: done
Creating journal (1280 blocks): done
Writing superblocks and filesystem accounting information: done

$ ./misc/dumpe2fs image
dumpe2fs 1.46-WIP (20-Mar-2020)
...
Journal features:         (none)
Total journal size:       1280k
Total journal blocks:     1280
Max transaction length:   1024
Fast commit length:       256
Journal sequence:         0x00000001
Journal start:            0

This patch also adds information about fast commit feature in mke2fs
and tune2fs man pages.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agolibext2fs: provide APIs to configure fast commit blocks
Harshad Shirwadkar [Wed, 20 Jan 2021 21:26:30 +0000 (13:26 -0800)] 
libext2fs: provide APIs to configure fast commit blocks

This patch adds new libext2fs that allow configuring number of fast
commit blocks in journal superblock. We also add a struct
ext2fs_journal_params which contains number of fast commit blocks and
number of normal journal blocks. With this patch, the preferred way
for configuring number of blocks with and without fast commits is:

struct ext2fs_journal_params params;

ext2fs_get_journal_params(&params, ...);
params.num_journal_blocks = ...;
params.num_fc_blocks = ...;
ext2fs_create_journal_superblock2(..., &params, ...);
         OR
ext2fs_add_journal_inode3(..., &params, ...);

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: port fc changes from kernel's recovery.c to e2fsck
Harshad Shirwadkar [Wed, 20 Jan 2021 21:26:29 +0000 (13:26 -0800)] 
e2fsck: port fc changes from kernel's recovery.c to e2fsck

This patch makes recovery.c identical with fast commit kernel changes.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Reviewed-by: Theodore Ts'o <tytso@mit.edu>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoe2fsck: add kernel endian-ness conversion macros
Harshad Shirwadkar [Wed, 20 Jan 2021 21:26:28 +0000 (13:26 -0800)] 
e2fsck: add kernel endian-ness conversion macros

In order to make recovery.c identical with kernel, we need endianness
conversion macros (such as cpu_to_be32 and friends) defined in
e2fsprogs. This patch defines these macros and also fixes recovery.c
to use these. These macros are also needed for fast commit recovery
patches later in this series.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoext2fs: move calculate_summary_stats to ext2fs lib
Harshad Shirwadkar [Wed, 20 Jan 2021 21:26:27 +0000 (13:26 -0800)] 
ext2fs: move calculate_summary_stats to ext2fs lib

The function calculate_summary_stats sets the global metadata of the
file system. Tune2fs had this function defined statically in
tune2fs.c. Fast commit replay needs this function to set global
metadata at the end of the replay phase. So, move this function to
libext2fs.

Signed-off-by: Harshad Shirwadkar <harshadshirwadkar@gmail.com>
Reviewed-by: Theodore Ts'o <tytso@mit.edu>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoEnable threaded support for e2fsprogs' applications.
Theodore Ts'o [Thu, 14 Jan 2021 00:27:23 +0000 (16:27 -0800)] 
Enable threaded support for e2fsprogs' applications.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoext2fs: parallel bitmap loading
Wang Shilong [Thu, 14 Jan 2021 00:27:22 +0000 (16:27 -0800)] 
ext2fs: parallel bitmap loading

In our benchmarking for PiB size filesystem, pass5 takes
10446s to finish and 99.5% of time takes on reading bitmaps.

It makes sense to reading bitmaps using multiple threads,
a quickly benchmark show 10446s to 626s with 64 threads.

[ This has all of many bug fixes for rw_bitmaps.c from the original
  luster patch set collapsed into a single commit.   In addition it has
  the new ext2fs_rw_bitmaps() api proposed by Ted. ]

Signed-off-by: Wang Shilong <wshilong@ddn.com>
Signed-off-by: Saranya Muruganandam <saranyamohan@google.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agolibext2fs: allow the unix_io manager's cache to be disabled and re-enabled
Theodore Ts'o [Thu, 14 Jan 2021 00:27:21 +0000 (16:27 -0800)] 
libext2fs: allow the unix_io manager's cache to be disabled and re-enabled

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agolibext2fs: add threading support to the I/O manager abstraction
Theodore Ts'o [Thu, 14 Jan 2021 00:27:20 +0000 (16:27 -0800)] 
libext2fs: add threading support to the I/O manager abstraction

Add initial implementation support for the unix_io manager.
Applications which want to use threading should pass in
IO_FLAG_THREADS when opening the channel.  Channels which support
threading (which as of this commit is unix_io and test_io if the
backing io_manager supports threading) will set the
CHANNEL_FLAGS_THREADS bit in io->flags.  Library code or applications
can test if threading is enabled by checking this flag.

Applications using libext2fs can pass in EXT2_FLAG_THREADS to
ext2fs_open() or ext2fs_open2() to request threading support.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoAdd configure and build support for the pthreads library
Theodore Ts'o [Thu, 14 Jan 2021 00:27:19 +0000 (16:27 -0800)] 
Add configure and build support for the pthreads library

Support for pthreads can be forcibly disabled by passing
"--without-pthread" to the configure script.

The actual changes in this commit are in configure.ac and MCONFIG.in;
the other files were generated as a result of running aclocal,
autoconf, and autoheader on a Debian testing system.

Note: the autoconf-archive package must now be installed before
rerunning aclocal, to supply the AX_PTHREAD macro.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoMerge branch 'maint' into next
Theodore Ts'o [Wed, 20 Jan 2021 04:58:57 +0000 (23:58 -0500)] 
Merge branch 'maint' into next

3 years agomke2fs: Escape double quotes when parsing mke2fs.conf
Lukas Czerner [Mon, 2 Nov 2020 14:26:31 +0000 (15:26 +0100)] 
mke2fs: Escape double quotes when parsing mke2fs.conf

Currently, when constructing the <default> configuration pseudo-file using
the profile-to-c.awk script we will just pass the double quotes as they
appear in the mke2fs.conf.

This is problematic, because the resulting default_profile.c will either
fail to compile because of syntax error, or leave the resulting
configuration invalid.

It can be reproduced by adding the following line somewhere into
mke2fs.conf configuration and forcing mke2fs to use the <default>
configuration by specifying nonexistent mke2fs.conf

MKE2FS_CONFIG="nonexistent" ./misc/mke2fs -T ext4 /dev/device

default_mntopts = "acl,user_xattr"
^ this will fail to compile

default_mntopts = ""
^ this will result in invalid config file

Syntax error in mke2fs config file (<default>, line #4)
       Unknown code prof 17

Fix it by escaping the double quotes with a backslash in
profile-to-c.awk script.

Signed-off-by: Lukas Czerner <lczerner@redhat.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agolibext2fs: add gnu.translator support
Romain Naour [Mon, 2 Nov 2020 13:03:19 +0000 (14:03 +0100)] 
libext2fs: add gnu.translator support

The support of setting (and reading) of passive translators from
GNU/Linux has been added to the Linux kernel by the commit [1].
The name index '10' has been reserved for GNU/Hurd.

Hurd passive translators are stored as a xattr value with name
"gnu.translator" [2].

If "gnu.translator" xattr value has been set before calling
mkfs.ext2, it will segfault since "gnu." is not present in
ea_names[].

$ setfattr -n gnu.translator -v "/hurd/exec\0" ${TARGET_DIR}/servers/exec
$ mkfs.ext2 -d ${TARGET_DIR} -o hurd -O ext_attr rootfs.ext2 "1G"

Adding "gnu." to ea_names[], allow to create ext2 filesystem
for GNU/Hurd with passive translator already set.

[1] https://git.savannah.gnu.org/cgit/hurd/hurd.git/commit/?id=a04c7bf83172faa7cb080fbe3b6c04a8415ca645
[2] https://lists.gnu.org/archive/html/bug-hurd/2016-08/msg00075.html

Signed-off-by: Romain Naour <romain.naour@gmail.com>
Cc: Theodore Ts'o <tytso@mit.edu>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agofilefrag: handle invalid st_dev and blksize cases
Luis Henriques [Wed, 28 Oct 2020 15:55:50 +0000 (15:55 +0000)] 
filefrag: handle invalid st_dev and blksize cases

It is possible to crash filefrag with a "Floating point exception" in
two different scenarios:

1. When fstat() returns a device ID set to 0
2. When FIGETBSZ ioctl returns a blocksize of 0

In both scenarios a divide-by-zero will occur in frag_report() because
variable blksize will be set to zero.

I've managed to trigger this crash with an old CephFS kernel client,
using xfstest generic/519.  The first scenario has been fixed by kernel
commit 75c9627efb72 ("ceph: map snapid to anonymous bdev ID").  The
second scenario is also fixed with commit 8f97d1e99149 ("vfs: fix
FIGETBSZ ioctl on an overlayfs file").

However, it is desirable to handle these two scenarios gracefully by
checking these conditions explicitly.

Signed-off-by: Luis Henriques <lhenriques@suse.de>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agocreate_inode: set xattrs to the root directory as well
Antoine Tenart [Fri, 17 Jul 2020 10:08:46 +0000 (12:08 +0200)] 
create_inode: set xattrs to the root directory as well

populate_fs do copy the xattrs for all files and directories, but the
root directory is skipped and as a result its extended attributes aren't
set. This is an issue when using mkfs to build a full system image that
can be used with SElinux in enforcing mode without making any runtime
fix at first boot.

This patch adds logic to set the root directory's extended attributes.

Signed-off-by: Antoine Tenart <antoine.tenart@bootlin.com>
Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agopo: reapply local e2fsprogs chages to po/Makefile.in.in
Theodore Ts'o [Wed, 25 Nov 2020 13:47:54 +0000 (08:47 -0500)] 
po: reapply local e2fsprogs chages to po/Makefile.in.in

These are the changes which are needed after running gettextize to
update to gettext 0.19.8 in the previous commit.

 * Add support for maintainer mode (which doesn't do as much given
     that gettext now has settings in Makevars which allows us to
     suppress automatic updates of the po and gmo files)
 * Add support to expand the '@' abbreviations in e2fsck/problem.c
     and give an explanation of how they work for translators
 * Add support for configure --enable-verbose-makecmds and default to
     "kernel-style" quieter make output --- this makes it easier
     to see warnings and errors by suppressing the distracting
     details.
 * Teach the makefile where to find the generated error table C files
     in the build directory.
 * Add make targets (e.g., all-static, fullcheck, coverage.txt) which
     are required by the top-level Makefile.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoUpdate gettext files to version 0.19.8
Theodore Ts'o [Wed, 25 Nov 2020 04:00:57 +0000 (23:00 -0500)] 
Update gettext files to version 0.19.8

This also removes the built-in "intl" directory as this is now
considered deprecated by the gettext package.  This means that we
won't try to use an internal version of gettext if it's not installed
on the build system.  We will simply disable NLS support in that case.

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agopo: update sr.po (from translationproject.org)
Мирослав Николић [Wed, 25 Nov 2020 03:36:54 +0000 (22:36 -0500)] 
po: update sr.po (from translationproject.org)

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agopo: update nl.po (from translationproject.org)
Benno Schulenberg [Wed, 25 Nov 2020 03:36:54 +0000 (22:36 -0500)] 
po: update nl.po (from translationproject.org)

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agopo: update ms.po (from translationproject.org)
Sharuzzaman Ahmat Raslan [Wed, 25 Nov 2020 03:36:54 +0000 (22:36 -0500)] 
po: update ms.po (from translationproject.org)

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agorelease notes: delete two files that are fully contained within v1.41.txt
Benno Schulenberg [Wed, 28 Oct 2020 03:41:06 +0000 (23:41 -0400)] 
release notes: delete two files that are fully contained within v1.41.txt

They are pure duplicates.

Signed-off-by: Benno Schulenberg <bensberg@telfort.nl>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agodebugfs: fix parse_uint for 64-bit fields
Theodore Ts'o [Tue, 6 Oct 2020 12:29:09 +0000 (08:29 -0400)] 
debugfs: fix parse_uint for 64-bit fields

The logic for handling 64-bit structure elements was reversed, which
caused attempts to set fields like kbytes_written to fail:

    % debugfs -w /tmp/foo.img
    debugfs 1.45.6 (20-Mar-2020)
    debugfs:  set_super_value kbytes_written 1024
    64-bit field kbytes_written has a second 64-bit field
    defined; BUG?!?

https://github.com/tytso/e2fsprogs/issues/36

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoDefine MKDIR_P in the Makefile.in files instead in MCONFIG.in
Theodore Ts'o [Mon, 5 Oct 2020 03:05:01 +0000 (23:05 -0400)] 
Define MKDIR_P in the Makefile.in files instead in MCONFIG.in

In the case where mkdir -p is not thread-safe (for example, if the
build environment is using busybox's mkdir) the configure script will
fall back to the slow (but safe) install-sh script.  In that case
MKDIR_P will be using a relative pathname; so we can't use speed
optimization of defining configure substitutions in MCONFIG.in, since
the substitution will be different depending on depth of the
subdirectory in the Makefile.in file.

https://github.com/tytso/e2fsprogs/issues/51

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoresize2fs: prevent block group descriptors from overflowing the first bg
Theodore Ts'o [Fri, 2 Oct 2020 18:47:25 +0000 (14:47 -0400)] 
resize2fs: prevent block group descriptors from overflowing the first bg

For 1k block file systems, resizing a file system larger than
1073610752 blocks will result in the size of the block group
descriptors to be so large that it will overlap with the backup
superblock in block group #1.   This problem can be reproduced via:

    mke2fs -t ext4 /tmp/foo.img 200M
    resize2fs /tmp/foo.img 1T
    e2fsck -f /tmp/foo.img

https://github.com/tytso/e2fsprogs/issues/50

Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agoAOSP: Fix a trivial type error
Yi Kong [Tue, 7 Jul 2020 14:32:12 +0000 (22:32 +0800)] 
AOSP: Fix a trivial type error

Comparing unsigned int with ULONG_MAX is always false.

Signed-off-by: Yi Kong <yikong@google.com>
Change-Id: Iae02aad1bcb271d3468828977be288ad04333821
From AOSP commit: 757a4d672dae1a15c57f5f0705ba90ed007da7e6

3 years agoAOSP: Include private/fs_config.h directly when needed
Tom Cherry [Wed, 4 Mar 2020 21:41:14 +0000 (13:41 -0800)] 
AOSP: Include private/fs_config.h directly when needed

This is no longer a transitive include of android_filesystem_config.h

Bug: 149785767
Test: build
Change-Id: I954adf7879fa811eb2b290a0983c84d47ecae26c
From AOSP commit: 9fa6bed983d5ddd7226af647a0c4c0d922227be4

3 years agoMerge branch 'maint' into next
Theodore Ts'o [Thu, 1 Oct 2020 21:13:27 +0000 (17:13 -0400)] 
Merge branch 'maint' into next

3 years agochattr/lsattr: Support dax attribute
Xiao Yang [Tue, 28 Jul 2020 05:33:21 +0000 (13:33 +0800)] 
chattr/lsattr: Support dax attribute

Use the letter 'x' to set/get dax attribute on a directory/file.

Signed-off-by: Xiao Yang <yangx.jy@cn.fujitsu.com>
Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agomke2fs: fix up check for hardlinks always false if inode > 0xFFFFFFFF
Hongxu Jia [Wed, 22 Jul 2020 01:25:03 +0000 (18:25 -0700)] 
mke2fs: fix up check for hardlinks always false if inode > 0xFFFFFFFF

While file has a large inode number (> 0xFFFFFFFF), mkfs.ext4 could
not parse hardlink correctly.

Prepare three hardlink files for mkfs.ext4

$ ls -il rootfs_ota/a rootfs_ota/boot/b rootfs_ota/boot/c
11026675846 -rw-r--r-- 3 hjia users 0 Jul 20 17:44 rootfs_ota/a
11026675846 -rw-r--r-- 3 hjia users 0 Jul 20 17:44 rootfs_ota/boot/b
11026675846 -rw-r--r-- 3 hjia users 0 Jul 20 17:44 rootfs_ota/boot/c

$ truncate -s 1M rootfs_ota.ext4

$ mkfs.ext4 -F -i 8192 rootfs_ota.ext4 -L otaroot -U fd5f8768-c779-4dc9-adde-165a3d863349 -d rootfs_ota

$ mkdir mnt && sudo mount rootfs_ota.ext4 mnt

$ ls -il mnt/a mnt/boot/b mnt/boot/c
12 -rw-r--r-- 1 hjia users 0 Jul 20 17:44 mnt/a
14 -rw-r--r-- 1 hjia users 0 Jul 20 17:44 mnt/boot/b
15 -rw-r--r-- 1 hjia users 0 Jul 20 17:44 mnt/boot/c

After applying this fix
$ ls -il mnt/a mnt/boot/b mnt/boot/c
12 -rw-r--r-- 3 hjia users 0 Jul 20 17:44 mnt/a
12 -rw-r--r-- 3 hjia users 0 Jul 20 17:44 mnt/boot/b
12 -rw-r--r-- 3 hjia users 0 Jul 20 17:44 mnt/boot/c

Since commit [382ed4a1 e2fsck: use proper types for variables][1]
applied, it used ext2_ino_t instead of ino_t for referencing inode
numbers, but the type of is_hardlink's `ino' should not be instead,
The ext2_ino_t is 32bit, if inode > 0xFFFFFFFF, its value will be
truncated.

Add a debug printf to show the value of inode, when it check for hardlink
files, it will always return false if inode > 0xFFFFFFFF
|--- a/misc/create_inode.c
|+++ b/misc/create_inode.c
|@@ -605,6 +605,7 @@ static int is_hardlink(struct hdlinks_s *hdlinks, dev_t dev, ext2_ino_t ino)
| {
|        int i;
|
|+       printf("%s %d, %lX, %lX\n", __FUNCTION__, __LINE__, hdlinks->hdl[i].src_ino, ino);
|        for (i = 0; i < hdlinks->count; i++) {
|                if (hdlinks->hdl[i].src_dev == dev &&
|                    hdlinks->hdl[i].src_ino == ino)

Here is debug message:
is_hardlink 608, 2913DB886913DB886

The length of ext2_ino_t is 32bit (typedef __u32 __bitwise ext2_ino_t;),
and ino_t is 64bit on 64bit system (such as x86-64), recover `ino' to ino_t.

[1] https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git/commit/?id=382ed4a1c2b60acb9db7631e86dda207bde6076e

Signed-off-by: Hongxu Jia <hongxu.jia@windriver.com>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>
3 years agomke2fs: Warn if fs block size is incompatible with DAX
Jan Kara [Thu, 9 Jul 2020 14:40:57 +0000 (16:40 +0200)] 
mke2fs: Warn if fs block size is incompatible with DAX

If we are creating filesystem on DAX capable device, warn if set block
size is incompatible with DAX to give admin some hint why DAX might not
be available.

Signed-off-by: Jan Kara <jack@suse.cz>
Signed-off-by: Theodore Ts'o <tytso@mit.edu>