How to sort big txt...
Clear all

How to sort big txt files in Linux?

Member Admin
Joined: 2 years ago
Posts: 31
Topic starter  

What is the fastest way I can do a sort on a 20-gigabyte file?

Should I split the file into smaller files (smaller than 2 GB), sort -u them, put them together, then split them again in a different size, sort -u again, etc.? Or are there any sorting scripts, programs that could handle files this big with my limited amount of RAM?

Topic Tags
Member Moderator
Joined: 2 years ago
Posts: 52

Using the sort command will probably be the fastest option.

sort --parallel=2 -u file-sorted.txt file.txt


Set the number of sorts run in parallel to n. By default, n is set to the number of available processors but limited to 8, as there are diminishing performance gains after that. Note also that using n threads increases the memory usage by a factor of log n.