How to sort big txt...
 
Share:
Notifications
Clear all

How to sort big txt files in Linux?


Neha
 Neha
(@asha)
Member Admin
Joined: 1 year ago
Posts: 31
Topic starter  

What is the fastest way I can do a sort on a 20-gigabyte file?

Should I split the file into smaller files (smaller than 2 GB), sort -u them, put them together, then split them again in a different size, sort -u again, etc.? Or are there any sorting scripts, programs that could handle files this big with my limited amount of RAM?


Quote
Topic Tags
myTechMint
(@mytechmint)
Member Moderator
Joined: 1 year ago
Posts: 52
 

Using the sort command will probably be the fastest option.

sort --parallel=2 -u file-sorted.txt file.txt

"--parallel=n"

Set the number of sorts run in parallel to n. By default, n is set to the number of available processors but limited to 8, as there are diminishing performance gains after that. Note also that using n threads increases the memory usage by a factor of log n.


ReplyQuote