https://unix.stackexchange.com/questions/769727/how-do-i-read-the-last-lines-of-a-huge-log-file

"I have a log of 55GB in size.

I tried:

cat logfile.log | tail

But this approach takes a lot of time. Is there any way to read huge
files faster or any other approach?"

:)

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal
Mastodon: @Kazinator@mstdn.ca

Kaz Kylheku <433-929-6894@kylheku.com> writes:

> https://unix.stackexchange.com/questions/769727/how-do-i-read-the-last-lines-of-a-huge-log-file
>
> "I have a log of 55GB in size.
>
> I tried:
>
> cat logfile.log | tail
>
> But this approach takes a lot of time. Is there any way to read huge
> files faster or any other approach?"
>
> :)

Nice. A quick test with a 500 MB file gives

real 0m0.296s
user 0m0.059s
sys 0m0.422s

for

time cat foo | tail

and

real 0m0.005s
user 0m0.001s
sys 0m0.008s

for

time tac foo | head

and

real 0m0.005s
user 0m0.005s
sys 0m0.000s

for

time tail foo

Best regards

Axel

On Tue, 5 Mar 2024, Kaz Kylheku wrote:

> https://unix.stackexchange.com/questions/769727/how-do-i-read-the-last-lines-of-a-huge-log-file
>
> "I have a log of 55GB in size.
>
> I tried:
>
> cat logfile.log | tail
>
> But this approach takes a lot of time. Is there any way to read huge
> files faster or any other approach?"
>
> :)

I would "chunk" it. When I was young I had to analyze several GB of logs
due to a security incident and what I did is to cut up the mega-file in
several chunks, and then do what ever I had to do to those chunks, and
then add the result to one file.

Don't know if that is possible or useful in your situation, but for me,
about 18 years ago, it worked beautifully.

Best regards,
Daniel

On 06.03.2024 10:42, D wrote:
>
>
> On Tue, 5 Mar 2024, Kaz Kylheku wrote:
>
>> https://unix.stackexchange.com/questions/769727/how-do-i-read-the-last-lines-of-a-huge-log-file
>>
>>
>> "I have a log of 55GB in size.
>>
>> I tried:
>>
>> cat logfile.log | tail
>>
>> But this approach takes a lot of time. Is there any way to read huge
>> files faster or any other approach?"
>>
>> :)
>
> I would "chunk" it. When I was young I had to analyze several GB of logs
> due to a security incident and what I did is to cut up the mega-file in
> several chunks, and then do what ever I had to do to those chunks, and
> then add the result to one file.
>
> Don't know if that is possible or useful in your situation, but for me,
> about 18 years ago, it worked beautifully.

The point here is that 'cat' reads the whole file and pipes all
to 'head'. While 'head' is capable of reading the file itself
if called as "tail logfile.log" and can thus directly address
the file from the rear. In other words, you don't need elaborate
sophisticated workarounds, just use tail in a straightforward way.
(Axel posted some timing numbers.)

Janis

>
> Best regards,
> Daniel
>

On 06.03.2024 20:33, Janis Papanagnou wrote:
> On 06.03.2024 10:42, D wrote:
>>
[...]
>>
>> I would "chunk" it. When I was young I had to analyze several GB of logs
>> due to a security incident and what I did is to cut up the mega-file in
>> several chunks, and then do what ever I had to do to those chunks, and
>> then add the result to one file.
>>
>> Don't know if that is possible or useful in your situation, but for me,
>> about 18 years ago, it worked beautifully.
>
> The point here is that 'cat' reads the whole file and pipes all
> to 'head'. While 'head' is capable of reading the file itself

to 'tail'. While 'tail' is capable of reading the file itself

> if called as "tail logfile.log" and can thus directly address
> the file from the rear. In other words, you don't need elaborate
> sophisticated workarounds, just use tail in a straightforward way.
> (Axel posted some timing numbers.)
>
> Janis
>
>>
>> Best regards,
>> Daniel
>>
>