Re: Output File Append Line running slow on server 2016 as text file gets larger - Unipaas


Andreas Sedlmeier
 
Edited

Hi Peter,

There was a discussion about log compression and rotation not so long ago. Just install a good log rotator and the problem should be gone.

What I see more and more often, because it's so incredible fast, reliable, distributed ... is streaming log information to Apache Kafka. 300.000 messages (log entries) per second, would be no issue at all. Think about message queues as alternative to writing to a log file :) (Samples: https://www.devglan.com/apache-kafka/stream-log4j-logs-to-kafka , https://medium.com/oneclicklabs-io/streaming-spring-boot-application-logs-to-apache-kafka-elk-k-stack-part-2-f3ed18f64a31 , ...)

P.S./EDIT: The thread I was referencing is following: https://magicu-l.groups.io/g/main/message/188573

When I was a young and ambitioned Magic programnmer I also experimented with logging to Windows Event log and debug console, with good results. Why the file append actually gets that slow when the file grows, I cant tell, that should not be the case because append is just open+seek to end+write. Maybe Magic does some kind of processing. You can have a look with a file monitor (process monitor from sysinternals f.i.) to see what happens, ...

Andreas

Join main@magicu-l.groups.io to automatically receive all group messages.