I suggest you ...

Fix issue with custom logs

Configured a script to tail a particular path in linux machine for aggregating the logs present in the file to Operations Management Suite.
I took this script from github,which actually aggregates the data faster by adding certain parameters for pushing the data to OMS.

<source>
type tail
path /var/log/sparkapp/yarn/sparkdriver.log
pos_file /var/opt/microsoft/omsagent/state/sparkappsdriverslog.pos
read_from_head true
run_interval 5s
tag oms.api.log_sparkappsdrivers
format multiline
format_firstline /\d{4}-\d{1,2}-\d{1,2}/
format1 /^(?<UtcTime>\d{4}-\d{1,2}-\d{1,2} \d{1,2}:\d{1,2}:\d{1,2},\d{1,3})\s+(?<LogLevel>[^\s]+)\s+(?<Message>.*)/
</source>

<filter oms.api.log_sparkappsdrivers>
type filter_hdinsight
</filter>

<match oms.api.log_sparkappsdrivers>
type out_oms_api
log_level info
num_threads 5
buffer_chunk_limit 10m
buffer_type file
buffer_path /var/opt/microsoft/omsagent/state/out_oms_sparkappsdriverslog.*.buffer
buffer_queue_limit 10
buffer_queue_full_action drop_oldest_chunk
flush_interval 20s
retry_limit 10
retry_wait 30s
max_retry_wait 9m
</match>

But the problem is that it worked for few days and then it stopped working.I guess this is because the custom logs in OMS is in still preview state.
Wish it can be resolved as soon as possible

1 vote
Vote
Sign in
Check!
(thinking…)
Reset
or sign in with
  • facebook
  • google
    Password icon
    I agree to the terms of service
    Signed in as (Sign out)
    You have left! (?) (thinking…)
    Anonymous shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

    0 comments

    Sign in
    Check!
    (thinking…)
    Reset
    or sign in with
    • facebook
    • google
      Password icon
      I agree to the terms of service
      Signed in as (Sign out)
      Submitting...

      Feedback and Knowledge Base