logging - Regex for log fields not always in order -


i hope able me on headache. bit stuck on regex/grok/logstash syntax. trying parse kibana logs, fields in logs not comes in same order.

here 3 logs exemple.

{"name":"kibana","hostname":"22e1923b59f3","pid":1,"level":30,"req":{"method":"post","url":"/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1438549037831","headers":{"host":"192.168.33.10:5601","connection":"keep-alive","content-length":"779","accept":"application/json, text/plain, */*","origin":"http://192.168.33.10:5601","user-agent":"mozilla/5.0 (windows nt 6.3; wow64) applewebkit/537.36 (khtml, gecko) chrome/44.0.2403.125 safari/537.36","content-type":"application/json;charset=utf-8","referer":"http://192.168.33.10:5601/","accept-encoding":"gzip, deflate","accept-language":"en-us,en;q=0.8,fr;q=0.6","cookie":"phpsessid=nhbu44n9jce33312je8346gm43; fastcoinlang=en; csrftoken=rai1svrsagpme490nhobxx9m1wyfufjn"},"remoteaddress":"192.168.33.1","remoteport":56797},"res":{"statuscode":200,"responsetime":180,"contentlength":158203},"msg":"post /_msearch?timeout=0&ignore_unavailable=true&preference=1438549037831 200 - 180ms","time":"2015-08-02t21:02:50.359z","v":0}  {"name":"kibana","hostname":"22e1923b59f3","pid":1,"level":30,"req":{"method":"get","url":"/worker-json.js","headers":{"host":"192.168.33.10:5601","connection":"keep-alive","user-agent":"mozilla/5.0 (windows nt 6.3; wow64) applewebkit/537.36 (khtml, gecko) chrome/44.0.2403.125 safari/537.36","accept":"*/*","referer":"http://192.168.33.10:5601/","accept-encoding":"gzip, deflate, sdch","accept-language":"en-us,en;q=0.8,fr;q=0.6","cookie":"phpsessid=nhbu44n9jce33312je8346gm43; fastcoinlang=en; csrftoken=rai1svrsagpme490nhobxx9m1wyfufjn","if-none-match":"w/\"10ca1-14e40807fc0\"","if-modified-since":"mon, 29 jun 2015 18:07:20 gmt"},"remoteaddress":"192.168.33.1","remoteport":56797},"res":{"statuscode":304,"responsetime":1,"contentlength":0},"msg":"get /worker-json.js 304 - 1ms","time":"2015-08-02t21:02:59.102z","v":0}  {"name":"kibana","hostname":"a274fbd7485b","pid":1,"level":30,"req":{"method":"get","url":"/bower_components/font-awesome/fonts/fontawesome-webfont.woff?v=4.2.0","headers":{"host":"192.168.33.10:5601","connection":"keep-alive","cache-control":"max-age=0","if-none-match":"w/\"ffac-14e40807bd8\"","if-modified-since":"mon, 29 jun 2015 18:07:19 gmt","user-agent":"mozilla/5.0 (windows nt 6.3; wow64) applewebkit/537.36 (khtml, gecko) chrome/44.0.2403.125 safari/537.36","accept":"*/*","referer":"http://192.168.33.10:5601/","accept-encoding":"gzip, deflate, sdch","accept-language":"en-us,en;q=0.8,fr;q=0.6","cookie":"phpsessid=nhbu44n9jce33312je8346gm43; fastcoinlang=en; csrftoken=rai1svrsagpme490nhobxx9m1wyfufjn"},"remoteaddress":"192.168.33.1","remoteport":59999},"res":{"statuscode":304,"responsetime":2,"contentlength":0},"msg":"get /bower_components/font-awesome/fonts/fontawesome-webfont.woff?v=4.2.0 304 - 2ms","time":"2015-08-02t22:55:19.190z","v":0} 

what parse far:

{\"name\":%{qs:name},\"hostname\":%{qs:hostname},\"pid\":%{int:pid},\"level\":%{int:level},\"req\":{\"method\":%{qs:req_method},\"url\":%{qs:req_url},\"headers\":{\"host\":%{qs:headers_host},\"connection\":%{qs:headers_connection}(,\"content-length\":%{qs:headers_content_length}|)(,\"accept\":%{qs:headers_accept}|)(,\"cache-control\":%{qs:headers_cache_control}|)(,\"if-none-match\":%{qs:headers_if_none_match}|)(,\"if-modified-since\":%{qs:headers_if_modified_since}|)(,\"origin\":%{qs:headers_origin}|)(,\"user-agent\":%{qs:headers_user_agent}|)(,\"accept\":%{qs:headers_accept}|)(,\"content-type\":%{qs:headers_content_type}|)(,\"referer\":%{qs:headers_referer}|)(,\"accept-encoding\":%{qs:headers_accept_encoding}|)(,\"accept-language\":%{qs:headers_accept_language}|)(,\"cookie\":%{qs:headers_cookie}|)},\"remoteaddress\":%{qs:remoteaddress},\"remoteport\":%{int:remoteport}},\"res\":{\"statuscode\":%{int:res_statuscode},\"responsetime\":%{int:res_responsetime},\"contentlength\":%{int:res_contentlength}},\"msg\":%{qs:res_msg},\"time\":\"%{timestamp_iso8601:time}\",\"v\":%{int:v}} 

with can take care if field missing, not when 1 field comes before/after one. how capture without taking care of field order?

if have tips on how more effective syntax take too!.


Comments

Popular posts from this blog

Upgrade php version of xampp not success -

c - Bitwise operation with (signed) enum value -

xslt - Unnest parent nodes by child node -