logging - Regex for log fields not always in order -


i hope able me on headache. bit stuck on regex/grok/logstash syntax. trying parse kibana logs, fields in logs not comes in same order.

here 3 logs exemple.

{"name":"kibana","hostname":"22e1923b59f3","pid":1,"level":30,"req":{"method":"post","url":"/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1438549037831","headers":{"host":"192.168.33.10:5601","connection":"keep-alive","content-length":"779","accept":"application/json, text/plain, */*","origin":"http://192.168.33.10:5601","user-agent":"mozilla/5.0 (windows nt 6.3; wow64) applewebkit/537.36 (khtml, gecko) chrome/44.0.2403.125 safari/537.36","content-type":"application/json;charset=utf-8","referer":"http://192.168.33.10:5601/","accept-encoding":"gzip, deflate","accept-language":"en-us,en;q=0.8,fr;q=0.6","cookie":"phpsessid=nhbu44n9jce33312je8346gm43; fastcoinlang=en; csrftoken=rai1svrsagpme490nhobxx9m1wyfufjn"},"remoteaddress":"192.168.33.1","remoteport":56797},"res":{"statuscode":200,"responsetime":180,"contentlength":158203},"msg":"post /_msearch?timeout=0&ignore_unavailable=true&preference=1438549037831 200 - 180ms","time":"2015-08-02t21:02:50.359z","v":0}  {"name":"kibana","hostname":"22e1923b59f3","pid":1,"level":30,"req":{"method":"get","url":"/worker-json.js","headers":{"host":"192.168.33.10:5601","connection":"keep-alive","user-agent":"mozilla/5.0 (windows nt 6.3; wow64) applewebkit/537.36 (khtml, gecko) chrome/44.0.2403.125 safari/537.36","accept":"*/*","referer":"http://192.168.33.10:5601/","accept-encoding":"gzip, deflate, sdch","accept-language":"en-us,en;q=0.8,fr;q=0.6","cookie":"phpsessid=nhbu44n9jce33312je8346gm43; fastcoinlang=en; csrftoken=rai1svrsagpme490nhobxx9m1wyfufjn","if-none-match":"w/\"10ca1-14e40807fc0\"","if-modified-since":"mon, 29 jun 2015 18:07:20 gmt"},"remoteaddress":"192.168.33.1","remoteport":56797},"res":{"statuscode":304,"responsetime":1,"contentlength":0},"msg":"get /worker-json.js 304 - 1ms","time":"2015-08-02t21:02:59.102z","v":0}  {"name":"kibana","hostname":"a274fbd7485b","pid":1,"level":30,"req":{"method":"get","url":"/bower_components/font-awesome/fonts/fontawesome-webfont.woff?v=4.2.0","headers":{"host":"192.168.33.10:5601","connection":"keep-alive","cache-control":"max-age=0","if-none-match":"w/\"ffac-14e40807bd8\"","if-modified-since":"mon, 29 jun 2015 18:07:19 gmt","user-agent":"mozilla/5.0 (windows nt 6.3; wow64) applewebkit/537.36 (khtml, gecko) chrome/44.0.2403.125 safari/537.36","accept":"*/*","referer":"http://192.168.33.10:5601/","accept-encoding":"gzip, deflate, sdch","accept-language":"en-us,en;q=0.8,fr;q=0.6","cookie":"phpsessid=nhbu44n9jce33312je8346gm43; fastcoinlang=en; csrftoken=rai1svrsagpme490nhobxx9m1wyfufjn"},"remoteaddress":"192.168.33.1","remoteport":59999},"res":{"statuscode":304,"responsetime":2,"contentlength":0},"msg":"get /bower_components/font-awesome/fonts/fontawesome-webfont.woff?v=4.2.0 304 - 2ms","time":"2015-08-02t22:55:19.190z","v":0} 

what parse far:

{\"name\":%{qs:name},\"hostname\":%{qs:hostname},\"pid\":%{int:pid},\"level\":%{int:level},\"req\":{\"method\":%{qs:req_method},\"url\":%{qs:req_url},\"headers\":{\"host\":%{qs:headers_host},\"connection\":%{qs:headers_connection}(,\"content-length\":%{qs:headers_content_length}|)(,\"accept\":%{qs:headers_accept}|)(,\"cache-control\":%{qs:headers_cache_control}|)(,\"if-none-match\":%{qs:headers_if_none_match}|)(,\"if-modified-since\":%{qs:headers_if_modified_since}|)(,\"origin\":%{qs:headers_origin}|)(,\"user-agent\":%{qs:headers_user_agent}|)(,\"accept\":%{qs:headers_accept}|)(,\"content-type\":%{qs:headers_content_type}|)(,\"referer\":%{qs:headers_referer}|)(,\"accept-encoding\":%{qs:headers_accept_encoding}|)(,\"accept-language\":%{qs:headers_accept_language}|)(,\"cookie\":%{qs:headers_cookie}|)},\"remoteaddress\":%{qs:remoteaddress},\"remoteport\":%{int:remoteport}},\"res\":{\"statuscode\":%{int:res_statuscode},\"responsetime\":%{int:res_responsetime},\"contentlength\":%{int:res_contentlength}},\"msg\":%{qs:res_msg},\"time\":\"%{timestamp_iso8601:time}\",\"v\":%{int:v}} 

with can take care if field missing, not when 1 field comes before/after one. how capture without taking care of field order?

if have tips on how more effective syntax take too!.


Comments

Popular posts from this blog

c - Bitwise operation with (signed) enum value -

xslt - Unnest parent nodes by child node -

YouTubePlayerFragment cannot be cast to android.support.v4.app.Fragment -