logging - Regex for log fields not always in order -

-

i hope able me on headache. bit stuck on regex/grok/logstash syntax. trying parse kibana logs, fields in logs not comes in same order.

here 3 logs exemple.

{"name":"kibana","hostname":"22e1923b59f3","pid":1,"level":30,"req":{"method":"post","url":"/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1438549037831","headers":{"host":"192.168.33.10:5601","connection":"keep-alive","content-length":"779","accept":"application/json, text/plain, */*","origin":"http://192.168.33.10:5601","user-agent":"mozilla/5.0 (windows nt 6.3; wow64) applewebkit/537.36 (khtml, gecko) chrome/44.0.2403.125 safari/537.36","content-type":"application/json;charset=utf-8","referer":"http://192.168.33.10:5601/","accept-encoding":"gzip, deflate","accept-language":"en-us,en;q=0.8,fr;q=0.6","cookie":"phpsessid=nhbu44n9jce33312je8346gm43; fastcoinlang=en; csrftoken=rai1svrsagpme490nhobxx9m1wyfufjn"},"remoteaddress":"192.168.33.1","remoteport":56797},"res":{"statuscode":200,"responsetime":180,"contentlength":158203},"msg":"post /_msearch?timeout=0&ignore_unavailable=true&preference=1438549037831 200 - 180ms","time":"2015-08-02t21:02:50.359z","v":0}  {"name":"kibana","hostname":"22e1923b59f3","pid":1,"level":30,"req":{"method":"get","url":"/worker-json.js","headers":{"host":"192.168.33.10:5601","connection":"keep-alive","user-agent":"mozilla/5.0 (windows nt 6.3; wow64) applewebkit/537.36 (khtml, gecko) chrome/44.0.2403.125 safari/537.36","accept":"*/*","referer":"http://192.168.33.10:5601/","accept-encoding":"gzip, deflate, sdch","accept-language":"en-us,en;q=0.8,fr;q=0.6","cookie":"phpsessid=nhbu44n9jce33312je8346gm43; fastcoinlang=en; csrftoken=rai1svrsagpme490nhobxx9m1wyfufjn","if-none-match":"w/\"10ca1-14e40807fc0\"","if-modified-since":"mon, 29 jun 2015 18:07:20 gmt"},"remoteaddress":"192.168.33.1","remoteport":56797},"res":{"statuscode":304,"responsetime":1,"contentlength":0},"msg":"get /worker-json.js 304 - 1ms","time":"2015-08-02t21:02:59.102z","v":0}  {"name":"kibana","hostname":"a274fbd7485b","pid":1,"level":30,"req":{"method":"get","url":"/bower_components/font-awesome/fonts/fontawesome-webfont.woff?v=4.2.0","headers":{"host":"192.168.33.10:5601","connection":"keep-alive","cache-control":"max-age=0","if-none-match":"w/\"ffac-14e40807bd8\"","if-modified-since":"mon, 29 jun 2015 18:07:19 gmt","user-agent":"mozilla/5.0 (windows nt 6.3; wow64) applewebkit/537.36 (khtml, gecko) chrome/44.0.2403.125 safari/537.36","accept":"*/*","referer":"http://192.168.33.10:5601/","accept-encoding":"gzip, deflate, sdch","accept-language":"en-us,en;q=0.8,fr;q=0.6","cookie":"phpsessid=nhbu44n9jce33312je8346gm43; fastcoinlang=en; csrftoken=rai1svrsagpme490nhobxx9m1wyfufjn"},"remoteaddress":"192.168.33.1","remoteport":59999},"res":{"statuscode":304,"responsetime":2,"contentlength":0},"msg":"get /bower_components/font-awesome/fonts/fontawesome-webfont.woff?v=4.2.0 304 - 2ms","time":"2015-08-02t22:55:19.190z","v":0}

what parse far:

{\"name\":%{qs:name},\"hostname\":%{qs:hostname},\"pid\":%{int:pid},\"level\":%{int:level},\"req\":{\"method\":%{qs:req_method},\"url\":%{qs:req_url},\"headers\":{\"host\":%{qs:headers_host},\"connection\":%{qs:headers_connection}(,\"content-length\":%{qs:headers_content_length}|)(,\"accept\":%{qs:headers_accept}|)(,\"cache-control\":%{qs:headers_cache_control}|)(,\"if-none-match\":%{qs:headers_if_none_match}|)(,\"if-modified-since\":%{qs:headers_if_modified_since}|)(,\"origin\":%{qs:headers_origin}|)(,\"user-agent\":%{qs:headers_user_agent}|)(,\"accept\":%{qs:headers_accept}|)(,\"content-type\":%{qs:headers_content_type}|)(,\"referer\":%{qs:headers_referer}|)(,\"accept-encoding\":%{qs:headers_accept_encoding}|)(,\"accept-language\":%{qs:headers_accept_language}|)(,\"cookie\":%{qs:headers_cookie}|)},\"remoteaddress\":%{qs:remoteaddress},\"remoteport\":%{int:remoteport}},\"res\":{\"statuscode\":%{int:res_statuscode},\"responsetime\":%{int:res_responsetime},\"contentlength\":%{int:res_contentlength}},\"msg\":%{qs:res_msg},\"time\":\"%{timestamp_iso8601:time}\",\"v\":%{int:v}}

with can take care if field missing, not when 1 field comes before/after one. how capture without taking care of field order?

if have tips on how more effective syntax take too!.


Comments

Post a Comment

Popular posts from this blog

python - Healpy: From Data to Healpix map -

-
Image
i have data grid rows represent theta (0, pi) , columns represent phi (0, 2*pi) , f(theta,phi) density of dark matter @ location. wanted calculate power spectrum , have decided use healpy. what can not understand how format data healpy use. if provide code (in python obvious reasons) or point me tutorial, great! have tried hand @ doing following code: #grid dimensions nrows*ncols (subject change) theta = np.linspace(0, np.pi, num=grid.shape[0])[:, none] phi = np.linspace(0, 2*np.pi, num=grid.shape[1]) nside = 512 print "pixel area: %.2f square degrees" % hp.nside2pixarea(nside, degrees=true) pix = hp.ang2pix(nside, theta, phi) healpix_map = np.zeros(hp.nside2npix(nside), dtype=np.double) healpix_map[pix] = grid but, when try execute code power spectrum. specifically, : cl = hp.anafast(healpix_map[pix], lmax=1024) i error: typeerror: bad number of pixels if point me tutorial or edit code great. more specifications: data in 2d np array , can change numrows/...
Read more

c - Bitwise operation with (signed) enum value -

-
i using enumerator values flags: typedef enum { = 0x00, b = 0x01u, // u has no influence, expected c = 0x02u, // u has no influence, expected ... } enum_name; volatile unsigned char* reg = someaddress; *reg |= b; according misra-c:2004 bitwise operations shall not done signed type. unfortunately, compiler iar use signed int (or short or char) underlying type of enums, , option can find relates size, not signedness ("--enum-is-int"). according iar c/c++ development guide arm , pages 169 , 211, can define type of enum s if enable iar language extensions ( -e command-line option, or project > options > c/c++ compiler > language > allow iar extensions in ide). in particular, should define "sentinel" value, make sure compiler chooses correct type. prefers signed types, , uses smallest possible integer type, sentinel should largest positive integer corresponding unsigned integer type can describe. example, typedef enum ...
Read more

xslt - Unnest parent nodes by child node -

-
in xml there items 0-n attributes , item should copied each attribute new item 1 attribute. given xml this: <?xml version="1.0" encoding="utf-8" ?> <items> <item> <name>a</name> <attributes> <attribute> <key>attribute1</key> <value>1</value> </attribute> </attributes> </item> <item> <name>b</name> </item> <item> <name>c</name> <attributes> <attribute> <key>attribute1</key> <value>5</value> </attribute> <attribute> <key>attribute2</key> <value>2</value> </attribute> <attribute> <key>attri...
Read more