python 3.x - Poor predictive performance for RandomForest in Spark -

-

this might long shot, has run poor predictive performance using randomforest mllib? here i'm doing:

  • spark 1.4.1 pyspark
  • python 3.4.2
  • ~30,000 tweets of text
  • 12289 1s , 15956 0s
  • whitespace tokenization , hashing trick feature selection using 10,000 features
  • run rf 100 trees , maxdepth of 4 , predict using features 1s observations.

so in theory, should predictions of close 12289 1s (especially if model overfits). i'm getting 0 1s, sounds ludicrous me , makes me suspect wrong code or i'm missing something. notice similar behavior (although not extreme) if play around settings. i'm getting normal behavior other classifiers, don't think it's setup that's problem.

for example: 

>>> lrm = logisticregressionwithsgd.train(lp, iterations=10)  >>> logit_predict = lrm.predict(predict_feat)  >>> logit_predict.sum()  9077   >>> nb = naivebayes.train(lp)  >>> nb_predict = nb.predict(predict_feat)  >>> nb_predict.sum()  10287.0  >>> rf = randomforest.trainclassifier(lp, numclasses=2, categoricalfeaturesinfo={}, numtrees=100, seed=422)  >>> rf_predict = rf.predict(predict_feat)  >>> rf_predict.sum()  0.0

this code run back didn't change in between. have possible explanation this?


Comments

Post a Comment

Popular posts from this blog

c - Bitwise operation with (signed) enum value -

-
i using enumerator values flags: typedef enum { = 0x00, b = 0x01u, // u has no influence, expected c = 0x02u, // u has no influence, expected ... } enum_name; volatile unsigned char* reg = someaddress; *reg |= b; according misra-c:2004 bitwise operations shall not done signed type. unfortunately, compiler iar use signed int (or short or char) underlying type of enums, , option can find relates size, not signedness ("--enum-is-int"). according iar c/c++ development guide arm , pages 169 , 211, can define type of enum s if enable iar language extensions ( -e command-line option, or project > options > c/c++ compiler > language > allow iar extensions in ide). in particular, should define "sentinel" value, make sure compiler chooses correct type. prefers signed types, , uses smallest possible integer type, sentinel should largest positive integer corresponding unsigned integer type can describe. example, typedef enum
Read more

xslt - Unnest parent nodes by child node -

-
in xml there items 0-n attributes , item should copied each attribute new item 1 attribute. given xml this: <?xml version="1.0" encoding="utf-8" ?> <items> <item> <name>a</name> <attributes> <attribute> <key>attribute1</key> <value>1</value> </attribute> </attributes> </item> <item> <name>b</name> </item> <item> <name>c</name> <attributes> <attribute> <key>attribute1</key> <value>5</value> </attribute> <attribute> <key>attribute2</key> <value>2</value> </attribute> <attribute> <key>attri
Read more

YouTubePlayerFragment cannot be cast to android.support.v4.app.Fragment -

-
im stuck on 3 days , have tried have come across after lots of searching ,this post last resort, im trying open youtubeplayerfragment using navigation drawer youtubeplayerfragment cannot cast android.support.v4.app.fragment problem edit question : have updated code have used youtubeplayersupportfragment instead of youtubeplayerfragment ,i error in displayview method (case 4: fragment = new youtube(); break;) in main activity saying cannot convert youtube fragment , cannot logcat since cant run it edit question 2: code updated again have runtime problem ,heres logcat 07-22 22:15:09.782: e/androidruntime(20603): fatal exception: main 07-22 22:15:09.782: e/androidruntime(20603): process: net.frankandwalters, pid: 20603 07-22 22:15:09.782: e/androidruntime(20603): java.lang.nullpointerexception: attempt invoke virtual method 'void com.google.android.youtube.player.youtubeplayerview.a()' on null object reference 07-22 22:15:09.782: e/androidruntime(20603): @ com.goog
Read more