python - Unable to scrape news headings from Hacker news -

-

i want scrape top news article's headline , link hacker news.

here code:

import scrapy scrapy.contrib.linkextractors.sgml import sgmllinkextractor  class hnitem(scrapy.item):     title=scrapy.field()     link=scrapy.field()  class hnspider(scrapy.spider):     name="hn"     allowed_domains=["https://news.ycombinator.com"]     start_urls=["https://news.ycombinator.com/"]     def parse(self,response):         item=hnitem()         item['title'] = response.xpath('//*[@id="hnmain"]/tbody/tr[3]/td/table/tbody/tr[1]/td[3]/a/text()').extract()         item['link'] = response.xpath('//*[@id="hnmain"]/tbody/tr[3]/td/table/tbody/tr[1]/td[3]/a/@href').extract()         print item['title']         print item['link']

but returns empty list.

p.s. beginner in python , in scrapy.

here ended when tried creating spider:

import scrapy  class hnitem(scrapy.item):     title = scrapy.field()     link = scrapy.field()  class hnspider(scrapy.spider):     name = 'hackernews'     allowed_domains = ['news.ycombinator.com'] # see javier's comment     start_urls = ['http://news.ycombinator.com/']      def parse(self,response):         sel = scrapy.selector(response)         item=hnitem()          # these xpaths can made more generic         item['title'] = sel.xpath("//tr[@class='athing']/td[3]/a[@href]/text()").extract()         item['link'] = sel.xpath("//tr[@class='athing']/td[3]/a/@href").extract()          # whatever want item. print,return, etc..         print item['title']         print item['link']

you can run command line with: scrapy runspider path/to/your_spider.py


Comments

Post a Comment

Popular posts from this blog

c - Bitwise operation with (signed) enum value -

-
i using enumerator values flags: typedef enum { = 0x00, b = 0x01u, // u has no influence, expected c = 0x02u, // u has no influence, expected ... } enum_name; volatile unsigned char* reg = someaddress; *reg |= b; according misra-c:2004 bitwise operations shall not done signed type. unfortunately, compiler iar use signed int (or short or char) underlying type of enums, , option can find relates size, not signedness ("--enum-is-int"). according iar c/c++ development guide arm , pages 169 , 211, can define type of enum s if enable iar language extensions ( -e command-line option, or project > options > c/c++ compiler > language > allow iar extensions in ide). in particular, should define "sentinel" value, make sure compiler chooses correct type. prefers signed types, , uses smallest possible integer type, sentinel should largest positive integer corresponding unsigned integer type can describe. example, typedef enum
Read more

xslt - Unnest parent nodes by child node -

-
in xml there items 0-n attributes , item should copied each attribute new item 1 attribute. given xml this: <?xml version="1.0" encoding="utf-8" ?> <items> <item> <name>a</name> <attributes> <attribute> <key>attribute1</key> <value>1</value> </attribute> </attributes> </item> <item> <name>b</name> </item> <item> <name>c</name> <attributes> <attribute> <key>attribute1</key> <value>5</value> </attribute> <attribute> <key>attribute2</key> <value>2</value> </attribute> <attribute> <key>attri
Read more

YouTubePlayerFragment cannot be cast to android.support.v4.app.Fragment -

-
im stuck on 3 days , have tried have come across after lots of searching ,this post last resort, im trying open youtubeplayerfragment using navigation drawer youtubeplayerfragment cannot cast android.support.v4.app.fragment problem edit question : have updated code have used youtubeplayersupportfragment instead of youtubeplayerfragment ,i error in displayview method (case 4: fragment = new youtube(); break;) in main activity saying cannot convert youtube fragment , cannot logcat since cant run it edit question 2: code updated again have runtime problem ,heres logcat 07-22 22:15:09.782: e/androidruntime(20603): fatal exception: main 07-22 22:15:09.782: e/androidruntime(20603): process: net.frankandwalters, pid: 20603 07-22 22:15:09.782: e/androidruntime(20603): java.lang.nullpointerexception: attempt invoke virtual method 'void com.google.android.youtube.player.youtubeplayerview.a()' on null object reference 07-22 22:15:09.782: e/androidruntime(20603): @ com.goog
Read more