web - How to make python webcrawler infinite and record link once -

-

with of thenewboston able create nice little web crawler in python. after watching videos played around , added couple of things it. i've tried make infinite in every link on every link every recorded, have failed in doing so. have problem of recording same link more once? how go fixing problem?

this code.

import requests bs4 import beautifulsoup  def spider(max_pages):     page = 1     while page <= max_pages:         url = ''         source_code = requests.get(url)         plain_text = source_code.text         soup = beautifulsoup(plain_text, "html.parser")         link in soup.findall("a"):             href = link.get("href")             title = link.get("title")             links = []             #print(href)             #print(title)             try:                 get_single_user_data(href)             except:                 pass         page += 1  def get_single_user_data(user_url):     source_code = requests.get(user_url)     plain_text = source_code.text     soup = beautifulsoup(plain_text, "html.parser")     #for item_name in soup.findall('span', {'id':'mm-saledscprc'}):     #   print(item_name.string)     link in soup.findall("a"):         href = link.get("href")         print(href)   spider(1)

i've tried make infinite in every link on every link every recorded

that's not going happen unless have decently sized datacentre. sake of it. need larger starting pool of websites crawl links other websites, , you'll far enough. start outbound links reddit or something.

i have problem of recording same link more once?

i recommend recording links have been using hash table record websites you've visited, , check if link there before visiting it.


Comments

Post a Comment

Popular posts from this blog

python - Healpy: From Data to Healpix map -

-
Image
i have data grid rows represent theta (0, pi) , columns represent phi (0, 2*pi) , f(theta,phi) density of dark matter @ location. wanted calculate power spectrum , have decided use healpy. what can not understand how format data healpy use. if provide code (in python obvious reasons) or point me tutorial, great! have tried hand @ doing following code: #grid dimensions nrows*ncols (subject change) theta = np.linspace(0, np.pi, num=grid.shape[0])[:, none] phi = np.linspace(0, 2*np.pi, num=grid.shape[1]) nside = 512 print "pixel area: %.2f square degrees" % hp.nside2pixarea(nside, degrees=true) pix = hp.ang2pix(nside, theta, phi) healpix_map = np.zeros(hp.nside2npix(nside), dtype=np.double) healpix_map[pix] = grid but, when try execute code power spectrum. specifically, : cl = hp.anafast(healpix_map[pix], lmax=1024) i error: typeerror: bad number of pixels if point me tutorial or edit code great. more specifications: data in 2d np array , can change numrows/...
Read more

c - Bitwise operation with (signed) enum value -

-
i using enumerator values flags: typedef enum { = 0x00, b = 0x01u, // u has no influence, expected c = 0x02u, // u has no influence, expected ... } enum_name; volatile unsigned char* reg = someaddress; *reg |= b; according misra-c:2004 bitwise operations shall not done signed type. unfortunately, compiler iar use signed int (or short or char) underlying type of enums, , option can find relates size, not signedness ("--enum-is-int"). according iar c/c++ development guide arm , pages 169 , 211, can define type of enum s if enable iar language extensions ( -e command-line option, or project > options > c/c++ compiler > language > allow iar extensions in ide). in particular, should define "sentinel" value, make sure compiler chooses correct type. prefers signed types, , uses smallest possible integer type, sentinel should largest positive integer corresponding unsigned integer type can describe. example, typedef enum ...
Read more

xslt - Unnest parent nodes by child node -

-
in xml there items 0-n attributes , item should copied each attribute new item 1 attribute. given xml this: <?xml version="1.0" encoding="utf-8" ?> <items> <item> <name>a</name> <attributes> <attribute> <key>attribute1</key> <value>1</value> </attribute> </attributes> </item> <item> <name>b</name> </item> <item> <name>c</name> <attributes> <attribute> <key>attribute1</key> <value>5</value> </attribute> <attribute> <key>attribute2</key> <value>2</value> </attribute> <attribute> <key>attri...
Read more