csv - Spark RDD External Storage -

-

i have written python code sum.py on summing numbers each csv file in directory data. going use apache-spark on amazon web service (aws), parallelize summation process each csv file. have done following steps:

  1. i've created 1 master , 2 slave nodes on aws.
  2. i used bash command $ scp -r -i my-key-pair.pem my_dir root@host_name upload directory my_dir onto aws cluster master node. folder my_dir contains 2 sub-directories: code , data, in which, code contains python code sum.py, , data contains csv files.
  3. i've login aws master node, , there used bash command $ ./spark/copy-dir /my_dir/code/ send code directory code contains sum.py slave nodes.
  4. on aws master node, i've put directory data containing csv files hdfs using $ ./ephemeral-hdfs/bin/hadoop fs -put /root/my_dir/data/.

now when submit application on aws master node: $ ./spark-submit ~/my_dir/code/sum.py, shows error worker node cannot find csv files. however, after send data directory data slave nodes using command copy-dir, works perfectly.

so confused problem. far know, driver program on master node loads csv files, creates rdd , sends separate tasks rdd each of slave nodes. means slave nodes don't need know original csv files, receive rdd master node. if true, why should send csv files each of slave node? also, if send csv files slave nodes, external disk storage on slave nodes gonna used. mean apache-spark costly tool parallel computing? appreciate if helps me on these 2 questions.

yes, have make data available nodes. however, each node try best load data concerned (its partition), , can tune level of parallelism best fit task. there many ways make data available nodes besides copying on each node's file system. consider using distributed file system, hdfs, or hosting files in accessible location each node, includes s3 or file server.


Comments

Post a Comment

Popular posts from this blog

python - Healpy: From Data to Healpix map -

-
Image
i have data grid rows represent theta (0, pi) , columns represent phi (0, 2*pi) , f(theta,phi) density of dark matter @ location. wanted calculate power spectrum , have decided use healpy. what can not understand how format data healpy use. if provide code (in python obvious reasons) or point me tutorial, great! have tried hand @ doing following code: #grid dimensions nrows*ncols (subject change) theta = np.linspace(0, np.pi, num=grid.shape[0])[:, none] phi = np.linspace(0, 2*np.pi, num=grid.shape[1]) nside = 512 print "pixel area: %.2f square degrees" % hp.nside2pixarea(nside, degrees=true) pix = hp.ang2pix(nside, theta, phi) healpix_map = np.zeros(hp.nside2npix(nside), dtype=np.double) healpix_map[pix] = grid but, when try execute code power spectrum. specifically, : cl = hp.anafast(healpix_map[pix], lmax=1024) i error: typeerror: bad number of pixels if point me tutorial or edit code great. more specifications: data in 2d np array , can change numrows/...
Read more

c - Bitwise operation with (signed) enum value -

-
i using enumerator values flags: typedef enum { = 0x00, b = 0x01u, // u has no influence, expected c = 0x02u, // u has no influence, expected ... } enum_name; volatile unsigned char* reg = someaddress; *reg |= b; according misra-c:2004 bitwise operations shall not done signed type. unfortunately, compiler iar use signed int (or short or char) underlying type of enums, , option can find relates size, not signedness ("--enum-is-int"). according iar c/c++ development guide arm , pages 169 , 211, can define type of enum s if enable iar language extensions ( -e command-line option, or project > options > c/c++ compiler > language > allow iar extensions in ide). in particular, should define "sentinel" value, make sure compiler chooses correct type. prefers signed types, , uses smallest possible integer type, sentinel should largest positive integer corresponding unsigned integer type can describe. example, typedef enum ...
Read more

xslt - Unnest parent nodes by child node -

-
in xml there items 0-n attributes , item should copied each attribute new item 1 attribute. given xml this: <?xml version="1.0" encoding="utf-8" ?> <items> <item> <name>a</name> <attributes> <attribute> <key>attribute1</key> <value>1</value> </attribute> </attributes> </item> <item> <name>b</name> </item> <item> <name>c</name> <attributes> <attribute> <key>attribute1</key> <value>5</value> </attribute> <attribute> <key>attribute2</key> <value>2</value> </attribute> <attribute> <key>attri...
Read more