python - Pyspark socket write error -

-

i'm trying read file(~600m csv file) pyspark. following error.

surprisingly same code works correctly scala.

i found issue page https://issues.apache.org/jira/browse/spark-12261 not work.

reading code:

import os pyspark import sparkcontext pyspark import sparkconf  datasetdir = 'd:\\datasets\\movielens\\ml-latest\\' ratingfile = 'ratings.csv'  conf = sparkconf().setappname("movie_recommendation-server").setmaster('local[2]') sc = sparkcontext(conf=conf)  ratingrdd = sc.textfile(os.path.join(datasetdir, ratingfile)) print(ratingrdd.take(1)[0])

i getting error: 

      16/04/25 09:00:04 error pythonrunner: python worker exited unexpectedly    (crashed)     java.net.socketexception: connection reset peer: socket write error     @ java.net.socketoutputstream.socketwrite0(native method)     @ java.net.socketoutputstream.socketwrite(socketoutputstream.java:109)     @ java.net.socketoutputstream.write(socketoutputstream.java:153)     @ java.io.bufferedoutputstream.flushbuffer(bufferedoutputstream.java:82)     @ java.io.bufferedoutputstream.write(bufferedoutputstream.java:126)     @ java.io.dataoutputstream.write(dataoutputstream.java:107)     @ java.io.filteroutputstream.write(filteroutputstream.java:97)     @ org.apache.spark.api.python.pythonrdd$.writeutf(pythonrdd.scala:622)     @ org.apache.spark.api.python.pythonrdd$.org$apache$spark$api$python$pythonrdd$$write$1(pythonrdd.scala:442)     @ org.apache.spark.api.python.pythonrdd$$anonfun$writeiteratortostream$1.apply(pythonrdd.scala:452)     @ org.apache.spark.api.python.pythonrdd$$anonfun$writeiteratortostream$1.apply(pythonrdd.scala:452)     @ scala.collection.iterator$class.foreach(iterator.scala:727)     @ scala.collection.abstractiterator.foreach(iterator.scala:1157)     @ org.apache.spark.api.python.pythonrdd$.writeiteratortostream(pythonrdd.scala:452)     @ org.apache.spark.api.python.pythonrunner$writerthread$$anonfun$run$3.apply(pythonrdd.scala:280)     @ org.apache.spark.util.utils$.loguncaughtexceptions(utils.scala:1765)     @ org.apache.spark.api.python.pythonrunner$writerthread.run(pythonrdd.scala:239)


Comments

Post a Comment

Popular posts from this blog

javascript - How to get current YouTube IDs via iMacros? -

-
i want viewed youtube video id , save csv via imacros, gets nothing. here js code: var videoid; videoid ="code:"; videoid +="url goto=https://www.youtube.com/watch?v=dbnywxdz_pa"+"\n"; videoid +="add !extract {{!urlcurrent}}"+"\n"; videoid +="set !var1 eval(\"var s=\"{{!urlcurrent}}\"; s.match((?<=watch\\?v=|/videos/|embed\\/)[^#\\&\\?]*);s[0]; \")"+"\n"; videoid +="add !extract {{!var1}}"+"\n"; videoid +="saveas type=extract folder=* file=url.csv"+"\n"; try this: var videoid = "url goto=https://www.youtube.com/watch?v=dbnywxdz_pa" + "\n"; videoid += "set !extract {{!urlcurrent}}" + "\n"; videoid += 'set !extract eval("\'{{!extract}}\'.match(/v=(.{11})/)[1];")' + "\n"; videoid += "saveas type=extract folder=* file=url.csv" + "\n"; iimplay
Read more

c# - Maintaining a program folder in program files out of date? -

-
for program i'm writing in vs c# wpf need store user related information on user's computer. far knew has been done creating folder in c:\program files , adding whatever program related info folder in subfolders or whatever. after doing browsing came across lot of people saying method out of date because access may denied create folder there, works administrative accounts, etc.. 1 site suggested saving c:\users\username\appdata\roaming or c:\users\username\appdata\local. question best , date method saving program data users computer? also take @ msdn article . it uwp apps have general idea put if developing wpf app. you may need these folders: environment.specialfolder.applicationdata environment.specialfolder.localapplicationdata environment.specialfolder.commonapplicationdata
Read more

emulation - Android map show my location didn't work -

-
i have been struggle current location in google map, tried official map demo, not working also. shows getmylocation button on upper right, didn't show blue dot on map current location. here code , i'm using emulator target api 23, guess may emulator's problem since code download directly github. advice appreciated. [screenshot][1] package com.example.mapdemo; import com.google.android.gms.maps.googlemap; import com.google.android.gms.maps.googlemap.onmylocationbuttonclicklistener; import com.google.android.gms.maps.onmapreadycallback; import com.google.android.gms.maps.supportmapfragment; import android.manifest; import android.content.pm.packagemanager; import android.os.bundle; import android.support.annotation.nonnull; import android.support.v4.app.activitycompat; import android.support.v4.content.contextcompat; import android.support.v7.app.appcompatactivity; import android.widget.toast; /** * demo
Read more