org.apache.thrift.transport.TTransportException error while Reading large JSON file in zeppelin scala -

-

i trying read large json file (1.5 gb) using zeppelin , scala.

zeppelin working on spark in local mode installed on ubuntu os on vm 10 gb ram. have alloted 8gb spark.executor.memory

my code below 

val inputfileweather="/home/shashi/incubator-zeppelin-master/data/ai/weather.json" val temp=sqlcontext.read.json(inputfileweather)

i getting following error

org.apache.thrift.transport.ttransportexception     @ org.apache.thrift.transport.tiostreamtransport.read(tiostreamtransport.java:132)     @ org.apache.thrift.transport.ttransport.readall(ttransport.java:86)     @ org.apache.thrift.protocol.tbinaryprotocol.readall(tbinaryprotocol.java:429)     @ org.apache.thrift.protocol.tbinaryprotocol.readi32(tbinaryprotocol.java:318)     @ org.apache.thrift.protocol.tbinaryprotocol.readmessagebegin(tbinaryprotocol.java:219)     @ org.apache.thrift.tserviceclient.receivebase(tserviceclient.java:69)     @ org.apache.zeppelin.interpreter.thrift.remoteinterpreterservice$client.recv_interpret(remoteinterpreterservice.java:241)     @ org.apache.zeppelin.interpreter.thrift.remoteinterpreterservice$client.interpret(remoteinterpreterservice.java:225)     @ org.apache.zeppelin.interpreter.remote.remoteinterpreter.interpret(remoteinterpreter.java:229)     @ org.apache.zeppelin.interpreter.lazyopeninterpreter.interpret(lazyopeninterpreter.java:93)     @ org.apache.zeppelin.notebook.paragraph.jobrun(paragraph.java:229)     @ org.apache.zeppelin.scheduler.job.run(job.java:171)     @ org.apache.zeppelin.scheduler.remotescheduler$jobrunner.run(remotescheduler.java:328)     @ java.util.concurrent.executors$runnableadapter.call(executors.java:471)     @ java.util.concurrent.futuretask.run(futuretask.java:262)     @ java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.access$201(scheduledthreadpoolexecutor.java:178)     @ java.util.concurrent.scheduledthreadpoolexecutor$scheduledfuturetask.run(scheduledthreadpoolexecutor.java:292)     @ java.util.concurrent.threadpoolexecutor.runworker(threadpoolexecutor.java:1145)     @ java.util.concurrent.threadpoolexecutor$worker.run(threadpoolexecutor.java:615)     @ java.lang.thread.run(thread.java:745)

the error got due problem in running spark interpreter, zeppelin not connect interpreter process.

you have check logs located in /path/to/zeppelin/logs/*.out know happening. perhaps in interpreter logs see oom.

i think 8gb executor memory on vm 10 gb unreasonable,(and how many executors starting?). have consider driver memeory well


Comments

Post a Comment

Popular posts from this blog

javascript - How to get current YouTube IDs via iMacros? -

-
i want viewed youtube video id , save csv via imacros, gets nothing. here js code: var videoid; videoid ="code:"; videoid +="url goto=https://www.youtube.com/watch?v=dbnywxdz_pa"+"\n"; videoid +="add !extract {{!urlcurrent}}"+"\n"; videoid +="set !var1 eval(\"var s=\"{{!urlcurrent}}\"; s.match((?<=watch\\?v=|/videos/|embed\\/)[^#\\&\\?]*);s[0]; \")"+"\n"; videoid +="add !extract {{!var1}}"+"\n"; videoid +="saveas type=extract folder=* file=url.csv"+"\n"; try this: var videoid = "url goto=https://www.youtube.com/watch?v=dbnywxdz_pa" + "\n"; videoid += "set !extract {{!urlcurrent}}" + "\n"; videoid += 'set !extract eval("\'{{!extract}}\'.match(/v=(.{11})/)[1];")' + "\n"; videoid += "saveas type=extract folder=* file=url.csv" + "\n"; iimplay
Read more

c# - Maintaining a program folder in program files out of date? -

-
for program i'm writing in vs c# wpf need store user related information on user's computer. far knew has been done creating folder in c:\program files , adding whatever program related info folder in subfolders or whatever. after doing browsing came across lot of people saying method out of date because access may denied create folder there, works administrative accounts, etc.. 1 site suggested saving c:\users\username\appdata\roaming or c:\users\username\appdata\local. question best , date method saving program data users computer? also take @ msdn article . it uwp apps have general idea put if developing wpf app. you may need these folders: environment.specialfolder.applicationdata environment.specialfolder.localapplicationdata environment.specialfolder.commonapplicationdata
Read more

emulation - Android map show my location didn't work -

-
i have been struggle current location in google map, tried official map demo, not working also. shows getmylocation button on upper right, didn't show blue dot on map current location. here code , i'm using emulator target api 23, guess may emulator's problem since code download directly github. advice appreciated. [screenshot][1] package com.example.mapdemo; import com.google.android.gms.maps.googlemap; import com.google.android.gms.maps.googlemap.onmylocationbuttonclicklistener; import com.google.android.gms.maps.onmapreadycallback; import com.google.android.gms.maps.supportmapfragment; import android.manifest; import android.content.pm.packagemanager; import android.os.bundle; import android.support.annotation.nonnull; import android.support.v4.app.activitycompat; import android.support.v4.content.contextcompat; import android.support.v7.app.appcompatactivity; import android.widget.toast; /** * demo
Read more