Reading multiple json files from Spark -

-

i have list of json files load in parallel.

i can't use read.json("*") cause files not in same folder , there no specific pattern can implement.

i've tried sc.parallelize(filelist).select(hivecontext.read.json) hive context, expected, doesn't exists in executor.

any ideas?

looks found solution:

val text sc.textfile("file1,file2....") val df = sqlcontext.read.json(text)

Comments

Post a Comment

Popular posts from this blog

javascript - Feed FileReader from server side files -

-
i´m starting customize/improve old audio editor project. can import audio tracks canvas via drag&drop computer. thing use audio tracks stored in server clicking on list of available tracks... instead of use <input type="file"> tags. how can read server side files filereader?ajax perhaps? in advance. this code file reader: player.prototype.loadfile = function(file, el) { //console.log(file); var reader = new filereader, filetypes = ['audio/mpeg', 'audio/mp3', 'audio/wave', 'audio/wav'], = this; if (filetypes.indexof(file.type) < 0) { throw('unsupported file format!'); } reader.onloadend = function(e) { if (e.target.readystate == filereader.done) { // done == 2 $('.progress').children().width('100%'); var onsuccess = function(audiobuffer) { $(el).trigger('audiee:fileloaded', [audiobuffer, file]); ...
Read more

c++ - What's the differece between of link to a dynamic file and as a input object? -

-
i use g++ link project (an executable mono_kitti) , project dependent on thirdparty library pangolin. when link action -lpangolin option: g++ -l../../lib -lorb_slam2 -lpangolin mono_kitti.o -o mono_kitti it returns: mono_kitti.o:(.data+0x0): undefined reference `vtable pangolin::handler' mono_kitti.o:(.data+0x8): undefined reference `vtable pangolin::handlerscroll' mono_kitti.o: in function `pangolin::handler::~handler()': mono_kitti.cc:(.text._zn8pangolin7handlerd2ev[_zn8pangolin7handlerd5ev]+0x13): undefined reference `vtable pangolin::handler' mono_kitti.o: in function `pangolin::handlerscroll::~handlerscroll()': mono_kitti.cc:(.text._zn8pangolin13handlerscrolld2ev[_zn8pangolin13handlerscrolld5ev]+0x13): undefined reference `vtable pangolin::handlerscroll' collect2: error: ld returned 1 exit status but when use command: g++ -l../../lib -lorb_slam2 mono_kitti.o /usr/local/lib/libpangolin.so -o mono_kitti it succeeded. but failed again when tr...
Read more

Android Unit Testing / Mockito: android.location.Location not mocked -

-
i trying learn basic junit , mockito testing on android. i'm trying write unit tests simple class handles finding user's location location services on behalf of activities need location information. i have been trying create "faked location" test with: @test public void testlocationreceived() throws exception { location fakelocation = new location(locationmanager.network_provider); fakelocation.setlongitude(100); fakelocation.setlatitude(-80); ... } but error: java.lang.runtimeexception: method setlongitude in android.location.location not mocked. i understand unit tests on android run on jvm, don't have access requires operating system / framework, 1 of cases well? if so, how tell classes can / can't used on jvm? do need instrumentation/device tests in addition jvm-based unit tests test 1 class? you should add following in build.gradle (app): testoptions { unittests.returndefaultvalues = true } more d...
Read more