python - Creating a Parquet file with PySpark on an AWS EMR cluster -

-

i'm trying spin spark cluster datbricks' csv package can create parquet files , stuff spark obviously.

this being done within aws emr, don't think i'm putting these options in correct place.

this command want send cluster spins up: spark-shell --packages com.databricks:spark-csv_2.10:1.4.0 --master yarn --driver-memory 4g --executor-memory 2g. i've tried putting on spark step - correct?

if cluster spun without being installed, how start pyspark package? correct: pyspark --packages com.databricks:spark-csv_2.10:1.4.0? can't tell if installed or not. not sure functions test

and in regards using package, correct creating parquet file:

df = sqlcontext.read.format('com.databricks.spark.csv').options(header='false').load('s3n://bucketname/nation.tbl', schema = customschema)  #is option1 df.write.parquet("s3n://bucketname/nation_parquet.parquet")  #or option2 df.select('nation_id', 'name', 'some_int', 'comment').write.parquet('com.databricks.spark.csv').save('s3n://bucketname/nation_parquet.tbl')

i'm not able find recent (mid 2015 , later) documentation regarding writing parquet file.

edit:

okay, i'm not sure if i'm creating dataframe correctly. if try run select queries on , show resultset, don't , instead error. here's tried running:

df = sqlcontext.read.format('com.databricks.spark.csv').options(header='false').load('s3n://bucketname/nation.tbl', schema = customschema)  df.registertemptable("region2")  tcp_interactions = sqlcontext.sql(""" select nation_id, name, comment region2 nation_id > 1 """)  tcp_interactions.show() #get weird java error: #caused by: java.lang.numberformatexception: input string: "0|algeria|0| haggle. final deposits detect slyly agai|"


Comments

Post a Comment

Popular posts from this blog

javascript - How to get current YouTube IDs via iMacros? -

-
i want viewed youtube video id , save csv via imacros, gets nothing. here js code: var videoid; videoid ="code:"; videoid +="url goto=https://www.youtube.com/watch?v=dbnywxdz_pa"+"\n"; videoid +="add !extract {{!urlcurrent}}"+"\n"; videoid +="set !var1 eval(\"var s=\"{{!urlcurrent}}\"; s.match((?<=watch\\?v=|/videos/|embed\\/)[^#\\&\\?]*);s[0]; \")"+"\n"; videoid +="add !extract {{!var1}}"+"\n"; videoid +="saveas type=extract folder=* file=url.csv"+"\n"; try this: var videoid = "url goto=https://www.youtube.com/watch?v=dbnywxdz_pa" + "\n"; videoid += "set !extract {{!urlcurrent}}" + "\n"; videoid += 'set !extract eval("\'{{!extract}}\'.match(/v=(.{11})/)[1];")' + "\n"; videoid += "saveas type=extract folder=* file=url.csv" + "\n"; iimplay
Read more

c# - Maintaining a program folder in program files out of date? -

-
for program i'm writing in vs c# wpf need store user related information on user's computer. far knew has been done creating folder in c:\program files , adding whatever program related info folder in subfolders or whatever. after doing browsing came across lot of people saying method out of date because access may denied create folder there, works administrative accounts, etc.. 1 site suggested saving c:\users\username\appdata\roaming or c:\users\username\appdata\local. question best , date method saving program data users computer? also take @ msdn article . it uwp apps have general idea put if developing wpf app. you may need these folders: environment.specialfolder.applicationdata environment.specialfolder.localapplicationdata environment.specialfolder.commonapplicationdata
Read more

emulation - Android map show my location didn't work -

-
i have been struggle current location in google map, tried official map demo, not working also. shows getmylocation button on upper right, didn't show blue dot on map current location. here code , i'm using emulator target api 23, guess may emulator's problem since code download directly github. advice appreciated. [screenshot][1] package com.example.mapdemo; import com.google.android.gms.maps.googlemap; import com.google.android.gms.maps.googlemap.onmylocationbuttonclicklistener; import com.google.android.gms.maps.onmapreadycallback; import com.google.android.gms.maps.supportmapfragment; import android.manifest; import android.content.pm.packagemanager; import android.os.bundle; import android.support.annotation.nonnull; import android.support.v4.app.activitycompat; import android.support.v4.content.contextcompat; import android.support.v7.app.appcompatactivity; import android.widget.toast; /** * demo
Read more