gasmaskhome

Is it possible to add a new column based on the maximum of previous columns where the previous colu… Read more Mode Of Row As A New Column In Pyspark Dataframe

I'm trying to learn Spark following some hello-word level example such as below, using pyspark.… Read more Pyspark: Method Isbarrier([]) Does Not Exist

I have a few questions which I would like to clarify before installation. Please bear with me as I … Read more Installing Pyspark On Windows

I am trying to wrap the constructor for pyspark Pipeline.init constructor, and monkey patch in the … Read more Wrapping Pyspark Pipeline.__init__ And Decorators

I'm using the elasticsearch-py client within PySpark using Python 3 and I'm running into a … Read more Elasticsearch Analyze() Not Compatible With Spark In Python?

Running a Python Spark Application via API call - On submitting the Application - response - Faile… Read more Sparkcontext Error - File Not Found /tmp/spark-events Does Not Exist

I am trying to create a LogisticRegression model (LogisticRegressionWithSGD), but its getting an er… Read more Logistic Regression Pyspark Mllib Issue With Multiple Labels

I need to creeate an new Spark DF MapType Column based on the existing columns where column name is… Read more Pyspark: Create Maptype Column From Existing Columns

I am trying to install apache spark to run locally on my windows machine. I have followed all instr… Read more No Module Named 'resource' Installing Apache Spark On Windows

Hy, I'm trying build a recommendation system with Spark I have a data frame with users email an… Read more Pyspark Add New Column Field With The Data Frame Row Number

So what I am trying to do is simply to convert fields: year, month, day, hour, minute (which are of… Read more Unsupportedoperationexception: Cannot Evalute Expression: .. When Adding New Column Withcolumn() And Udf()

I have multiple CSV files in my folder myfolder.tar.gz. Which I created in this way: first put all … Read more Filtering Files Using Specific Pattern When Reading Tar.gz Archive In Pyspark

I am trying to read XML file: df = spark.read.format('com.databricks.spark.xml').load('… Read more Read Xml Using Pyspark In Jupyter Notebook

I am using Apache spark in batch mode. I have set up an entire pipeline that transforms text into T… Read more 'sparksession' Object Has No Attribute 'serializer' When Evaluating A Classifier In Pyspark

My problem is based on the similar question here PySpark: Add a new column with a tuple created fro… Read more Create A Tuple Out Of Two Columns - Pyspark

I asked the question a while back for python, but now I need to do the same thing in PySpark. I hav… Read more Pyspark Merge Multiple Columns Into A Json Column

I have an RDD structure like: rdd = [[[1],[2],[3]], [[4],[5]], [[6]], [[7],[8],[9],[10]]] and I wa… Read more How To Flatten Nested Lists In Pyspark?

I'm running a very simple Spark job on AWS EMR and can't seem to get any log output from my… Read more Aws Emr Spark Python Logging

I have a dataframe with a very large number of columns (>30000). I'm filling it with 1 and 0… Read more Efficient Column Processing In Pyspark

I wrote a pyspark script that reads two json files, coGroup them and sends the result to an elastic… Read more Pyspark: Ship Jar Dependency With Spark-submit