gasmaskhome

Is it possible to add a new column based on the maximum of previous columns where the previous colu… Read more Mode Of Row As A New Column In Pyspark Dataframe

Is there any way to use custom record delimiters while reading a csv file in pyspark. In my file re… Read more Read A File In Pyspark With Custom Column And Record Delmiter

I am using Apache spark in batch mode. I have set up an entire pipeline that transforms text into T… Read more 'sparksession' Object Has No Attribute 'serializer' When Evaluating A Classifier In Pyspark

I have a dataframe with a very large number of columns (>30000). I'm filling it with 1 and 0… Read more Efficient Column Processing In Pyspark

I'm stuck in a process where I need to perform some action for each column value in my Datafram… Read more Pyspark Best Alternative For Using Spark Sql/df Withing A Udf?

How to create a new column in PySpark and fill this column with the date of today? This is what I t… Read more Assertionerror: Col Should Be Column

I have two dataframe which has been readed from two csv files. +---+----------+-----------------+ |… Read more Joining Dataframes With Same Coumn Name In Pyspark

I'm attempting to compare one row in a dataframe with the next to see the difference in timesta… Read more Pyspark, Compare Two Rows In Dataframe

How to parse JSON string of nested lists to spark data frame in pyspark ? Input data frame: +------… Read more Spark: How To Parse Json String Of Nested Lists To Spark Data Frame?

I have a DataFrame: +-----+--------+---------+ | usn|log_type|item_code| +-----+--------+--------… Read more How To Make An Integer Index Row?

I am trying to measure the performance impact on having to copy a dataframe from scala to python an… Read more Pyspark. Transformer That Generates A Random Number Generates Always The Same Number

I am having some troubles reading the following CSV data in UTF-16: FullName, FullLabel, Type TEST.… Read more How To Read Csv File With Additional Comma In Quotes Using Pyspark?

I have the following CSV (sample) id timestamp routeid creationdate parameter… Read more Converting Complex Rdd To A Flatten Rdd With Pyspark

I have the below code. I am using pyspark 1.2.1 with python 2.7 (cpython) for colname in shuffle_co… Read more Improve Parallelism In Spark Sql

I'm currently working with Spark 2.1 and have a main script that calls a helper module that con… Read more Pyspark 2.1: Importing Module With Udf's Breaks Hive Connectivity

I don't know why I receive the message WARN KMeans: The input data is not directly cached, whi… Read more Pyspark 2: Kmeans The Input Data Is Not Directly Cached

I have a StructField in a dataframe that is not nullable. Simple example: import pyspark.sql.functi… Read more Can I Change The Nullability Of A Column In My Spark Dataframe?

I have searched online and the solutions provided online didn't resolve my issue. I am trying t… Read more How To Read Parquet Files Under A Directory Using Pyspark?

In PySpark, I want to calculate the correlation between two dataframe vectors, using the following … Read more Attributeerror: 'nonetype' Object Has No Attribute 'setcallsite'