Issue
I am trying to extract the max value of a column "ID" in Spark DataFrame and to increment whenever an insert is performed.
I am able to print the max value but am not able to store it in variable
max_id = df.agg({"ID":"max"}).collect()[0]
print max_id["max(ID)"]
id2 = max_id["max{"ID"}]
throws error
SyntaxError: invalid syntax
How to extract the same into a variable and increment it?
Solution
I figured out how to extract the max value of a column as int.
maxid=int(df.describe("ID").filter("summary = 'max'").select("ID").collect()[0].asDict()['ID']
print maxid ---result :3
id2=maxid+1 print id2 ---result 4
Answered By - Tejas
0 comments:
Post a Comment
Note: Only a member of this blog may post a comment.