windowSpec = W\
.partitionBy(F.col("pos"))\
.orderBy(
F.col("pos").desc(),
F.col("op_ts").desc(),
F.col("current_ts").desc()
)
df = df\
.withColumn(
"row_num",
F.row_number().over(windowSpec)
)\
.filter(F.col("row_num") > '1')
Preview:
downloadDownload PNG
downloadDownload JPEG
downloadDownload SVG
Tip: You can change the style, width & colours of the snippet with the inspect tool before clicking Download!
Click to optimize width for Twitter