Python 使用isin通过排除过滤PySpark dataframe

Python是一种面向对象的、动态语义的、高级的、解释性的编程语言。它可以快速开发应用程序，并且可以用作脚本语言或胶水语言，将现有的组件组合在一起，找到其高级内置的数据结构，再加上动态类型和动态绑定，使其特别具有吸引力。

PySpark Dataframe

在PySpark数据帧中，数据以命名列的形式组织，这些数据帧是可以在不同计算机上运行的分布式数据集。这些数据帧可以来自现有的弹性分布式数据集(RDD)、外部数据库或结构化数据文件。

语法 – Isin()

isin(list_)

list_a =参数将一个列的值作为值列表。

这用于检查或过滤DataFrame的值是否出现在值列表中，使用PySpark的isin()或IN操作符。

Column类的isin()函数返回一个布尔值。如果参数的评估值包括表达式的值，则返回True。

示例1

# Import the necessary libraries
from pyspark.sql.functions import col

# Create a PySpark DataFrame
data = [("Karthik", 25), ("Vijay", 30), ("Kruthik", 35), ("Sricharan", 40), ("Aditi", 45)]
df = spark.createDataFrame(data, ["name", "age"])

# Define the list of values to exclude
exclusion_list = [25, 30]

# Filter out rows where age is in the exclusion list
filtered_df = df.filter(~col("age").isin(exclusion_list))

# Display the filtered DataFrame
filtered_df.show()

输出

+------+---+
|  name|age|
+------+---+
|Kruthik| 35|
| Sricharan| 40|
|   Aditi| 45|
+------+---+

示例2

# Import the necessary libraries
from pyspark.sql.functions import col 
# Create a PySpark DataFrame
data = [("Karthik", "New York"), ("Vijay", "Chicago"), ("Kruthik", "San Francisco"), ("Sricharan", "Los Angeles"), ("Aditi", "Miami")]
df = spark.createDataFrame(data, ["name", "city"])
# Define the list of values to exclude
exclusion_list = ["New York", "Chicago", "Miami"] 
# Filter out rows where city is in the exclusion list
filtered_df = df.filter(~col("city").isin(exclusion_list))
# Display the filtered DataFrame
filtered_df.show()

输出

+------+--------------+
| name|     city|
+------+--------------+
|Sricharan |   Los Angeles|
|Kruthik|San Francisco|
+------+--------------+

创建用于演示的数据框

示例1

# Create a PySpark DataFrame
data = [("Alice", "New York"), ("Bob", "Chicago"), ("Charlie", "San Francisco"), ("David", 
"Los Angeles"), ("Eva", "Miami")]
df = spark.createDataFrame(data, ["name", "city"])
# Define the list of values to exclude
exclusion_list = ["New York", "Chicago", "Miami"]
# Filter out rows where city is in the exclusion list and name is not "David"
filtered_df = df.filter(~(col("city").isin(exclusion_list) & (col("name") != "David")))
# Display the filtered DataFrame
filtered_df.show()

输出

+------+--------------+
|  name|     city|
+------+--------------+
|David |   Los Angeles|
|Charlie|San Francisco|
+------+--------------+

示例2

# importing module
import pyspark
# importing sparksession from pyspark.sql module
from pyspark.sql import SparkSession
# creating sparksession and giving an app name
spark = SparkSession.builder.appName('sparkdf').getOrCreate()
# list of students data with null values
# we can define null values with none
data = [[1, "Karthik", "Sharma"],
            [2, "Kruthik", "Ballari"],
            [3, "Vijay", "Kumar"],
            [4, "Aditi", "Gupta"],
            [5, "Sricharan", "Sharma"]]
# specify column names
columns = ['ID', 'NAME', 'Lastname']
# creating a dataframe from the lists of data
dataframe = spark.createDataFrame(data, columns)
dataframe.show()

输出

+------+---------+----------------+
ID      NAME  LASTNAME
+------+---------+----------------+
1       |Karthik| Sharma |
2       |Kruthik| Ballari |
3       |Vijay| Kumar |
4       |Aditi| Gupta |
5       |Sricharan| Sharma |

示例3

下面的代码演示了从数据框的列中获取名称并显示它们。

filter() ：此子句用于检查条件并给出结果，两者相似

语法

dataframe.filter(condition)

# Getting Kruthik's name
dataframe.filter((dataframe.NAME).isin(['Kruthik'])).show()

输出

+------+---------+----------------+
ID      NAME  LASTNAME
+------+---------+----------------+
1       |Kruthik| Ballari |

示例4

以下程序演示了从姓氏为Sharma的数据框中获取数据并打印他们的全名。

where() ：此子句用于检查条件并给出结果。

语法

dataframe.where(condition)

# Fetching names of people whose last name is Sharma
dataframe.where((dataframe.college).isin(['Sharma'])).show()

输出

+------+---------+----------------+
ID      NAME  LASTNAME
+------+---------+----------------+
1        |Karthik| Sharma |
2        |Sricharan| Sharma |