Find the difference between the total number of CITY entries in the table and the number of distinct CITY entries in the table. The STATION table is described as follows:

Problem Difficulty Level : Easy

Data Structure

• ID
• City
• State
• Lattitude
• Longitude

Data for station table

In CSV Format

In Spark we will solve this problem using two ways

1. Using PySpark Functions
2. Using Spark SQL

Use below notebook for solution

Problem Solution First Part

In Postgre SQL We will load data from CSV using PostgreSQL Import functionality. And then we will solve this problem.

Output Query

Problem Solution

Please also follow below blog for understanding this problem