Blog on Data Patterns

From Pandas to Polars: A Paradigm Shift in DataFrame Processing

Wed, 15 Jan 2025 00:00:00 +0530

Welcome to the first installment of our Polars blog series! If you’ve spent years mastering Pandas and are curious about what makes Polars the talk of the data community, this post is your gateway. We’ll explore the fundamental differences between these libraries, understand the core concepts unique to Polars, and get you productive with minimal friction.

The bottom line is straightforward: Polars is not just a faster Pandas—it’s a fundamentally different approach to DataFrame operations that prioritizes performance, parallelism, and a declarative expression-based paradigm.

Advanced Pandas GroupBy and Window Functions for Data Engineering and Analytics

Thu, 18 Apr 2024 00:00:00 +0530

Session Outline: Advanced GroupBy and Window Functions in Pandas

1. Introduction to GroupBy and Its Role in Data Engineering (5 minutes)

Overview: Recap of GroupBy as a split-apply-combine paradigm, as defined by Wes McKinney in Python for Data Analysis.
Data Engineering Context: Why GroupBy is critical for ETL pipelines, data aggregation, and analytics workflows.
Key Insight from Experts:
- Wes McKinney emphasizes GroupBy for flexible data aggregation and transformation.
- Matt Harrison advocates for chaining operations and leveraging GroupBy for concise, readable code (Effective Pandas).
Use cases:
- Aggregating transactional data for reporting.
- Feature engineering for machine learning.
- Time-series analysis and cohort analysis.

2. Core GroupBy Operations and Best Practices (10 minutes)

Basic syntax: df.groupby('column')
Aggregation functions: mean(), sum(), count(), min(), max(), std().
Grouping by single/multiple columns, index levels, and functions.
Best Practices (Matt Harrison):
- Use method chaining for clarity.
- Leverage agg() for multiple aggregations.
- Understand the difference between transform() and apply().

3. Advanced GroupBy Techniques (15 minutes)

Custom Aggregation Functions:

def custom_agg(x):
    return pd.Series({
        'mean': x.mean(),
        'std': x.std(),
        'count': x.count()
    })

df.groupby('group').agg(custom_agg)

Filtering Groups: filter()
Transformation: transform() for group-wise operations that return data aligned with the original DataFrame.
Apply: apply() for more complex operations.
Resampling Time Series Data:
```
df.resample('M').mean()
```

4. Window Functions: Rolling, Expanding, and EWM (15 minutes)

Rolling Windows:

df['rolling_avg'] = df['value'].rolling(window=3).mean()

Expanding Windows:

df['expanding_sum'] = df['value'].expanding().sum()

Exponentially Weighted Windows:

df['ewm'] = df['value'].ewm(span=3).mean()

Custom Window Functions:

def custom_window(x):
    return x[-1] - x[0]

df['window_diff'] = df['value'].rolling(window=3).apply(custom_window)

5. Performance Considerations (5 minutes)

Vectorized Operations: Always prefer built-in methods over apply when possible.
Categorical Data: Convert string columns to categories for memory efficiency.
Numba and Cython: For performance-critical sections.
Dask: For out-of-core computations on large datasets.

6. Real-world Examples (10 minutes)

Customer Segmentation:

customer_stats = df.groupby('customer_id').agg({
    'purchase_amount': ['sum', 'mean', 'count'],
    'purchase_date': ['min', 'max']
})

Time-series Analysis:

# 7-day rolling average
df['7d_avg'] = df.groupby('product_id')['sales'].transform(
    lambda x: x.rolling('7D').mean()
)

Cohort Analysis:

cohort = df.groupby(['cohort', 'period']).agg({
    'user_id': 'nunique',
    'revenue': 'sum'
})

7. Common Pitfalls and How to Avoid Them (5 minutes)

Setting with Copy Warning: Use .copy() when creating new DataFrames.
Memory Usage: Be mindful of group sizes.
Performance Bottlenecks: Profile your code with %timeit.
Missing Data: Handle NaN values before grouping.

8. Hands-on Exercise (20 minutes)

Dataset: Sample sales data with date, product, region, and sales amount.
Tasks:
1. Calculate total sales by region and product.
2. Find the 7-day rolling average of sales.
3. Identify the top-performing product in each region.
4. Calculate month-over-month growth rate.

9. Q&A and Wrap-up (5 minutes)

Key Takeaways:
- GroupBy is a powerful tool for data aggregation and transformation.
- Window functions enable sophisticated time-series analysis.
- Always consider performance implications.
Resources:
- Python for Data Analysis by Wes McKinney
- Effective Pandas by Matt Harrison
- Pandas documentation on GroupBy and Window Functions

Example Code Snippets

# Sample data
data = {
    'Date': pd.date_range(start='2023-01-01', periods=100, freq='D'),
    'Product': np.random.choice(['A', 'B', 'C'], 100),
    'Region': np.random.choice(['North', 'South', 'East', 'West'], 100),
    'Sales': np.random.randint(100, 1000, 100)
}
df = pd.DataFrame(data)

# Group by multiple columns
grouped = df.groupby(['Region', 'Product'])['Sales'].agg(['sum', 'mean', 'count'])

# Rolling window calculation
df['Rolling_7D'] = df.groupby('Product')['Sales'].transform(
    lambda x: x.rolling('7D').mean()
)

# Pivot table for visualization
pivot = pd.pivot_table(
    df, 
    values='Sales', 
    index='Date', 
    columns='Region', 
    aggfunc='sum'
)

This session will provide attendees with practical, hands-on experience with advanced Pandas operations, equipping them with the skills needed to tackle complex data manipulation tasks in their projects.

Time Intelligence in PowerBI

Sun, 01 Nov 2020 00:00:00 +0000

Data Analysis Expressions (DAX) includes time-intelligence functions that enable you to manipulate data using time periods, including days, months, quarters, and years, and then build and compare calculations over those periods.

Data Model

The date table can be created using the following DAX:

DateTable

DateTable = 
ADDCOLUMNS(
    CALENDAR(EOMONTH(min(Data_Blog[SalesDate]),-1), EOMONTH(max(Data_Blog[SalesDate]),3)),
    "Start of Month", EOMONTH([Date],-1)+1,
    "Year", Year([Date]),
    "Quarter",CONCATENATE("QTR ",QUARTER([Date])),
    "MonthNumber", Month([Date]),
    "Month", FORMAT([Date], "MMMM"),
    "Day", DAY ( [Date] )
)

Measures

Sales - Sum

= CALCULATE(sum(Data_Blog[Sales]))

Sales_MoM - Previous Month Sales

var _prevmonthsales = CALCULATE([Sales],DATEADD(DateTable[Date],-1,MONTH))  
return [Sales] - _prevmonthsales

Sales_MoM % - Percentage change with Previous Month Sales

var _prevmonthsales = CALCULATE([Sales],DATEADD(DateTable[Date],-1,MONTH))  
return DIVIDE([Sales] - _prevmonthsales,_prevmonthsales)

Sales_MovingAverage - Moving Average of last three months

[Quick Measure - Rolling Average]

IF(
	ISFILTERED('Data_Blog'[SalesDate]),
	ERROR("Time intelligence quick measures can only be grouped or filtered by the Power BI-provided date hierarchy or primary date column."),
	VAR __LAST_DATE = ENDOFMONTH(DateTable[Date])
	VAR __DATE_PERIOD =
		DATESBETWEEN(
			DateTable[Date],
			STARTOFMONTH(DATEADD(__LAST_DATE, -2, MONTH)),
			__LAST_DATE
		)
	RETURN
		AVERAGEX(
			CALCULATETABLE(
				SUMMARIZE(
					VALUES('Data_Blog'),
					DateTable[Year],
					DateTable[Quarter] ,
					DateTable[MonthNumber],
					DateTable[Month]
				),
				__DATE_PERIOD
			),
			CALCULATE(SUM('Data_Blog'[Sales]), ALL(DateTable[Date]))
		)
)

Sales_PreviousMonth - Parallel Period - Last Month

= CALCULATE([Sales],PARALLELPERIOD(DateTable[Date],-1,MONTH))

Sales_PreviousYear - Previous Year Sum

= CALCULATE([Sales],PREVIOUSYEAR(DateTable[Date]))

Sales_QTD - Quarter Till Date - Running Total

= TOTALQTD([Sales],DateTable[Date])

Sales_SamePeriodLastYear - Same Period Last Year

= CALCULATE([Sales],SAMEPERIODLASTYEAR(DateTable[Date]))

Sales_YTD - Year Till Date - Running Total

= TOTALYTD([Sales],'DateTable'[Date])

Handle multiple datetime formats representation using lubridate

Sun, 26 Apr 2020 00:00:00 +0000

Handling Heterogeneous datetime formats

We`ll be using parse_date_time function of lubridate package.
Multple datetime formats can be passed under orders attribute of the above function as shown below -

library(lubridate)
dates <- c( "1/22/2020 17:00", "2020-02-02 09:43:01", "2/1/2020 11:53")
parse_date_time(dates, orders = c('%Y-%m-%d %H:%M;%S','%m/%d/%Y %H:%M', '%m/%d/%y %H:%M'))

Output -

## [1] "2020-01-22 17:00:00 UTC" "2020-02-02 09:43:01 UTC"
## [3] "2020-02-01 11:53:00 UTC"

Download Data from a github folder using httr package in R

Fri, 24 Apr 2020 00:00:00 +0000

Import Libraries and Request to the master branch of git

The following chunk of code, import required libraries and request the master branch of the required git repository.

library(httr)
library(tidyverse)

req <- GET("https://api.github.com/repos/CSSEGISandData/COVID-19/git/trees/master?recursive=1")
req

Output -

## Response [https://api.github.com/repos/CSSEGISandData/COVID-19/git/trees/master?recursive=1]
##   Date: 2020-04-24 08:14
##   Status: 200
##   Content-Type: application/json; charset=utf-8
##   Size: 88.9 kB
## {
##   "sha": "###############################",
##   "url": "https://api.github.com/repos/CSSEGISandData/COVID-19/git/trees/65ba...
##   "tree": [
##     {
##       "path": ".gitignore",  
##       "mode": "100644",
##       "type": "blob",
##       "sha": "##############################",
##       "size": 9,
## ...

Extracting the path variable from the output of above request

This chunk access the path variable which stores all the paths associated in this master brnanch

Hackerrank SQL - Advanced Select

Sun, 01 Dec 2019 00:00:00 +0000

Following problems are discussed here. To solve the same, please visit hackerrank website{:target="_blank"}.

New Companies

select 
company.company_code,
company.founder,
count(distinct lead_manager.lead_manager_code),
count(distinct senior_manager.senior_manager_code), 
count(distinct manager.manager_code),
count(distinct employee.employee_code)
from company
left join 
    lead_manager on company.company_code = lead_manager.company_code
left join 
    senior_manager on company.company_code = senior_manager.company_code and
    lead_manager.lead_manager_code = senior_manager.lead_manager_code
left join 
    manager on company.company_code = manager.company_code and
    lead_manager.lead_manager_code = manager.lead_manager_code and 
    senior_manager.senior_manager_code = manager.senior_manager_code
left join 
    employee on company.company_code = employee.company_code and
    lead_manager.lead_manager_code = employee.lead_manager_code and 
    senior_manager.senior_manager_code = employee.senior_manager_code and 
    manager.manager_code = employee.manager_code
group by company.company_code, company.founder
order by company.company_code

Binary Tree Nodes

select N,
CASE
    WHEN N in (select N from BST where P is NUll) then "Root"
    WHEN N in (select P from BST) then "Inner"
    Else "Leaf"
end as flag 
from BST
order by N

Occupations


select doc_name,prof_name,singer_name,actor_name from(
(select 
 (row_number() over (order by name)) as doc_rank, 
 name as doc_name 
 from occupations 
 where occupation = "Doctor"
) as t1
full outer join 
(select 
 (row_number() over (order by name)) as prof_rank, 
 name as prof_name 
 from occupations 
 where occupation = "Professor") as t2
on t1.doc_rank = t2.prof_rank
full outer join (
    select 
 (row_number() over (order by name)) as actor_rank, 
 name as actor_name 
 from occupations 
 where occupation = "Actor") as t3
on t2.prof_rank = t3.actor_rank
full outer join (
    select 
 (row_number() over (order by name)) as singer_rank, 
 name as singer_name 
 from occupations 
 where occupation = "Singer") as t4
on t3.actor_rank = t4.singer_rank)

The PADS


select name+"("+left(occupation,1)+")" from occupations order by name, occupation

select "There are a total of "+cast(count(name) as varchar )+" "+lower(occupation)+"s." from occupations group by occupation 
order by count(name), occupation

Type of Triangle

select 
case 
    when a+b > c and b+c > a and c+a > b then 
        case 
            when a = b and b = c then 'Equilateral' 
            when a = b or b = c or a = c then 'Isosceles' 
            else 'Scalene' 
        end
    else 'Not A Triangle' 
end 
from triangles;

Hackerrank SQL - Aggregation

Sun, 01 Dec 2019 00:00:00 +0000

Following problems are discussed here. To solve the same, please visit hackerrank website{:target="_blank"}.

Weather Observation Station 17

select cast(Long_w as decimal(9,4)) from station where lat_n = (select min(Lat_n)  from station where Lat_n  > 38.7780)

Weather Observation Station 18

select  cast((max(long_w) - min(long_W) + max(lat_n) - min(lat_n)) as decimal(9,4)) from station

Weather Observation Station 19

select  
cast
(
    sqrt
    (
        (square
            (max(long_w) - min(long_W)) 
        + 
        (square
            (max(lat_n) - min(lat_n)))
        )
    )
    as decimal(9,4)
) from station

Weather Observation Station 20

with rowtabel  as (select lat_n, (row_number() over (order by lat_n)) as rn from station)
select cast(lat_n as decimal(9,4)) from rowtabel where  rn in (select (count(lat_n)/2)+1 from station)

List, Create and Move Folder within Azure Datalake Store Gen1 using R and Rest API

Sun, 15 Sep 2019 00:00:00 +0000

The following code snippets are on creating a connection to Azure Data Lake Storage Gen1 using R with Service-to-Service authentication with client secret and client id using REST API to list, create and move any folder in a azure data lake store gen1.

LISTSTATUS - To list everything in a folder
MKDIRS - To create new folder
RENAME - To move a folder to new location

Import Prerequisite and Authenticate

library(tidyverse)
library(httr)
library(curl)
library(jsonlite)

authentication_token <- function(tenant, client_id, client_secret){
  h <- new_handle()
  handle_setform(h,
                 "grant_type"="client_credentials",
                 "resource"="https://management.core.windows.net/",
                 "client_id" = client_id,
                 "client_secret" = client_secret
  )
  path = stringr::str_interp("https://login.windows.net/${tenant}/oauth2/token")
  req <- curl_fetch_memory(path, handle = h)
  res <- fromJSON(rawToChar(req$content))
  return(paste("Bearer",res$access_token))
}

// Generate token using above created function
token <- authentication_token(tenant = "TENANT",
                              client_id = "CLIENT ID",
                              client_secret = "CLIENT SECRET")

List Folder

folders <- httr::GET("https://${DATALAKE-STORE-NAME}.azuredatalakestore.net/webhdfs/v1/${FOLDER-PATH}?op=LISTSTATUS",
               add_headers(Authorization = token))

content(folders)

Create Folder

mkdir_operation <- stringr::str_interp("https://${DATALAKE-STORE-NAME}.azuredatalakestore.net/webhdfs/v1${FOLDER-PATH}?op=MKDIRS")

httr::PUT(mkdir_operation, add_headers(Authorization = token))

Move Folder

move_operation <- stringr::str_interp("https://${DATALAKE-STORE-NAME}.azuredatalakestore.net/webhdfs/v1${SOURCE-FOLDER-PATH}?op=RENAME&destination=${DESTINATION-FOLDER-PATH}")

httr::PUT(move_operation, add_headers(Authorization = token))

Read and Write Excel from Azure Datalake Store using R and Rest API

Fri, 06 Sep 2019 00:00:00 +0000

Import Prerequisite

library(httr)
library(curl)  
library(stringr)
library(readxl)

Authenticate

authentication_token <- function(tenant, client_id, client_secret){
  h <- new_handle()
  handle_setform(h,
                 "grant_type"="client_credentials",
                 "resource"="https://management.core.windows.net/",
                 "client_id" = client_id,
                 "client_secret" = client_secret
  )
  path = stringr::str_interp("https://login.windows.net/${tenant}/oauth2/token")
  req <- curl_fetch_memory(path, handle = h)
  res <- fromJSON(rawToChar(req$content))
  return(paste("Bearer",res$access_token))
}

// Generate token using above created function
token <- authentication_token(tenant = "TENANT",
                              client_id = "CLIENT ID",
                              client_secret = "CLIENT SECRET")

Read Binary

datalake_store_name = 'Enter-DataLake-Store-Name'
initial_path <- stringr::str_interp("https://${datalake_store_name}.azuredatalakestore.net/webhdfs/v1/")
query_string <- "?op=OPEN&read=true"
path <- 'Enter-Excel-File-Datalake-Path'
file_path <- paste0(initial_path, path, query_string)
// Request the file using GET and authentication token
file_request <- httr::GET(file_path, add_headers(Authorization = token))
// Read the file content
excel_data <- content(file_request)

Write Binary

// Open a local file in write mode with .xlsx extension, choose .xls if the orignal file is of .xls extension
local_file_path <- 'Enter-Local-File-Path.xlsx'
write_excel_data <- file(local_file_path, 'wb')
// write the file content to local file
writeBin(excel_data, write_excel_data)
// close the local file
close(write_excel_data)

Read Excel

// read the locally create excel file using readxl library
// enter sheet name value to open the required sheet
excel_data <- readxl::read_xlsx(local_file_path, sheet='Sheet-Name')

Access Azure Active Directory Groups using R

Mon, 02 Sep 2019 00:00:00 +0000

The following code snippets are on creating a connection to Azure Active Directoy using R with Service-to-Service authentication with client secret and client id using REST API and fetch Active Directory`s groups and associated details with them like owners and memebers.

Import Prerequisite

library(httr)
library(jsonlite)
library(curl)

Authenticate

h <- new_handle()
handle_setform(h,
               "grant_type"="client_credentials",
               "scope" = "https://graph.microsoft.com/.default",
               "client_id"= "ENTER CLIENT ID HERE",
               "client_secret"="ENTER CLIENT SECRET HERE",
               "tenant_id"= "ENTER TENANT ID HERE"
)
authentication_url <- "https://login.microsoftonline.com/{ENTER-TENANT-ID-HERE}/oauth2/v2.0/token"
authentication_response <- fromJSON(rawToChar(curl_fetch_memory(authentication_url, handle = h)$content))
authentication_token <- paste(authentication_response$token_type, authentication_response$access_token)
// token_type is "Bearer" in this case and hence authentication token starts with "Bearer "

// Function to return response using above authentication token and a request link
get_response <- function(request_link, auth_token){
  request <- GET(request_link, add_headers(Authorization = auth_token))
  response <- fromJSON(content((request),"text"))
  return(response)
}

Groups

link <- "https://graph.microsoft.com/v1.0/groups"
groups <- get_response(link, authentication_token)

// dataframe which contains all group details is stored in value variable
groups$value

// The first hit will return only 100 results. To get more, use the 
// @odata.nextLink property value as the next link value, and loop through it
groups[["@odata.nextLink"]]

Each group created above will have an unique group id associated with it. This id value can be used to get each group members and owners details as well, as shown below :

Part 5 - Plotting Using Seaborn - Radar

Mon, 26 Aug 2019 00:00:00 +0000

Introduction and Data preparation

Please follow the folloing links regarding data preparation and previous posts to follow along -

For Data Preparation - Part 0 - Plotting Using Seaborn - Data Preparation{:target="_blank"}
For Part 1 - Part 1 - Plotting Using Seaborn - Violin, Box and Line Plot{:target="_blank"}
For Part 2 - Part 2 - Plotting Using Seaborn - Distribution Plot, Facet Grid{:target="_blank"}
For Part 3 - Part 3 - Plotting Using Seaborn - Donut{:target="_blank"}

Part 2 - Plotting Using Seaborn - Distribution Plot, Facet Grid

Fri, 23 Aug 2019 00:00:00 +0000

Introduction and Data preparation

Please follow the folloing links regarding data preparation and previous posts to follow along -

For Data Preparation - Part 0 - Plotting Using Seaborn - Data Preparation{:target="_blank"}
For Part 1 - Part 1 - Plotting Using Seaborn - Violin, Box and Line Plot{:target="_blank"}

Subset of data based on complexity

test_scores_easy = test_scores[test_scores['Complexity']=='Easy']
test_scores_medium = test_scores[test_scores['Complexity']=='Medium']
test_scores_hard = test_scores[test_scores['Complexity']=='Difficult']

Distribution of score percentage across track in test with easy complexity

sns.set(style="whitegrid")
g = sns.FacetGrid(test_scores_easy, col='Track', row='Test Name', height = 4, aspect =2.5)
g.map(sns.distplot, "Percent", kde = False, hist = True, rug = False)
g.set_titles(size =20)
g.set_xlabels(size = 25)
g.set_ylabels(size = 25, label = "Participants")
g.set_yticklabels(fontsize =25)
g.set_xticklabels(fontsize =25, labels = [0,0,20,40,60,80,100])
g.fig.suptitle('Distribution of score percentage across track in test with easy complexity', fontsize=40, x = 0.5, y = 1.05)
#plb.savefig('Distribution_easy.png',dpi=50,bbox_inches='tight')

Part 3 - Plotting Using Seaborn - Donut

Fri, 23 Aug 2019 00:00:00 +0000

Introduction and Data preparation

Please follow the folloing links regarding data preparation and previous posts to follow along -

For Data Preparation - Part 0 - Plotting Using Seaborn - Data Preparation{:target="_blank"}
For Part 1 - Part 1 - Plotting Using Seaborn - Violin, Box and Line Plot{:target="_blank"}
For Part 2 - Part 2 - Plotting Using Seaborn - Distribution Plot, Facet Grid{:target="_blank"}

Basic prelimanries for Donut Chart

company_headcount = pd.melt(test_participant, id_vars=['Designation'], 
                            value_vars=['Engineering', 'Quality Assurance', 'Support']) \
                        .rename(columns={"variable": "Track", "value": "Headcount"})

for_donuts = test_scores.groupby(['Track','Designation'])[['Participant identifier']].nunique().reset_index()

participant_matrix = pd.merge(company_headcount,for_donuts,how = 'left',left_on = ['Track', 'Designation'],
                              right_on= ['Track', 'Designation'])

participant_matrix_eng = participant_matrix[participant_matrix['Track'] == 'Engineering']
participant_matrix_sup = participant_matrix[participant_matrix['Track'] == 'Support']
participant_matrix_qa = participant_matrix[participant_matrix['Track'] == 'Quality Assurance']

Participation proportion of all Track

participant_matrix = participant_matrix.groupby(['Track'])[['Headcount','Participant identifier']].sum().reset_index()
participant_matrix['Headcount Percent'] = round(participant_matrix['Headcount']/participant_matrix['Headcount'].sum(),4)*100
participant_matrix['Percent'] =  round((participant_matrix['Participant identifier']*participant_matrix['Headcount Percent']/participant_matrix['Headcount']),2)
participant_matrix['Not Participated'] = participant_matrix['Headcount Percent'] - participant_matrix['Percent']
participant_matrix['actual percent'] = participant_matrix['Participant identifier']*100/participant_matrix['Headcount'].sum()
group_names=list(participant_matrix['Track'] + " (" + participant_matrix['Headcount'].astype(int).astype(str) + ", "+ participant_matrix['Headcount Percent'].astype(int).astype(str) + "% )")
group_size=list(participant_matrix['Headcount Percent'])


subgroup_names=[(str(int(round(x,0)))+"%") for t in  list(zip(participant_matrix['Percent'], 
                                                              participant_matrix['Not Participated'])) for x in t]
subgroup_size=[x for t in  list(zip(participant_matrix['Percent'], participant_matrix['Not Participated'])) for x in t]

pie_size= list(participant_matrix['actual percent'])
pie_size.append(100 - sum(pie_size))
pie_name = list(zip(participant_matrix['Track'], participant_matrix['Participant identifier'].astype(int)))
NotParticipated = "Not Participated, " + str(int(participant_matrix['Headcount'].sum() - participant_matrix['Participant identifier'].sum()))
pie_name.append(NotParticipated)
    
# Create colors
a, b, c, d, e=["#e4f1fe","#c5eff7", "#81cfe0","#00b5cc","#52b3d9"]
 
bbox_props = dict(boxstyle="square, pad=0.3", fc="w", ec="k", lw=0.72)
# First Ring (outside)
fig, ax = plt.subplots(figsize=(15,15))
ax.axis('equal')
mypie, autotexts = ax.pie(group_size, radius=1.5, labels=group_names, colors=[a,b,c,d,e] ,labeldistance=1,
                         textprops=dict(color="black"))
plt.setp( mypie, width=0.3, edgecolor='white')
plt.setp(autotexts, size=24, weight="bold")

 
# Second Ring (Inside)
mypie2,autotexts2 = ax.pie(subgroup_size, radius=1.2, labels=subgroup_names, labeldistance=1.1, 
                   colors=["#55efc4","#ff7675","#55efc4","#ff7675","#55efc4","#ff7675","#55efc4","#ff7675","#55efc4","#ff7675"],
                    textprops=dict(color="black")
                 )

mypie3,autotexts3 = ax.pie(pie_size, radius=0.7, labels=pie_name, labeldistance=1, 
                  colors=["#e4f1fe","#c5eff7", "#81cfe0","#ff7675"],
                    textprops=dict(color="black"),rotatelabels =True
                 )

fig.suptitle('Participation proportion of all Track', fontsize=50, x = 0.5, y = 1.05)

plt.setp(autotexts2, size=24, weight="bold")
plt.setp(autotexts3, size=15, weight="bold")
plt.setp(mypie2,width=0.3, edgecolor='white')
plt.setp(mypie3,width=0.7, edgecolor='white')
#plb.savefig('Participation.png',dpi=100,bbox_inches='tight')

Part 4 - Plotting Using Seaborn - Heatmap, Lollipop Plot, Scatter Plot

Fri, 23 Aug 2019 00:00:00 +0000

Introduction and Data preparation

Please follow the folloing links regarding data preparation and previous posts to follow along -

For Data Preparation - Part 0 - Plotting Using Seaborn - Data Preparation{:target="_blank"}
For Part 1 - Part 1 - Plotting Using Seaborn - Violin, Box and Line Plot{:target="_blank"}
For Part 2 - Part 2 - Plotting Using Seaborn - Distribution Plot, Facet Grid{:target="_blank"}
For Part 3 - Part 3 - Plotting Using Seaborn - Donut{:target="_blank"}

Part 1 - Plotting Using Seaborn - Violin, Box and Line Plot

Wed, 21 Aug 2019 00:00:00 +0000

Introduction and Data preparation

Please follow the folloing links regarding data preparation and previous posts to follow along -

For Data Preparation - Part 0 - Plotting Using Seaborn - Data Preparation{:target="_blank"}

Violin Plot showing distribution of score for each track by complexity

plot1 = sns.catplot(y="Track",
                 x="Score",
                 data=test_scores, 
                 jitter=True, 
                 height=9, 
                 aspect=0.5,
                 kind = "violin", 
                 col = "Complexity",
                 cut = 0,
                 col_order = ["Easy","Medium","Difficult"],
                 palette = sns.color_palette(["#a05195","#d45087","#f95d6a"])) \
                .set_ylabels(fontsize=25) \
                .set_xlabels(fontsize = 25, label = "Score") \
                .set_xticklabels(fontsize=20) \
                .set_yticklabels(fontsize=20) \
                .set_titles(size = 20) 

plot1.fig.subplots_adjust(top=0.8)
plot1.fig.suptitle('Violin Plot showing distribution of score for each track by complexity',size = 30)
#plot1.savefig("Violin Plot -1.png",dpi=100,bbox_inches='tight')

Part 0 - Plotting Using Seaborn - Data Preparation

Tue, 20 Aug 2019 00:00:00 +0000

Import Preliminaries and datasets

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib.pylab as plb
import warnings
warnings.filterwarnings('ignore')

test_scores = pd.read_csv("Data/Test scores.csv", parse_dates=['Test taken date'])
test_master = pd.read_csv("Data/Test master.csv")
test_participant = pd.read_csv("Data/Audience summary.csv")

We have three datasets, namely -

Test Scores Dataset

This contains scores of each particpant in the test they appeared.

test_scores.head()

	Participant identifier	Test Name	Test taken date	Track	Designation	Score
0	37MCTM	If conditional	2018-11-23	Engineering	Lead	18
1	37MCTM	Determiners and Quantifiers	2018-11-23	Engineering	Lead	28
2	37MCTM	Modals	2018-11-23	Engineering	Lead	22
3	37MCTM	Tenses	2018-11-13	Engineering	Lead	12
4	37MCTM	Pronouns	2018-11-13	Engineering	Lead	15

---

Test Master

This is about the other details associated with each test.

Nearest Neighbors using L2 and L1 Distance

Sat, 20 Jul 2019 00:00:00 +0000

Preliminaries
Distance Matrics
- L2 Norm
- L1 Norm
Nearest Neighbor
- Using L2 Distance
- Using L1 Distance
Predictions
Errors
Confusion Matrix
- Using Pandas
- From Scratch

Preliminaries

import numpy as np

# Load data set and code labels as 0 = ’NO’, 1 = ’DH’, 2 = ’SL’
labels = [b'NO', b'DH', b'SL']
data = np.loadtxt('column_3C.dat', converters={6: lambda s: labels.index(s)} )

# Separate features from labels
x = data[:,0:6]
y = data[:,6]

# Divide into training and test set
training_indices = list(range(0,20)) + list(range(40,188)) + list(range(230,310))
test_indices = list(range(20,40)) + list(range(188,230))

trainx = x[training_indices,:]
trainy = y[training_indices]
testx = x[test_indices,:]
testy = y[test_indices]

Distance Metrics

L2 norm

distance_l2norm = np.sqrt(np.sum(np.square(a-b)))

Full Outer Join in DAX in PowerBI

Wed, 10 Jul 2019 00:00:00 +0000

Full Outer Join

According to Wikipedia{:target="_blank"} -

“Conceptually, a full outer join combines the effect of applying both left and right outer joins. Where rows in the FULL OUTER JOINed tables do not match, the result set will have NULL values for every column of the table that lacks a matching row. For those rows that do match, a single row will be produced in the result set (containing columns populated from both tables).”

Exception handling in Python

Mon, 08 Jul 2019 00:00:00 +0000

Try and Except

Handle Exceptions

The following statement will throw an error as string can’t be parsed into integers.

hello = "hello world"
print("Converting string to int")
print(int(hello))
print("Done!")

Output:

Converting string to int

---------------------------------------------------------------------------

ValueError                                Traceback (most recent call last)

<ipython-input-2-0017d7718fd0> in <module>
      1 print("Converting string to int")
----> 2 print(int(hello))
      3 print("Done!")


ValueError: invalid literal for int() with base 10: 'hello world'

We can catch the error created in above code execution, by using try…except clause in python.

Functions in Python - return, scope, args and kwargs

Sat, 29 Jun 2019 00:00:00 +0000

Return in Function

Function execution stops once it reaches a return statement.

def add_num(x1,x2):
    return x1+x2
add_num(1,2)

Output:

The function will return 20, the addition expression will not execute as before that a return statement is reached.

Break, Continue and Pass in Python

Fri, 28 Jun 2019 00:00:00 +0000

Continue vs Break

Continue

print("The loop will skip the value when value of i is 2 and restart from next value of i - ")
for i in range(0,4):
    if i == 2:
        continue
    else:
        print(i+100)

Output:

The loop will skip the value when value of i is 2 and restart from next value of i - 
100
101
103

Break

print("The loop will break when value of i is 2. No more further execution! - ")
for i in range(0,4):
    if i == 2:
        break
    else:
        print(i+100)

Output:

The loop will break when value of i is 2. No more further execution! - 
100
101

Continue vs Pass

Continue

print('When value of i is 2, it will start from next iteration - ')
for i in range(0,4):
    if i==2:
        continue
    else:
        print(100+i)
    print(i)

Output:

When value of i is 2, it will start from next iteration - 
100
0
101
1
103
3

Pass

print('When value of i is 2, it does nothing and passes execution to next statement - ')
for i in range(0,4):
    if i==2:
        pass
    else:
        print(100+i)
    print(i)

Output:

When value of i is 2, it does nothing and passes execution to next statement - 
100
0
101
1
2
103
3

Jupyter Notebook Link - Break, Continue and Pass in Python{:target="_blank"}

Connect to azure datalake store using R

Mon, 10 Jun 2019 00:00:00 +0000

The following code snippets are on creating a connection to Azure Data Lake Storage Gen1 using R with Service-to-Service authentication with client secret and client id using REST API. Follow the link, for more details on different ways to connect to Azure Data Lake Storage Gen1

Import Prerequisite

library(httr)
library(curl)  
library(stringr)

Authenticate

authentication_token <- function(tenant, client_id, client_secret){
  h <- new_handle()
  handle_setform(h,
                 "grant_type"="client_credentials",
                 "resource"="https://management.core.windows.net/",
                 "client_id" = client_id,
                 "client_secret" = client_secret
  )
  path = str_interp("https://login.windows.net/${tenant}/oauth2/token")
  req <- curl_fetch_memory(path, handle = h)
  res <- fromJSON(rawToChar(req$content))
  return(paste("Bearer",res$access_token))
}

token <- authentication_token(tenant = "TENANT",
                              client_id = "CLIENT ID",
                              client_secret = "CLIENT SECRET")

Read

load_data <- function(datalake, path, auth_token){
  file_path = str_interp("https://${datalake}.azuredatalakestore.net/webhdfs/v1/${path}?op=OPEN&read=true")
  r <- httr::GET(file_path, add_headers(Authorization = auth_token))
  return(read.csv(textConnection(content(r, 'text', encoding="UTF-8")), check.names=FALSE))
}

datalake_name <- "NAME OF THE DATALAKE"
file_path <- "FILE PATH IN THE DATALAKE FOLDER"
authentication_token <- "TOKEN CREATED"

load_data(datalake_name, file_path, authentication_token)

Write

upload_data <- function(dataset, datalake, path, auth_token){
  write.csv(dataset, textConnection("filecontent","w"), row.names=F)
  file_path <- str_interp("https://${datalake}.azuredatalakestore.net/webhdfs/v1/${path}?op=CREATE&overwrite=true&write=true")
  httr::PUT(file_path,
            body = filecontent,
            add_headers(Authorization = auth_token,
                        "Transfer-Encoding" = "chunked"))
  return("The file is uploaded")
}

dataset <- "DATASET TO UPLOAD"
datalake_name <- "NAME OF THE DATALAKE"
file_path <- "FILE PATH IN THE DATALAKE FOLDER"
authentication_token <- "TOKEN CREATED"

upload_data(dataset, datalake_name, file_path, authentication_token)

Business Days with Custom Holidays

Thu, 30 May 2019 00:00:00 +0000

Importing Packages and Datasets

import pandas as pd

start_date = ['2019-06-03', '2019-06-13', '2019-10-01', '2019-09-01']
end_date =  ['2019-08-31', '2019-06-21', '2019-10-25', '2019-12-25']
data = pd.DataFrame(list(zip(start_date,end_date)), columns = ['Start Date', 'End Date'])
data

	Start Date	End Date
0	2019-06-03	2019-08-31
1	2019-06-13	2019-06-21
2	2019-10-01	2019-10-25
3	2019-09-01	2019-12-25

---

Custm Holidays List

holiday_dates = [pd.datetime(2019, 8, 15), pd.datetime(2019, 10, 2), pd.datetime(2019, 10, 8),
                 pd.datetime(2019, 10, 28), pd.datetime(2019, 12, 25)]

Example - calculate business days (excluding weekends) with custom holidays

# Exclude weekends and custom holidays
pd.bdate_range(pd.datetime(2019, 8, 1), pd.datetime(2019, 8, 31),  holidays=holiday_dates, freq='C', weekmask = None)

DatetimeIndex(['2019-08-01', '2019-08-02', '2019-08-05', '2019-08-06',
               '2019-08-07', '2019-08-08', '2019-08-09', '2019-08-12',
               '2019-08-13', '2019-08-14', '2019-08-16', '2019-08-19',
               '2019-08-20', '2019-08-21', '2019-08-22', '2019-08-23',
               '2019-08-26', '2019-08-27', '2019-08-28', '2019-08-29',
               '2019-08-30'],
              dtype='datetime64[ns]', freq='C')

Applying the same using lambda

data['Business Days'] = data.apply(lambda x: len(pd.bdate_range(x['Start Date'],
                                                                x['End Date'],
                                                                holidays=holiday_dates,
                                                                freq='C',
                                                                weekmask = None)), axis = 1)
data

	Start Date	End Date	Business Days
0	2019-06-03	2019-08-31	64
1	2019-06-13	2019-06-21	7
2	2019-10-01	2019-10-25	17
3	2019-09-01	2019-12-25	79

---

Notebook Link - Business Days with Custom Holidays{:target="_blank"}

Months between two dates

Thu, 30 May 2019 00:00:00 +0000

Importing Packages and Datasets

import pandas as pd
import numpy as np

start_date = ['2019-06-03', '2019-06-13', '2018-11-05', '2019-05-31', '2019-06-01', '2019-09-01']
end_date =  ['2019-08-31', '2019-08-23', '2018-11-25', '2019-07-1', '2019-07-31', '2019-10-25']
data = pd.DataFrame(list(zip(start_date,end_date)), columns = ['Start Date', 'End Date'])

	Start Date	End Date
0	2019-06-03	2019-08-31
1	2019-06-13	2019-08-23
2	2018-11-05	2018-11-25
3	2019-05-31	2019-07-1
4	2019-06-01	2019-07-31
5	2019-09-01	2019-10-25

---

Logic

# Store data index as a new column which will act as a primary key will be used later and initiate a blank new dataframe
data['Key'] = data.reset_index(drop = True).index
final = pd.DataFrame()
# Start the loop which will iterate on each particular combination of start data and end date -
for i in range(0,len(data)):
    start_date = data['Start Date'][i]
    end_date = data['End Date'][i]
    print("-----------------------------------------------------------------------------")
    print("Start Date is {0}, End Date is {1}".format(start_date, end_date))
    print("-----------------------------------------------------------------------------")
# If both dates fall in same month and year, then the date range is the same ( no consecutive months exists), check for same and proceed if the any of these are not same
# Store Month start date falling between two dates in  a new dataframe object
# Store Month end dates falling between two dates
# If first month start date is not equal to the start date of the date range, then we need to shift the index by one to accommodate for the enddate of the first month
    if ((pd.to_datetime(start_date).month != pd.to_datetime(end_date).month) | (pd.to_datetime(start_date).year != pd.to_datetime(end_date).year)):
        month_startdates = pd.DataFrame(pd.date_range(start_date, end_date, freq = 'MS'), columns=['Month_StartDate'])
        month_enddates = pd.DataFrame(pd.date_range(start_date, end_date, freq = 'M'), columns = ['Month_EndDate'])
        print("Ranges of months in the above date range : \n {0} \n {1}".format(month_startdates, month_enddates))
        if not pd.to_datetime(start_date).month == pd.to_datetime(month_startdates['Month_StartDate'][0]).month:
            month_startdates.index = range(1,len(month_startdates)+1)
        df = pd.concat([month_startdates,month_enddates], axis = 1)
        print("\n After adjusting for indexes : \n {0} ".format(df))
        df['Key'] = i
        final = final.append(df).reset_index(drop = True)
        print("\n Map it to orignal index, so that it can be merged with oringal DF : \n {0}".format(df))
    else:
# If month and year of both start date and end date are same, simply, create the record with the same
        final.append({'Month_StartDate':pd.to_datetime(start_date),
              'Month_EndDate': pd.to_datetime(end_date),
              'Key': i}, ignore_index =True)

Example 1 : When i = 1

-----------------------------------------------------------------------------
Start Date is 2019-06-13, End Date is 2019-08-23
-----------------------------------------------------------------------------
Ranges of months in the above date range :
   Month_StartDate
0      2019-07-01
1      2019-08-01
   Month_EndDate
0    2019-06-30
1    2019-07-31

 After adjusting for indexes :
   Month_StartDate Month_EndDate
0             NaT    2019-06-30
1      2019-07-01    2019-07-31
2      2019-08-01           NaT

 Map it to orignal index, so that it can be merged with oringal DF :
   Month_StartDate Month_EndDate  Key
0             NaT    2019-06-30    1
1      2019-07-01    2019-07-31    1
2      2019-08-01           NaT    1

Example 2 : When i = 3

-----------------------------------------------------------------------------
Start Date is 2019-05-31, End Date is 2019-07-1
-----------------------------------------------------------------------------
Ranges of months in the above date range :
   Month_StartDate
0      2019-06-01
1      2019-07-01
   Month_EndDate
0    2019-05-31
1    2019-06-30

 After adjusting for indexes :
   Month_StartDate Month_EndDate
0             NaT    2019-05-31
1      2019-06-01    2019-06-30
2      2019-07-01           NaT

 Map it to orignal index, so that it can be merged with oringal DF :
   Month_StartDate Month_EndDate  Key
0             NaT    2019-05-31    3
1      2019-06-01    2019-06-30    3
2      2019-07-01           NaT    3

Merge with orignal data on key and index

data = pd.merge(data, final, how = 'left')
data

	Start Date	End Date	Key	Month_StartDate	Month_EndDate
0	2019-06-03	2019-08-31	0	NaT	2019-06-30
1	2019-06-03	2019-08-31	0	2019-07-01	2019-07-31
2	2019-06-03	2019-08-31	0	2019-08-01	2019-08-31
3	2019-06-13	2019-08-23	1	NaT	2019-06-30
4	2019-06-13	2019-08-23	1	2019-07-01	2019-07-31
5	2019-06-13	2019-08-23	1	2019-08-01	NaT
6	2018-11-05	2018-11-25	2	NaT	NaT
7	2019-05-31	2019-07-1	3	NaT	2019-05-31
8	2019-05-31	2019-07-1	3	2019-06-01	2019-06-30
9	2019-05-31	2019-07-1	3	2019-07-01	NaT
10	2019-06-01	2019-07-31	4	2019-06-01	2019-06-30
11	2019-06-01	2019-07-31	4	2019-07-01	2019-07-31
12	2019-09-01	2019-10-25	5	2019-09-01	2019-09-30
13	2019-09-01	2019-10-25	5	2019-10-01	NaT

---

Adjustment for missing start and end date

data.loc[data['Month_StartDate'].isnull(),'Month_StartDate'] = pd.to_datetime(data['Start Date'])
data.loc[data['Month_EndDate'].isnull(),'Month_EndDate'] = pd.to_datetime(data['End Date'])
data

	Start Date	End Date	Key	Month_StartDate	Month_EndDate
0	2019-06-03	2019-08-31	0	2019-06-03	2019-06-30
1	2019-06-03	2019-08-31	0	2019-07-01	2019-07-31
2	2019-06-03	2019-08-31	0	2019-08-01	2019-08-31
3	2019-06-13	2019-08-23	1	2019-06-13	2019-06-30
4	2019-06-13	2019-08-23	1	2019-07-01	2019-07-31
5	2019-06-13	2019-08-23	1	2019-08-01	2019-08-23
6	2018-11-05	2018-11-25	2	2018-11-05	2018-11-25
7	2019-05-31	2019-07-1	3	2019-05-31	2019-05-31
8	2019-05-31	2019-07-1	3	2019-06-01	2019-06-30
9	2019-05-31	2019-07-1	3	2019-07-01	2019-07-01
10	2019-06-01	2019-07-31	4	2019-06-01	2019-06-30
11	2019-06-01	2019-07-31	4	2019-07-01	2019-07-31
12	2019-09-01	2019-10-25	5	2019-09-01	2019-09-30
13	2019-09-01	2019-10-25	5	2019-10-01	2019-10-25

---

Jupyter Notebook Link - Months between two dates{:target="_blank"}

Applying functions over pandas dataframe using apply, applymap and map

Tue, 23 Apr 2019 00:00:00 +0000

Importing Packages and Datasets

import pandas as pd
import numpy as np
data = pd.DataFrame(np.random.rand(4, 3)*100,
                    columns=['Physics','Çhemistry','Maths'],
                    index = ['Student 1', 'Student 2','Student 3','Student 4'])
data

	Physics	Çhemistry	Maths
Student 1	59.488944	14.888411	52.794760
Student 2	21.872113	66.481646	87.190572
Student 3	9.885919	54.449674	58.696036
Student 4	33.804378	6.286295	30.373699

---

Applymap

RoundUpto2Decimal = lambda x: round(x,2)
data.applymap(RoundUpto2Decimal)

	Physics	Çhemistry	Maths
Student 1	59.49	14.89	52.79
Student 2	21.87	66.48	87.19
Student 3	9.89	54.45	58.70
Student 4	33.80	6.29	30.37

---

Apply

AverageMarks = lambda x: np.mean(x)
data.apply(AverageMarks)

Physics      31.262839
Çhemistry    35.526506
Maths        57.263767
dtype: float64

data.apply(AverageMarks,axis = 1)

Student 1    42.390705
Student 2    58.514777
Student 3    41.010543
Student 4    23.488124
dtype: float64

data.apply(lambda x: (x-np.min(x))/(np.max(x)-np.min(x)))

	Physics	Çhemistry	Maths
Student 1	1.000000	0.142903	0.394620
Student 2	0.241642	1.000000	1.000000
Student 3	0.000000	0.800118	0.498485
Student 4	0.482198	0.000000	0.000000

Map

SquareOfMarks = lambda x: x**2
data['Physics'].map(SquareOfMarks)

Student 1    3538.934482
Student 2     478.389328
Student 3      97.731389
Student 4    1142.735997
Name: Physics, dtype: float64

Notebook Link - Applying functions over pandas dataframe using apply, applymap and map{:target="_blank"}

Filtering using mask and where in pandas

Wed, 03 Apr 2019 00:00:00 +0000

Filtering a dataframe can be achieved in multiple ways using pandas. There are times when you simply need to update a column based on a condition which is true or vice-versa. In pandas dataframe there are some inbuilt methods to achieve the same using .where() and .mask().

df.where - Replace value when condition is false
df.mask - Replace value when condition is true

Initiating a dummy dataframe with some columns to understand the same -

String Interpolation in Python

Thu, 28 Mar 2019 00:00:00 +0000

String operations in python, sometimes can be a bit tedious, specially when we need to pass variables within Strings.

Although there are multiple ways to achieve the same, but some of the them are

String interpolation
str.format()

Declaring some variables -

Name = 'Python'
Framework = 'Django'

Using str.format()

"Language - {0}, Framework - {1}".format(Name, Framework)

‘Language - Python, Framework - Django’

Using f-strings format - string interpolation

f"Language - {Name}, Framework - {Framework}"

‘Language - Python, Framework - Django’

Join, Merge, Append and Concatenate

Mon, 25 Mar 2019 00:00:00 +0000

Working with multiple data frames often involves joining two or more tables to in bring out more no. of columns from another table by joining on some sort of relationship which exists within a table or appending two tables which is adding one or more table over another table with keeping the same order of columns.

Example of append data -> monthly files of revenue sheets of a company and wee need at end of the year to be clubbed into single table.

Attributes, Methods and Functions in python

Mon, 07 Jan 2019 00:00:00 +0000

Understand the concept of attributes, methods and functions under the context of a dataframe

Attributes

Attributes are the features of any object. They can be accessed by following a dot and the name of the following attribute.

For example: person.age, person.height
here, age and height are the attributes of the person object

Methods and Functions

Methods are always associated with an object where as the Functions are not dependent on any object. In simple term a method is on a object where as a function is independent of object.

Subsetting a dataframe in pandas

Sat, 05 Jan 2019 00:00:00 +0000

Importing packages and datasets

import pandas as pd
# Fetching data from url as csv by mentioning values of various paramters
data = pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data",
                   header = None,
                   index_col = False,
                   names = ['sepal_length','sepal_width','petal_length','petal_width','iris_class'])
# Unique classes of iris datasets
data.iris_class.unique()

array([‘Iris-setosa’, ‘Iris-versicolor’, ‘Iris-virginica’], dtype=object)

Subsetting

data_setosa = data[data.iris_class == 'Iris-setosa']
data_versicolor = data[data.iris_class == 'Iris-versicolor']
data_virginica = data[data.iris_class == 'Iris-virginica']

'''
Now we can have a look at descriptive statistics summary for each of the subset and can make inference like following -
* Each of the subset is of same size i.e., 50
* Average Sepal and Petal Length is lowest in setosa and highest in virginica
'''

data_setosa.describe().T

Metric	count	mean	std	min	25%	50%	75%	max
sepal_length	50.0	5.006	0.352490	4.3	4.800	5.0	5.200	5.8
sepal_width	50.0	3.418	0.381024	2.3	3.125	3.4	3.675	4.4
petal_length	50.0	1.464	0.173511	1.0	1.400	1.5	1.575	1.9
petal_width	50.0	0.244	0.107210	0.1	0.200	0.2	0.300	0.6

data_versicolor.describe().T

Metric	count	mean	std	min	25%	50%	75%	max
sepal_length	50.0	5.936	0.516171	4.9	5.600	5.90	6.3	7.0
sepal_width	50.0	2.770	0.313798	2.0	2.525	2.80	3.0	3.4
petal_length	50.0	4.260	0.469911	3.0	4.000	4.35	4.6	5.1
petal_width	50.0	1.326	0.197753	1.0	1.200	1.30	1.5	1.8

data_virginica.describe().T

Metric	count	mean	std	min	25%	50%	75%	max
sepal_length	50.0	6.588	0.635880	4.9	6.225	6.50	6.900	7.9
sepal_width	50.0	2.974	0.322497	2.2	2.800	3.00	3.175	3.8
petal_length	50.0	5.552	0.551895	4.5	5.100	5.55	5.875	6.9
petal_width	50.0	2.026	0.274650	1.4	1.800	2.00	2.300	2.5

Indexing and Sorting a dataframe using iloc and loc