Pandas commands for data analysis are the built-in functions and methods in Python’s pandas library that let you load, clean, transform, aggregate, and visualize tabular data. You use them by typing a method name after a DataFrame or Series object, such as df.head() or df.groupby('col').mean(). This article covers the most useful commands, how they work, and common pitfalls.
Background: What Pandas Is and Why It Matters
Pandas is an open-source Python library created by Wes McKinney in 2008 to make data manipulation fast and expressive. Its two core data structures are the Series (a one-dimensional labeled array) and the DataFrame (a two-dimensional table with rows and columns). Most real-world data analysis in Python starts with pandas because it can read nearly any tabular format—CSV, Excel, SQL, JSON, Parquet—and turn it into a DataFrame in one command.
Pandas commands are often chained together: you load data, filter rows, create new columns, group by a category, and then plot the result. Unlike base Python loops, pandas operations are vectorized, meaning they run on entire columns at once in C code, which is dramatically faster. For a data analyst or scientist, pandas is usually the first tool reached for after raw data arrives.
Core Commands: Reading and Inspecting Data
The first commands you’ll use are for loading and looking at your data. The most common are:
pd.read_csv('file.csv')— loads a CSV file into a DataFrame. Variants includeread_excel,read_sql,read_json, andread_parquet.df.head(n)— shows the first n rows (default 5).df.tail(n)shows the last n.df.info()— prints column names, non-null counts, and data types. Essential for spotting missing values.df.describe()— gives summary statistics (count, mean, std, min, quartiles, max) for numeric columns.df.shape— returns a tuple (rows, columns).df.columns— lists column names.df.dtypes— shows the data type of each column.
These commands are your first look at any dataset. Running df.info() immediately tells you if a column is misread as object instead of datetime, or if there are nulls you need to handle.
Cleaning and Preparing Data
Real data is messy. Pandas provides a suite of commands to clean it:
df.dropna()— removes rows with missing values. Useaxis=1to drop columns, orsubset=['col']to limit the check.df.fillna(value)— replaces NaN with a constant, a column mean, or a forward fill (method='ffill').df.rename(columns={'old':'new'})— renames columns.df.drop(columns=['col1','col2'])— removes columns.df.astype({'col':'float32'})— converts data types.df.duplicated()anddf.drop_duplicates()— find and remove duplicate rows.df['col'].str.strip()— removes whitespace from string columns.
A common mistake is using inplace=True on these methods. While it works, the pandas team recommends avoiding it; instead, assign the result back: df = df.dropna(). This makes your code clearer and avoids accidental mutation of the original DataFrame.
Selecting and Filtering Data
To get the rows and columns you need, pandas offers two main approaches: label-based (.loc) and position-based (.iloc).
df['col']— returns a Series for one column.df[['col1','col2']]— returns a DataFrame with the listed columns.df.loc[rows, cols]— selects by label. Example:df.loc[df['age'] > 30, ['name','age']].df.iloc[rows, cols]— selects by integer position. Example:df.iloc[0:5, 0:2]gives first 5 rows and first 2 columns.- Boolean masking:
df[df['sales'] > 1000]returns rows where the condition is True. df.query('sales > 1000 and region == "West"')— a string-based filter that can be more readable.
Remember: df['col'] with a single bracket returns a Series, while df[['col']] returns a DataFrame. This distinction matters when you call methods that behave differently on Series vs DataFrame (e.g., apply).
Grouping and Aggregating
Grouping is where pandas shines for summary analysis. The core command is groupby(), which splits data into groups, applies a function, and combines the results.
df.groupby('region')['sales'].sum()— total sales per region.df.groupby(['region','year'])['sales'].mean()— average sales by region and year.df.groupby('region').agg({'sales':['sum','mean'], 'profit':'max'})— multiple aggregations on different columns.df.pivot_table(values='sales', index='region', columns='year', aggfunc='sum')— creates a pivot table, which is a reshaped groupby result.
After a groupby, the result is often a Series or DataFrame with the grouping columns as the index. Use .reset_index() to turn the index back into a regular column, which is usually what you want for plotting or exporting.
Merging, Joining, and Concatenating
When you have multiple DataFrames, you need to combine them. The main commands are:
pd.merge(df1, df2, on='key')— SQL-style join. Specifyhow='left','right','inner', or'outer'.df1.join(df2, on='key')— joins on the index by default; can also join on a column.pd.concat([df1, df2], axis=0)— stacks rows vertically (axis=0) or columns horizontally (axis=1).
A frequent mistake is merging on columns with different names or data types (e.g., one is int, the other is string). Always check that the key columns have the same dtype before merging. Also, be aware of duplicate keys—they cause a Cartesian product, multiplying rows unexpectedly.
Common Mistakes and How to Avoid Them
Even experienced analysts hit these pandas pitfalls:
- Chained indexing:
df[df['a']>1]['b'] = 5can raise a SettingWithCopyWarning and may not modify the original. Usedf.loc[df['a']>1, 'b'] = 5instead. - Using
inplace=Trueinconsistently: It’s deprecated in many methods. Just assign the result. - Forgetting that
groupbydrops NaN groups by default: Usedropna=Falseif you want to keep them. - Mixing data types in a column: This can cause
df['col'].astype(float)to fail. Check withdf['col'].apply(type).unique(). - Assuming
applyis fast: It loops in Python, so it’s slow for large data. Prefer vectorized operations orgroupby.transformwhen possible.
Alternatives and When Not to Use Pandas
Pandas is not always the best tool. For datasets larger than memory, consider Dask or Polars, which are designed for out-of-core or parallel processing. For very small data, plain Python lists and dictionaries may be simpler. For database queries, SQL is often more efficient than loading everything into pandas. For statistical modeling, you’ll typically convert a DataFrame to NumPy arrays or use libraries like statsmodels that accept DataFrames directly.
Pandas also has a learning curve around the index. Many beginners struggle with reset_index, set_index, and how operations align on the index. If you find yourself constantly fighting the index, consider using df.reset_index(drop=True) early to keep a simple integer index.
Frequently Asked Questions
What is the difference between loc and iloc?
loc selects by label (row names or column names), while iloc selects by integer position (0-based). For example, df.loc[0] returns the row with index label 0, which may not be the first row if the index is non-integer. df.iloc[0] always returns the first row. Use loc when you know the index value, and iloc when you want the nth row.
How do I apply a function to every row or column?
Use df.apply(func, axis=0) to apply along columns (default) or axis=1 for rows. For element-wise operations on a single column, use df['col'].apply(func). For faster performance, prefer vectorized operations or df['new'] = df['col'].map(dict) for simple mappings.
Why does my groupby result have a weird index?
By default, groupby uses the grouping columns as the new index. To get a flat DataFrame with those columns as regular columns, call .reset_index() after the aggregation. For example: df.groupby('region')['sales'].sum().reset_index().
How do I handle missing values without dropping too much data?
First, check the extent with df.isnull().sum(). If a column has many missing values, consider filling with a median or mode, or creating a flag column for missingness. If rows have few missing values, you can drop them with df.dropna(thresh=3) to keep rows with at least 3 non-null values. For time series, forward fill (df.fillna(method='ffill')) is often appropriate.
What is the fastest way to read a large CSV?
Use pd.read_csv('file.csv', dtype={'col':'int32'}, parse_dates=['date']) to reduce memory. Specify the usecols parameter to load only needed columns. For very large files, consider reading in chunks with chunksize or using pd.read_csv(..., engine='pyarrow') if pyarrow is installed.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.
More cards in this drawer
- Best Streaming Remote: Top Picks for Every Setup
- Claude Pro Hidden Features Power Users Try
- Tesla Rental Prices: Why They’re So Low255.00 USD
- Best Family Western Movies to Stream Right Now
- Luxury V6 Sports Sedan: Can It Easily Top 100k Miles?
- Puppets in TV Shows: Classic Proof They’re Better