A DataFrame is the table at the heart of pandas: rows, named columns, a row index, and a column dtype per column. Selection, filtering, the index, and missing data are the building blocks; most pandas code becomes a small set of patterns applied in different orders. This lesson covers df[col], df.loc[], df.iloc[], boolean filtering, adding and dropping columns, the index, sorting, renaming, missing data, dtypes, and the SettingWithCopyWarning.

Pandas has three ways to pull data out of a DataFrame: bracket selection on columns, .loc[] for label-based access, and .iloc[] for position-based access. They look similar but mean different things, and mixing them up is a common source of pandas bugs.

Start with a small product catalog:

The index is the product ID (P001 through P005), not the default RangeIndex of 0, 1, 2, .... The columns are name, category, price, stock.

Column Selection with df[col]

df["price"] returns a single column as a Series. df[["price", "stock"]] (note the double brackets) returns a new DataFrame with just those columns. The double-bracket form is a list of column names, which is how pandas decides "give me a DataFrame, not a Series."

The Series version has the index on the left and a single column of values. The DataFrame version preserves the table shape with the same index but only the requested columns. Both keep the original index.

Label-Based Access with .loc[]

.loc[] selects by label: the row index value, the column name, or both. The general form is df.loc[row_label, column_label]. Either side can be a single value, a list of values, or a slice.

The first call picks one row by its index label and returns it as a Series (one column per original column). The second pulls one cell. The third returns a DataFrame with two rows and two columns.

.loc[] slices are inclusive on both ends, which is different from Python's usual half-open slicing:

"P002":"P004" includes P004. The column slice "name":"price" includes price. Inclusive slicing matches how labeled ranges read in English ("from P002 through P004"), but it diverges from Python list slicing.

.loc[] performs a label lookup per index entry, which is O(1) per lookup on a sorted or hashed index. Repeated .loc calls in a Python loop are slow because each call has the same per-call overhead; vectorize with a single .loc[mask] or .isin([...]) instead.

Position-Based Access with .iloc[]

.iloc[] selects by integer position, ignoring whatever the index labels are. It's the same shape as .loc[] but with integers instead of labels.

Position 0 is the first row regardless of its label. Position 2 in a row is the third column (price). .iloc[] slices are half-open like normal Python: 0:3 is positions 0, 1, 2, not 0 through 3.

The contrast matters when the index isn't 0, 1, 2, .... With the product-ID index above, products.loc["P002"] and products.iloc[1] happen to return the same row, but the two operations mean different things. Change the index order (with sort_values or reindex) and the integer positions shift while the labels stay attached to their data.

What's wrong with this code?

.loc[0] looks for an index label of 0. The index contains the strings "P003", "P001", "P002", none of which equal the integer 0. To get the first row by position, use .iloc[0]. To get the row labeled "P001", use .loc["P001"].

Fix:

Pandas filters rows by passing a Boolean Series the same length as the DataFrame. The rows where the value is True are kept; the rest are dropped.

products["price"] > 50 produces a Series of [False, True, False, True, False]. Passing that Series back into products[...] keeps the rows aligned with True. The original DataFrame is untouched; the result is a new DataFrame.

Combine conditions with & (and), | (or), ~ (not). Each piece needs its own parentheses because & binds tighter than >:

Two conditions joined with &. Both must be true for the row to survive. Without the parentheses around each comparison, pandas would try to evaluate "Electronics" & products["stock"] first and raise an error.

For "value in a set" filters, use .isin():

.isin([...]) returns a Boolean Series that's True wherever the value is one of the listed options. It's the pandas equivalent of x in {"Audio", "Cables"} on a single value, but applied to every row at once.

The query() method is a string-based alternative that some teams prefer for long filter expressions:

The string syntax avoids the parenthesis-heavy bracket form and lets you reference columns as bare names. It's a readability choice; performance is similar for most cases.

Adding a column is a single assignment. Pandas figures out the alignment by the index.

The expression products["price"] * products["stock"] is a Series of element-wise products, aligned by index. The assignment creates a new column called total_value in place. No copy of the DataFrame is made.

You can also assign a scalar (the value is broadcast across every row) or a list of the right length:

The scalar "USD" becomes the value at every row. The list of five floats lines up one per row.

For multiple columns at once, assign() returns a new DataFrame with the additions without mutating the original. It fits cleanly in method chains:

assign() takes keyword arguments where each key is a new column name. The value can be a Series or a function that takes the DataFrame and returns one. The second new column uses the first one (d["final_price"]) because assign() adds them in order. The original products is unchanged; priced is a new object.

To drop columns, use drop(columns=[...]):

drop() returns a new DataFrame by default. Pass inplace=True to modify the original, although most pandas code avoids inplace because it makes method chains awkward and the memory savings are small. To drop rows by label, use drop(index=[...]) instead.

The index is the row labels. It serves three jobs at once: as a row identifier (rows can be looked up by it), as the basis for alignment in operations between Series and DataFrames, and as the join key for merge() when using the index.

set_index and reset_index

set_index("name") promotes a column to the index. reset_index() does the reverse: turn the index into a regular column and give the DataFrame a fresh RangeIndex starting at 0.

After set_index("name"), the product name moves to the index column on the left and isn't part of the regular columns anymore. After reset_index(), it's back as a column and the index is 0, 1, 2, .... Neither call modifies the original products; both return new DataFrames.

A good index pays off for label-based lookups, joins, and time-based slicing. A DataFrame of orders indexed by order_id makes df.loc["ORD-1023"] instant. A DataFrame indexed by a DatetimeIndex makes df.loc["2026-01":"2026-03"] work as a date range filter.

MultiIndex Briefly

A MultiIndex is an index with more than one level. Useful when each row is identified by a combination of values, like (category, product_id) or (year, month).

The index now has two levels. .loc["Electronics"] returns every row in that group; .loc[("Electronics", "Webcam")] returns the single row. MultiIndex is powerful for grouped data but adds complexity, and most pandas code keeps the index single-level until there's a real reason to escalate. The pandas.MultiIndex docs cover the full API.

Sorting

sort_values() sorts rows by the values in one or more columns. sort_index() sorts by the index labels.

The first sort is by price ascending. The second sorts by two columns: category ascending (alphabetical), then within each category by price descending. The ascending parameter takes either a single bool or a list matching the column list.

Both calls return new DataFrames. The index labels travel with the rows; the index doesn't get reset just because the order changes. For a clean 0, 1, 2, ... index after sorting, chain .reset_index(drop=True).

reindex and rename

reindex() conforms a DataFrame to a new set of row labels (or column labels). Missing labels get NaN. Extra labels are dropped.

P099 isn't in the original index, so its entire row is NaN. The dtype of integer columns like stock was promoted to float because integers in pandas can't represent NaN (the older int dtypes can't; the newer nullable Int64 can).

rename() changes labels. Pass a dict for columns= to rename columns, or index= for the index:

Only the listed columns are renamed; the rest keep their original names. rename() is forgiving of keys that don't exist (they're ignored without warning), which is convenient most of the time and occasionally hides typos. Pass errors="raise" to catch them.

The diagram shows the four index-related transforms side by side. set_index and rename change identifiers; sort_values rearranges rows; reindex enforces a target shape and fills gaps with NaN. Each returns a new DataFrame; the original is untouched.

Pandas marks missing values with NaN (a special floating-point value from NumPy). Almost every dataset has some missing values, and the handling is one of pandas' core jobs.

np.nan is the canonical missing-numeric marker. None in an object column (strings) is treated as missing for most operations. rating was inferred as float64 to accommodate the NaN values, not int64.

Detecting Missing Values

isna() (alias isnull()) returns a Boolean DataFrame the same shape as the original. True means missing.

The first call gives the cell-by-cell map. The second sums by column, which is the standard quick check for "how much is missing where?". The comment column's empty string "" is not counted as missing because an empty string isn't NaN; pandas treats it as a value.

Filling Missing Values

fillna(value) replaces every NaN with the given value. You can pass a scalar (one fill for everything), a dict (per-column fills), or a method like forward-fill for time series.

Per-column fills work when different columns mean different things when missing. A missing rating is reasonably 0 (or the mean); a missing customer is reasonably "anonymous". Filling all NaNs with a single value like 0 would make the customer name 0 too.

fillna() returns a new DataFrame by default. For a large DataFrame, that's a copy. When memory is tight and the original isn't needed, pass inplace=True or assign back to the same name.

Dropping Missing Values

dropna() removes rows (or columns) that contain any NaN. Tune it with how, subset, and thresh:

The first call drops every row with any NaN (rows 2, 3, 4 all lose something). The second drops only rows where rating is NaN, which keeps row 3 (missing customer, present rating) and drops rows 2 and 4. subset targets missingness in specific columns.

thresh=N keeps rows with at least N non-NaN values. how="all" only drops rows where every value is NaN. Default is how="any".

Each column has a dtype: the data type of its values. Pandas infers a reasonable dtype during DataFrame creation, but real data often arrives with wrong types (numbers stored as strings, dates as strings) and needs conversion.

object is pandas' catch-all dtype, almost always meaning Python strings. float64 and int64 are NumPy numeric types. There's also bool, datetime64[ns], category, and the newer nullable types like Int64 (capital I) and string.

Converting Columns

astype() changes a column's dtype. It raises if a value can't be converted:

astype(float) parsed the strings as floating-point numbers. If any string was unparseable, it would raise ValueError. For tolerant conversion that turns bad values into NaN, use pd.to_numeric(series, errors="coerce"):

"N/A" couldn't be parsed, so it became NaN. The other values converted normally. Use this pattern for any column that came from a CSV or external source where the cleanliness of the data isn't guaranteed.

For dates, pd.to_datetime() is the parallel:

After conversion, the column supports .dt.year, .dt.month, date arithmetic, and time-based slicing.

Categorical Columns

When a column has a small number of repeating string values (categories, statuses), converting to category saves memory and speeds up groupby:

The category dtype stores each unique value once and replaces each row's value with an integer pointer. For a column of 5 categories repeated across millions of rows, the savings are large. For a column of mostly-unique strings, it's a small loss.

This warning shows up when assigning to a slice of a DataFrame and pandas can't tell whether the assignment was meant to modify the original or a copy. Avoiding it requires understanding what produces a view versus a copy.

What's wrong with this code?

products[products["price"] < 30] could be a view into the original or a fresh copy; pandas doesn't guarantee which. Assigning cheap["on_sale"] = True might modify products or might not, depending on internal details. The warning flags the ambiguity.

Two correct patterns.

Fix 1: Use `.loc[]` on the original.

Single .loc[] call, both the row filter and the column name in one indexing operation. Pandas knows the write target is products and does it cleanly.

Fix 2: Make an explicit copy when a separate DataFrame is wanted.

.copy() produces a guaranteed-new DataFrame. Subsequent assignments to cheap never touch products. The cost is the memory of the copy, which is usually negligible.

The deeper rule: avoid chained indexing when assigning. df[col][row] = value and df[row][col] = value are both forms of "chain two selections and then assign," and both can trigger the warning. Use df.loc[row, col] = value instead. Chained indexing for reading is fine; chained indexing for writing is the problem.