Series

The columns of a DataFrame are constructed using the Series class - these are a 1D array-like object containing values of the same type (similar to a numpy array).

pd.Series([1,2,3,4])

0    1
1    2
2    3
3    4
dtype: int64

pd.Series(["C","B","A"])

0    C
1    B
2    A
dtype: str

pd.Series([True])

0    True
dtype: bool

pd.Series(range(5))

0    0
1    1
2    2
3    3
4    4
dtype: int64

pd.Series([1,"A",True])

0       1
1       A
2    True
dtype: object

Series methods

Once constructed the components of a series can be accessed via the array and index attributes.

s = pd.Series([4,2,1,3])

s

0    4
1    2
2    1
3    3
dtype: int64

s.array

<NumpyExtensionArray>
[4, 2, 1, 3]
Length: 4, dtype: int64

s.index

RangeIndex(start=0, stop=4, step=1)

An index (row names) can also be explicitly provided when constructing a Series,

t = pd.Series([4,2,1,3], index=["a","b","c","d"])

t

a    4
b    2
c    1
d    3
dtype: int64

t.array

<NumpyExtensionArray>
[4, 2, 1, 3]
Length: 4, dtype: int64

t.index

Index(['a', 'b', 'c', 'd'], dtype='str')

Series + NumPy

Series objects are compatible with NumPy-like functions (i.e. vectorized)

t = pd.Series([4,2,1,3], index=["a","b","c","d"])

t + 1

a    5
b    3
c    2
d    4
dtype: int64

t / 2 + 1

a    3.0
b    2.0
c    1.5
d    2.5
dtype: float64

np.log(t)

a    1.386294
b    0.693147
c    0.000000
d    1.098612
dtype: float64

np.exp(-t**2/2)

a    0.000335
b    0.135335
c    0.606531
d    0.011109
dtype: float64

Series indexing

Series can be subset by their index values (not position) or by logical expressions (as with NumPy arrays or R vectors)

t = pd.Series([4,2,1,3], index=["a","b","c","d"])

t["c"]

np.int64(1)

t[["a","d"]]

a    4
d    3
dtype: int64

t[t == 3]

d    3
dtype: int64

t[t % 2 == 0]

a    4
b    2
dtype: int64

t["d"] = 6; t

a    4
b    2
c    1
d    6
dtype: int64

Position based indexing

Series cannot be directly indexed by position, but the iloc attribute can be used for this purpose.

t[1]

KeyError: 1

t[[1,2]]

KeyError: "None of [Index([1, 2], dtype='int64')] are in the [index]"

t.iloc[1]

np.int64(2)

t.iloc[[1,2]]

b    2
c    1
dtype: int64

Index alignment

When performing operations with multiple series, generally pandas will attempt to align the operation by the index values,

m = pd.Series([1,2,3,4], index = ["a","b","c","d"])
n = pd.Series([4,3,2,1], index = ["d","c","b","a"])
o = pd.Series([1,1,1,1,1], index = ["b","d","a","c","e"])

m + n

a    2
b    4
c    6
d    8
dtype: int64

n + m

a    2
b    4
c    6
d    8
dtype: int64

n + o

a    2.0
b    3.0
c    4.0
d    5.0
e    NaN
dtype: float64

Series and dicts

Series can also be constructed from dictionaries, in which case the keys are used as the index.

d = {"anna": "A+", "bob": "B-", "carol": "C", "dave": "D+"}
pd.Series(d)

anna     A+
bob      B-
carol     C
dave     D+
dtype: str

pd.Series(d, index = ["dave","carol","bob","anna"])

dave     D+
carol     C
bob      B-
anna     A+
dtype: str

Missing values

Pandas encodes missing values using NaN (mostly),

s1 = pd.Series(
  {"anna": "A+", "bob": "B-", 
   "carol": "C", "dave": "D+"}, 
  index = ["erin","dave","carol",
           "bob","anna"]
)

s2 = pd.Series(
  {"anna": 97, "bob": 82, 
   "carol": 75, "dave": 68}, 
  index = ["erin","dave","carol",
           "bob","anna"],
  dtype = 'int64'
)

s1

erin     NaN
dave      D+
carol      C
bob       B-
anna      A+
dtype: str

pd.isna(s1)

erin      True
dave     False
carol    False
bob      False
anna     False
dtype: bool

s2

erin      NaN
dave     68.0
carol    75.0
bob      82.0
anna     97.0
dtype: float64

pd.isna(s2)

erin      True
dave     False
carol    False
bob      False
anna     False
dtype: bool

Missing via None

In some cases None can also be used as a missing value, for example:

pd.Series([1,2,3,None])

0    1.0
1    2.0
2    3.0
3    NaN
dtype: float64

pd.Series([True,False,None])

0     True
1    False
2     None
dtype: object

pd.isna( pd.Series([1,2,3,None]) )

0    False
1    False
2    False
3     True
dtype: bool

pd.isna( pd.Series([True,False,None]) )

0    False
1    False
2     True
dtype: bool

Native NAs

If instead of using base dtypes we use Pandas’ built-in dtypes we get “native” support for missing values,

pd.Series(
  [1,2,3,None], 
  dtype = pd.Int64Dtype()
)

0       1
1       2
2       3
3    <NA>
dtype: Int64

pd.Series(
  [True, False,None], 
  dtype = pd.BooleanDtype()
)

0     True
1    False
2     <NA>
dtype: boolean

String series

Series containing strings can have their strings accessed via the str attribute,

s = pd.Series(["the quick", "brown fox", "jumps over", "a lazy dog"])

s

0     the quick
1     brown fox
2    jumps over
3    a lazy dog
dtype: str

s.str.upper()

0     THE QUICK
1     BROWN FOX
2    JUMPS OVER
3    A LAZY DOG
dtype: str

s.str.split(" ")

0      [the, quick]
1      [brown, fox]
2     [jumps, over]
3    [a, lazy, dog]
dtype: object

s.str.split(" ").str[1]

0    quick
1      fox
2     over
3     lazy
dtype: object

pd.Series([1,2,3]).str

AttributeError: Can only use .str accessor with string values, not integer. Did you mean: 'std'?

Categorical Series

pd.Series(
  ["Mon", "Tue", "Wed", "Thu", "Fri"]
)

0    Mon
1    Tue
2    Wed
3    Thu
4    Fri
dtype: str

pd.Series(
  ["Mon", "Tue", "Wed", "Thu", "Fri"],
  dtype="category"
)

0    Mon
1    Tue
2    Wed
3    Thu
4    Fri
dtype: category
Categories (5, str): ['Fri', 'Mon', 'Thu', 'Tue', 'Wed']

pd.Series(
  ["Mon", "Tue", "Wed", "Thu", "Fri"], 
  dtype=pd.CategoricalDtype(ordered=True)
)

0    Mon
1    Tue
2    Wed
3    Thu
4    Fri
dtype: category
Categories (5, str): ['Fri' < 'Mon' < 'Thu' < 'Tue' < 'Wed']

Category orders

pd.Series(
  ["Tue", "Thu", "Mon", "Sat"], 
  dtype=pd.CategoricalDtype(
    categories=["Mon", "Tue", "Wed", "Thu", "Fri"], 
    ordered=True
  )
)

0    Tue
1    Thu
2    Mon
3    NaN
dtype: category
Categories (5, str): ['Mon' < 'Tue' < 'Wed' < 'Thu' < 'Fri']

DataFrame

iris = pd.read_csv("data/iris.csv")
type(iris)

<class 'pandas.DataFrame'>

iris

Constructing DataFrames

We just saw how to read a DataFrame via read_csv(), DataFrames can also be constructed via DataFrame(), in general this is done using a dictionary of columns / Series.

n = 5
d = {
  "id":     np.random.randint(100, 999, n),
  "weight": np.random.normal(70, 20, n),
  "height": np.random.normal(170, 15, n),
  "date":   pd.date_range(start='2/1/2026', periods=n, freq='D')
}

df = pd.DataFrame(d); df

DataFrame from nparray

2d ndarrays can also be used to construct a DataFrame - generally it is a good idea to provide column and row names (indexes)

pd.DataFrame(
  np.diag([1,2,3]),
  columns = ["x","y","z"]
)

pd.DataFrame(
  np.diag([1,2,3]),
  index = ["x","y","z"]
)

pd.DataFrame(
  np.tri(5,3,-1),
  columns = ["x","y","z"],
  index = ["a","b","c","d","e"]
)

DataFrame attributes & methods

df.size

20

df.shape

(5, 4)

df.info()

<class 'pandas.DataFrame'>
RangeIndex: 5 entries, 0 to 4
Data columns (total 4 columns):
 #   Column  Non-Null Count  Dtype         
---  ------  --------------  -----         
 0   id      5 non-null      int64         
 1   weight  5 non-null      float64       
 2   height  5 non-null      float64       
 3   date    5 non-null      datetime64[us]
dtypes: datetime64[us](1), float64(2), int64(1)
memory usage: 292.0 bytes

df.dtypes

id                 int64
weight           float64
height           float64
date      datetime64[us]
dtype: object

df.columns

Index(['id', 'weight', 'height', 'date'], dtype='str')

df.index

RangeIndex(start=0, stop=5, step=1)

df.axes

[RangeIndex(start=0, stop=5, step=1), Index(['id', 'weight', 'height', 'date'], dtype='str')]

DataFrame indexing

df["id"]

0    200
1    995
2    238
3    768
4    342
Name: id, dtype: int64

df.id

0    200
1    995
2    238
3    768
4    342
Name: id, dtype: int64

df[1:3]

df[0::2]

df[0]

KeyError: 0

Indexing by position

As with Series, position based indexing is done via the iloc attribute

df.iloc[1]

id                        995
weight              69.552249
height             196.166608
date      2026-02-02 00:00:00
Name: 1, dtype: object

df.iloc[[1]]

df.iloc[0:2]

df.iloc[1:3,1:3]

df.iloc[0:3, [0,3]]

df.iloc[0:3, [True, True, False, False]]

df.iloc[lambda x: x.index % 2 != 0]

Index by name

df.index = (["anna","bob","carol", "dave", "erin"]); df

df.loc["anna"]

id                        200
weight              57.715447
height             156.243857
date      2026-02-01 00:00:00
Name: anna, dtype: object

df.loc[["anna"]]

type(df.loc["anna"])

<class 'pandas.Series'>

type(df.loc[["anna"]])

<class 'pandas.DataFrame'>

Views vs. Copies

d = pd.DataFrame(np.arange(6).reshape(3,2), columns = ["x","y"]); d

v = d.iloc[0:2,0:2]; v

d.iloc[0,1] = -1; d

v

v.iloc[0,0] = 100; v

d

Element access

df

df[0,0]

KeyError: (0, 0)

df.iat[0,0]

np.int64(200)

df.id[0]

KeyError: 0

df[0:1].id[0]

KeyError: 0

df["anna", "id"]

KeyError: ('anna', 'id')

df.at["anna", "id"]

np.int64(200)

df["id"]["anna"]

np.int64(200)

df["id"][0]

KeyError: 0

Columns and index

When constructing a DataFrame we can specify the indexes for both the rows (index) and columns (columns),

df1 = pd.DataFrame(
  np.random.randn(5, 3), 
  columns=['A', 'B', 'C']
)
df1

df2 = pd.DataFrame(
  np.random.randn(3, 3), 
  index=['x','y','z'], 
  columns=['A', 'B', 'C']
)
df2

df1.columns

Index(['A', 'B', 'C'], dtype='str')

df1.index

RangeIndex(start=0, stop=5, step=1)

df2.columns

Index(['A', 'B', 'C'], dtype='str')

df2.index

Index(['x', 'y', 'z'], dtype='str')

Index objects

pandas’ Index class and its subclasses provide the infrastructure necessary for lookups, data alignment, and other related tasks. You can think of them as being an immutable multiset (i.e. duplicate values are allowed).

pd.Index(['A','B','C'])

Index(['A', 'B', 'C'], dtype='str')

pd.Index(['A','B','C','A'])

Index(['A', 'B', 'C', 'A'], dtype='str')

pd.Index(range(5))

RangeIndex(start=0, stop=5, step=1)

pd.Index(list(range(5)))

Index([0, 1, 2, 3, 4], dtype='int64')

Index names

Index objects can have names which are shown when printing the DataFrame or Index,

df = pd.DataFrame(
  np.random.randn(3, 3), 
  index=pd.Index(['x','y','z'], name="rows"),
  columns=pd.Index(['A', 'B', 'C'], name="cols")
)
df

df.columns

Index(['A', 'B', 'C'], dtype='str', name='cols')

df.index

Index(['x', 'y', 'z'], dtype='str', name='rows')

Indexes and missing values

It is possible for an index to contain missing values (e.g. np.nan) but this is generally a bad idea and should be avoided.

pd.Index([1,2,3,np.nan,5])

Index([1.0, 2.0, 3.0, nan, 5.0], dtype='float64')

pd.Index(["A","B",np.nan,"D", None])

Index(['A', 'B', nan, 'D', nan], dtype='str')

pd.Index([1,2,3,np.nan,5]).fillna(0)

Index([1.0, 2.0, 3.0, 0.0, 5.0], dtype='float64')

pd.Index(["A","B",np.nan,"D", None]).fillna("Z")

Index(['A', 'B', 'Z', 'D', 'Z'], dtype='str')

Changing a DataFrame’s index

data

data.set_index('a')

data.set_index('a').reset_index()

drop argument

data.set_index('c', drop=False)

data.set_index('c').reset_index(drop=True)

Creating a new index

New index values can be attached to a DataFrame via reindex(),

data

data.reindex(columns=["a","b","c","d","e"])

data.reindex(["w","x","y","z"])

data.index = ["w","x","y","z"]; data

data.reindex(range(4,0,-1))

data.index = range(4,0,-1); data

MultiIndex objects

tuples = [('A','x'), ('A','y'),
          ('B','x'), ('B','y'),
          ('C','x'), ('C','y')]
pd.MultiIndex.from_tuples(
  tuples, names=["1st","2nd"]
)

MultiIndex([('A', 'x'),
            ('A', 'y'),
            ('B', 'x'),
            ('B', 'y'),
            ('C', 'x'),
            ('C', 'y')],
           names=['1st', '2nd'])

pd.MultiIndex.from_product(
  [["A","B","C"],
   ["x","y"]], 
  names=["1st","2nd"]
)

MultiIndex([('A', 'x'),
            ('A', 'y'),
            ('B', 'x'),
            ('B', 'y'),
            ('C', 'x'),
            ('C', 'y')],
           names=['1st', '2nd'])

DataFrame with MultiIndex

idx = pd.MultiIndex.from_tuples(
  tuples, names=["1st","2nd"]
)

pd.DataFrame(
  np.random.rand(6,2), 
  index = idx, 
  columns=["m","n"]
)

Column MultiIndex

MultiIndexes can also be used for columns as well,

cidx = pd.MultiIndex.from_product(
  [["A","B"],["x","y"]], names=["c1","c2"]
)

pd.DataFrame(
  np.random.rand(4,4), columns = cidx
)

ridx = pd.MultiIndex.from_product(
  [["m","n"],["l","p"]], names=["r1","r2"]
)

pd.DataFrame(
  np.random.rand(4,4), 
  index= ridx, columns = cidx
)

MultiIndex indexing

data

data["A"]

data["x"]

KeyError: 'x'

data["m"]

KeyError: 'm'

data["m","A"]

KeyError: ('m', 'A')

data["A","x"]

r1  r2
m   l     0.179763
    p     0.220432
n   l     0.823848
    p     0.344536
Name: (A, x), dtype: float64

data["A"]["x"]

r1  r2
m   l     0.179763
    p     0.220432
n   l     0.823848
    p     0.344536
Name: x, dtype: float64

MultiIndex indexing via iloc

data.iloc[0]

c1  c2
A   x     0.179763
    y     0.787379
B   x     0.565183
    y     0.665208
Name: (m, l), dtype: float64

type(data.iloc[0])

<class 'pandas.Series'>

data.iloc[(0,1)]

np.float64(0.7873788607957514)

data.iloc[[0,1]]

data.iloc[:,0]

r1  r2
m   l     0.179763
    p     0.220432
n   l     0.823848
    p     0.344536
Name: (A, x), dtype: float64

type(data.iloc[:,0])

<class 'pandas.Series'>

data.iloc[0,1]

np.float64(0.7873788607957514)

data.iloc[0,[0,1]]

c1  c2
A   x     0.179763
    y     0.787379
Name: (m, l), dtype: float64

MultiIndex indexing via loc

data.loc["m"]

data.loc["l"]

KeyError: 'l'

data.loc[:,"A"]

data.loc[("m","l")]

c1  c2
A   x     0.179763
    y     0.787379
B   x     0.565183
    y     0.665208
Name: (m, l), dtype: float64

data.loc[:,("A","y")]

r1  r2
m   l     0.787379
    p     0.483390
n   l     0.654454
    p     0.541508
Name: (A, y), dtype: float64

Fancier indexing with loc

Index slices can also be used with combinations of indexes and index tuples,

data.loc["m":"n"]

data.loc[("m","p"):"n"]

data.loc[("m","l"):("n","l")]

data.loc[[("m","p"),("n","l")]]

Selecting nested levels

The previous methods don’t give easy access to indexing on nested index levels, this is possible via the cross-section method xs(),

data.xs("p", level="r2")

data.xs("y", level="c2", axis=1)

data.xs("m", level="r1")

data.xs("B", level="c1", axis=1)

Setting MultiIndexes

It is also possible to construct a MultiIndex or modify an existing one using set_index() and reset_index(),

data

data.set_index(['a','b'])

data.set_index(['a','b']).reset_index()

data.set_index('c', append=True)

data.set_index(['a','b']).reset_index(level=1)

Filtering rows

The query() method can be used for filtering rows, it evaluates a string expression in the context of the data frame.

df.query('date == "2026-02-01"')

df.query('weight > 50')

df.query('weight > 50 & height < 165')

qid = 202
df.query('id == @qid')

Selecting Columns

Beyond the use of loc() and iloc() there is also the filter() method which can be used to select columns (or indices) by name with pattern matching

df.filter(items=["id","weight"])

df.filter(like = "i")

df.filter(regex="ght$")

df.filter(like="a", axis=0)

Adding columns

Indexing with assignment allows for inplace modification of a DataFrame, while assign() creates a new object (but is chainable)

df['student'] = [True, True, True, None]
df['age'] = [19, 22, None, None]
df

df.assign(
  student = lambda x: np.where(x.student, "yes", "no"),
  rand = np.random.rand(n)
)

pd.col()

As of pandas 3.0.0 there is also a pd.col() function which can be used within methods like assign() and query() to refer to columns by name,

df.assign(
  bmi = pd.col("weight") / np.power(pd.col("height")/100, 2)
)

Removing columns (and rows)

Columns or rows can be removed via the drop() method,

df.drop(['student'])

KeyError: "['student'] not found in axis"

df.drop(['student'], axis=1)

df.drop(['anna','dave'])

Sorting

DataFrames can be sorted on one or more columns via sort_values(),

df

df.sort_values(by=["student","id"], ascending=[True,False])

join vs merge vs concat

All three can be used to combine data frames,

• concat() stacks DataFrames on either axis, with basic alignment based on (row) indexes. join argument only supports “inner” and “outer”.

• merge() aligns based on one or more shared columns. how supports “inner”, “outer”, “left”, “right”, and “cross”.

• join() uses merge() behind the scenes, but prefers to join based on (row) indexes. Also has different default how compared to merge(), “left” vs “inner”.

Next week

• Pivoting

• Split-Apply-Combine (group by / summarize)

• Pandas & pyarrow

• Polars